Enhancing VAEs for Collaborative Filtering: Flexible Priors & Gating Mechanisms

Neural network based models for collaborative filtering have started to gain attention recently. One branch of research is based on using deep generative models to model user preferences where variational autoencoders were shown to produce state-of-the-art results. However, there are some potentially problematic characteristics of the current variational autoencoder for CF. The first is the too simplistic prior that VAEs incorporate for learning the latent representations of user preference. The other is the model's inability to learn deeper representations with more than one hidden layer for each network. Our goal is to incorporate appropriate techniques to mitigate the aforementioned problems of variational autoencoder CF and further improve the recommendation performance. Our work is the first to apply flexible priors to collaborative filtering and show that simple priors (in original VAEs) may be too restrictive to fully model user preferences and setting a more flexible prior gives significant gains. We experiment with the VampPrior, originally proposed for image generation, to examine the effect of flexible priors in CF. We also show that VampPriors coupled with gating mechanisms outperform SOTA results including the Variational Autoencoder for Collaborative Filtering by meaningful margins on 2 popular benchmark datasets (MovieLens & Netflix)

Enhancing VAEs for Collaborative Filtering: Flexible Priors & Gating Mechanisms

학위논문(석사)--서울대학교 대학원 :융합과학기술대학원 융합과학부(디지털정보융합전공),2019. 8. 서봉원.Since Matrix Factorization based linear models have been dominant in the Collaborative Filtering context for a long time in the past, Neural Network based CF Models for recommendation have started to gain attention recently. One branch of research is based on using deep generative models to model user preferences and Variational Autoencoders where shown to give state-of-the-art results. However, there are some potentially problematic characteristics of the current Variational Autoencoder for CF. The first is the too simplistic prior VAEs incorporate for learning the latent representations of user preference, which may be restricting the model from learning more expressive and richer latent variables that could boost recommendation performance. The other is the models inability to learn deeper representations with more than one hidden layer. Our goal is to incorporate appropriate techniques in order to mitigate the aforementioned problems of Variational Autoencoder CF and further improve the recommendation performance of VAE based Collaborative Fil-tering. We bring the VampPrior, which successfully made improvements for image generation to tackle the restrictive prior problem. We also adopt Gat-ed Linear Units (GLUs) which were used in stacked convolutions for lan-guage modeling to control information flow in the easily deepening auto-encoder framework. We show that such simple priors (in original VAEs) may be too restric-tive to fully model user preferences and setting a more flexible prior gives significant gains. We also show that VAMP priors coupled with gating mechanisms outperform SOTA results including the Variational Autoencoder for Collaborative Filtering by meaningful margins on 4 benchmark datasets (MovieLens, Netflix, Pinterest, Melon).최근 뉴럴넷 기반 협업필터링 추천알고리즘이 주목을 받고 있다. 그 중 한 갈래의 연구는 깊은 생성모형 (Deep Generative Model)을 이용해 사용자들의 선호를 모델링하는 방법이다. 이중 Variational Autoencoder를 (VAE) 이용한 방법이 최근 state-of-the-art (SOTA) 성능을 보여주었다. 그러나 VAE를 이용한 협업필터링 알고리즘은 현재 몇 가지의 잠재적인 문제점들을 지니고 있다. 첫 번째는 사용자 선호를 압축하는 잠재변수를 학습하는 과정에서 매우 단순한 사전분포를 사용한다는 것이다. 또 다른 문제점은 모델이 현재 여러 단을 이용한 깊은 인코더와 디코더를 사용하지 못하고 있다는 것이다. 본 연구는 최신기술들을 활용하여 앞선 문제점들을 해결하고 VAE를 이용한 협업필터링 알고리즘의 추천성능을 더욱 높이는 것이 목표이다. 본 연구는 협업필터링 문제에 더 복잡한 사전분포 (Flexible Prior)를 적용한 첫 연구로서, 기존의 단순한 사전분포가 모델의 표현력을 제한할 수 있으며 더 복잡한 사전분포를 정의함으로써 모델의 성능을 더욱 높일 수 있음을 보였다. 이를 위해 이미지 생성 문제에서 좋은 결과를 보인 VampPrior를 이용해 실험을 진행하였다. 또한 VampPrior를 Gating Mechanisim과 함께 사용하였을 때 기존 SOTA를 넘어서는 성능을 보임을 추천알고리즘에서 사용되는 대표적인 데이터셋들을 통해 보여준다.1 INTRODUCTION 1 1.1 Background and Motivation 1 1.2 Research Goal 3 1.3 Enhancing VAEs for Collaborative Filtering 3 1.4 Experiments 5 1.5 Contributions 5 2 RELATED WORK 7 2.1 Collaborative Filtering 7 2.1.1 Traditional methods & Matrix-Factorization based CF 8 2.1.2 Autoencoders for CF 12 2.2 Deep Generative Models (VAE) 17 2.2.1 Variational Bayes 18 2.2.2 Variational Autoencoder 18 2.3 Variational Autoencoder for Collaborative Filtering 20 2.3.1 VAE for CF 21 2.4 Recent research in Computer Vision & Deep Learning 24 2.4.1 VampPrior 24 2.4.2 Gated CNN 25 3 METHOD 28 3.1 Flexible Prior 29 3.1.1 Motivation 29 3.1.2 VampPrior 30 3.1.3 Hierarchical Stochastic Units 31 3.2 Gating Mechanism 32 3.2.1 Motivation 32 3.2.2 Gated Linear Units 34 4 EXPERIMENT 35 4.1 Setup 35 4.1.1 Baseline Models 35 4.1.2 Proposed Models 37 4.1.3 Strong Generalization 37 4.1.4 Evaluation Metrics 38 4.2 Datasets 38 4.3 Configurations 39 4.4 Results 40 4.4.1 Model Performance 40 4.4.5 Further Analysis on the Effect of Gating 44 5 CONCLUSION 45 Bibliography 47 국문초록 51Maste

ロバスト性，正則化，タスク不変性に関する変分オートエンコーダの改善

京都大学新制・課程博士博士(情報学)甲第24725号情博第813号新制||情||137(附属図書館)京都大学大学院情報学研究科知能情報学専攻(主査)教授 鹿島 久嗣, 教授 山本 章博, 教授 吉川 正俊学位規則第4条第1項該当Doctor of InformaticsKyoto UniversityDFA