Deep Learning based Recommender System: A Survey and New Perspectives

With the ever-growing volume of online information, recommender systems have been an effective strategy to overcome such information overload. The utility of recommender systems cannot be overstated, given its widespread adoption in many web applications, along with its potential impact to ameliorate many problems related to over-choice. In recent years, deep learning has garnered considerable interest in many research fields such as computer vision and natural language processing, owing not only to stellar performance but also the attractive property of learning feature representations from scratch. The influence of deep learning is also pervasive, recently demonstrating its effectiveness when applied to information retrieval and recommender systems research. Evidently, the field of deep learning in recommender system is flourishing. This article aims to provide a comprehensive review of recent research efforts on deep learning based recommender systems. More concretely, we provide and devise a taxonomy of deep learning based recommendation models, along with providing a comprehensive summary of the state-of-the-art. Finally, we expand on current trends and provide new perspectives pertaining to this new exciting development of the field.Comment: The paper has been accepted by ACM Computing Surveys. https://doi.acm.org/10.1145/328502

Neural Networks for Personalized Recommender Systems

The recommender system is an essential tool for companies and users. A successful recommender system not only can help companies promote their products and services, but also benefit users by filtering out unwanted information. Thus, recommender systems are growing to be indispensable in a wide range of industries. Moreover, due to the fact that neural networks have been proved to be efficient and scalable, they are widely studied and applied to various fields. This thesis aims at developing methods for recommender systems by adapting neural networks. By exploring to adapt neural networks to recommender systems, this thesis investigates challenges that recommender systems are facing, and presents approaches to these challenges. Specifically, these challenges include: (1) data sparsity, (2) the complex relationships between users and items, (3) dynamic user preferences. To address the data sparsity, this thesis proposes to learn both collaborative features and content representations to generate recommendations in case of sparse data. Moreover, it proposes an architecture for the training process to further improve the quality of recommendations. To dynamically learn users' preferences, the thesis proposes to learn temporal features to capture dynamic changes of users' preferences. In this way, both the users' general preferences and the latest interactions are considered. To learn the complex relationships, this thesis also proposes a geometric method to measure nonlinear metric to learn the complex relationship among users and items. Moreover, the relationships between items are also considered to avoid potential problems

Graph Transformer for Recommendation

This paper presents a novel approach to representation learning in recommender systems by integrating generative self-supervised learning with graph transformer architecture. We highlight the importance of high-quality data augmentation with relevant self-supervised pretext tasks for improving performance. Towards this end, we propose a new approach that automates the self-supervision augmentation process through a rationale-aware generative SSL that distills informative user-item interaction patterns. The proposed recommender with Graph TransFormer (GFormer) that offers parameterized collaborative rationale discovery for selective augmentation while preserving global-aware user-item relationships. In GFormer, we allow the rationale-aware SSL to inspire graph collaborative filtering with task-adaptive invariant rationalization in graph transformer. The experimental results reveal that our GFormer has the capability to consistently improve the performance over baselines on different datasets. Several in-depth experiments further investigate the invariant rationale-aware augmentation from various aspects. The source code for this work is publicly available at: https://github.com/HKUDS/GFormer.Comment: Accepted by SIGIR'202

A Survey of Graph Neural Networks for Social Recommender Systems

Social recommender systems (SocialRS) simultaneously leverage user-to-item interactions as well as user-to-user social relations for the task of generating item recommendations to users. Additionally exploiting social relations is clearly effective in understanding users' tastes due to the effects of homophily and social influence. For this reason, SocialRS has increasingly attracted attention. In particular, with the advance of Graph Neural Networks (GNN), many GNN-based SocialRS methods have been developed recently. Therefore, we conduct a comprehensive and systematic review of the literature on GNN-based SocialRS. In this survey, we first identify 80 papers on GNN-based SocialRS after annotating 2151 papers by following the PRISMA framework (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). Then, we comprehensively review them in terms of their inputs and architectures to propose a novel taxonomy: (1) input taxonomy includes 5 groups of input type notations and 7 groups of input representation notations; (2) architecture taxonomy includes 8 groups of GNN encoder, 2 groups of decoder, and 12 groups of loss function notations. We classify the GNN-based SocialRS methods into several categories as per the taxonomy and describe their details. Furthermore, we summarize the benchmark datasets and metrics widely used to evaluate the GNN-based SocialRS methods. Finally, we conclude this survey by presenting some future research directions.Comment: GitHub repository with the curated list of papers: https://github.com/claws-lab/awesome-GNN-social-recsy

Recent Developments in Recommender Systems: A Survey

In this technical survey, we comprehensively summarize the latest advancements in the field of recommender systems. The objective of this study is to provide an overview of the current state-of-the-art in the field and highlight the latest trends in the development of recommender systems. The study starts with a comprehensive summary of the main taxonomy of recommender systems, including personalized and group recommender systems, and then delves into the category of knowledge-based recommender systems. In addition, the survey analyzes the robustness, data bias, and fairness issues in recommender systems, summarizing the evaluation metrics used to assess the performance of these systems. Finally, the study provides insights into the latest trends in the development of recommender systems and highlights the new directions for future research in the field

Representation Learning for Texts and Graphs: A Unified Perspective on Efficiency, Multimodality, and Adaptability

[...] This thesis is situated between natural language processing and graph representation learning and investigates selected connections. First, we introduce matrix embeddings as an efficient text representation sensitive to word order. [...] Experiments with ten linguistic probing tasks, 11 supervised, and five unsupervised downstream tasks reveal that vector and matrix embeddings have complementary strengths and that a jointly trained hybrid model outperforms both. Second, a popular pretrained language model, BERT, is distilled into matrix embeddings. [...] The results on the GLUE benchmark show that these models are competitive with other recent contextualized language models while being more efficient in time and space. Third, we compare three model types for text classification: bag-of-words, sequence-, and graph-based models. Experiments on five datasets show that, surprisingly, a wide multilayer perceptron on top of a bag-of-words representation is competitive with recent graph-based approaches, questioning the necessity of graphs synthesized from the text. [...] Fourth, we investigate the connection between text and graph data in document-based recommender systems for citations and subject labels. Experiments on six datasets show that the title as side information improves the performance of autoencoder models. [...] We find that the meaning of item co-occurrence is crucial for the choice of input modalities and an appropriate model. Fifth, we introduce a generic framework for lifelong learning on evolving graphs in which new nodes, edges, and classes appear over time. [...] The results show that by reusing previous parameters in incremental training, it is possible to employ smaller history sizes with only a slight decrease in accuracy compared to training with complete history. Moreover, weighting the binary cross-entropy loss function is crucial to mitigate the problem of class imbalance when detecting newly emerging classes. [...

Hierarchical Information and Data Modeling for Neural-based Recommender Systems

In the era of information flooding, efficient information retrieval has become a non-neglectful problem. Recommender system, an information filtering system that provides customized suggestions of items that are most likely to be of interest to users has been applied to many customer-oriented services. Recently, more and more neural-based and graph-based models have been studied and adapted in recommender systems, owing to their superiority in dealing with fundamental machine learning problems. However, most existing approaches merely focus on accuracy improvements and ignore that higher recommendation accuracy does not directly imply better recommendations. This thesis aims to propose novel methods for recommender systems that enable higher recommendation accuracy and higher recommendation satisfaction simultaneously. In this thesis, three approaches are proposed from two perspectives, which are (1) generating more personalized individual embeddings and (2) reducing inference latency. To improve the embedding learning process, the valuable information stored in pairwise preference differences and the hierarchical structures exhibited in user (item) latent relationships are explored and investigated. Firstly, a novel and straightforward pointwise training strategy, named Difference Embedding (DifE), is proposed to capture the valuable information retained in pairwise preference differences. More specifically, by using a novel projection on the designed pairwise differences function, the final derived pointwise loss function allows recommendation models to encode valuable personalized information and achieve more customized predictions. Moreover, a U-shaped graph convolutional network-based recommender system, named UGCN, is proposed to explore the implicit and inherent hierarchies of the user or item. Concretely, with the hierarchical encoding-decoding process, the UGCN model is able to capture user-item relationships at various resolution scales and would finally result in better preference customization. To reduce inference latency, two knowledge distillation methods are also proposed in the model construction and training process. By training with the valuable information distilled from the sophisticated teacher recommenders, the compact student model can achieve extraordinary recommendation performances and light model architecture simultaneously

MetaRec: Meta-Learning Meets Recommendation Systems

Artificial neural networks (ANNs) have recently received increasing attention as powerful modeling tools to improve the performance of recommendation systems. Meta-learning, on the other hand, is a paradigm that has re-surged in popularity within the broader machine learning community over the past several years. In this thesis, we will explore the intersection of these two domains and work on developing methods for integrating meta-learning to design more accurate and flexible recommendation systems. In the present work, we propose a meta-learning framework for the design of collaborative filtering methods in recommendation systems, drawing from ideas, models, and solutions from modern approaches in both the meta-learning and recommendation system literature, applying them to recommendation tasks to obtain improved generalization performance. Our proposed framework, MetaRec, includes and unifies the main state-of-the-art models in recommendation systems, extending them to be flexibly configured and efficiently operate with limited data. We empirically test the architectures created under our MetaRec framework on several recommendation benchmark datasets using a plethora of evaluation metrics and find that by taking a meta-learning approach to the collaborative filtering problem, we observe notable gains in predictive performance

개인화 코디 추천을 위한 위음성 증류 및 대조 학습

학위논문(석사) -- 서울대학교대학원 : 공과대학 컴퓨터공학부, 2022.2. 이상구.Personalized outfit recommendation has recently been in the spotlight with the rapid growth of the online fashion industry. However, recommending outfits has two significant challenges that should be addressed. The first challenge is that outfit recommendation often requires a complex and large model that utilizes visual information, incurring huge memory and time costs. One natural way to mitigate this problem is to compress such a cumbersome model with knowledge distillation (KD) techniques that leverage knowledge from a pretrained teacher model. However, it is hard to apply existing KD approaches in recommender systems (RS) to the outfit recommendation because they require the ranking of all possible outfits while the number of outfits grows exponentially to the number of consisting clothing items. Therefore, we propose a new KD framework for outfit recommendation, called False Negative Distillation (FND), which exploits false-negative information from the teacher model while not requiring the ranking of all candidates. The second challenge is that the explosive number of outfit candidates amplifying the data sparsity problem, often leading to poor outfit representation. To tackle this issue, inspired by the recent success of contrastive learning (CL), we introduce a CL framework for outfit representation learning with two proposed data augmentation methods. Quantitative and qualitative experiments on outfit recommendation datasets demonstrate the effectiveness and soundness of our proposed methods.최근 온라인 패션 산업이 급성장하면서 개인화 코디 추천이 각광받고 있다. 그러나 코디 추천은 해결해야 할 두 가지 중요한 챌린지가 있다. 첫 번째 챌린지는 코디 추천이 주로 시각 정보를 활용하는 복잡하고 큰 모델을 필요로 하기 때문에 상당한 시간과 메모리 비용이 발생한다는 것이다. 이 문제를 완화하는 한 가지 자연스러운 방법은 사전 훈련된 교사 모델의 지식을 활용하는 지식 증류 기술을 이용하여 이러한 성가신 모델을 압축하는 것이다. 그러나 추천 시스템의 기존 지식 증류 접근법은 가능한 모든 코디의 순위를 필요로 하며, 코디의 수는 구성되는 의상의 수에 따라 기하급수적으로 증가하기 때문에 코디 추천에 기존 지식 증류 접근법을 적용하는 것은 상당히 까다로운 작업이다. 따라서 우리는 모든 후보 코디의 순위를 요구하지 않으면서 교사 모델의 위음성 정보를 활용하는 위음성 증류라는 새로운 지식 증류 프레임워크를 제안한다. 두 번째 챌린지는 코디 후보의 폭발적인 수로 인해 데이터 희소성 문제가 증폭되어 종종 코디 표현(representation)이 좋지 않다는 것이다. 이 문제를 해결하기 위해 최근 대조 학습의 성공에 영감을 받아 새로운 두 가지 데이터 증강 기법을 사용하는 코디 표현 학습을 위한 대조 학습 프레임워크를 제안한다. 우리는 코디 추천 데이터 세트에 대한 양적 및 질적 실험을 통해 제안된 방법의 효과와 타당성을 보인다.Abstract i Contents ii List of Tables v List of Figures vi 1 Introduction 1 2 Related Work 5 2.1 Outfit Recommendation 5 2.2 Knowledge Distillation 6 2.3 Contrastive Learning 6 3 Approach 7 3.1 Background: Computing the Preference Score to an Outfit 8 3.1.1 Set Transformer 9 3.1.2 Preference score prediction 10 3.2 False Negative Distillation 10 3.2.1 Teacher model 10 3.2.2 Student model 11 3.3 Contrastive Learning for Outfits 13 3.3.1 Erase 14 3.3.2 Replace 14 3.4 Final Objective: FND-CL 14 3.5 Profiling Cold Starters 15 3.5.1 Average (avg) 16 3.5.2 Weighted Average (w-avg) 16 4 Experiment 17 4.1 Experimental Design 17 4.1.1 Datasets 17 4.1.2 Evaluation metrics 18 4.1.3 Considered methods 18 4.1.4 Implementation details 19 4.2 Performance Comparison 20 4.3 Performance on Cold Starters 21 4.4 Performance on Hard Negative Outfits 22 4.5 Performance with Different α 23 4.6 Performance with Different Augmentations 24 4.7 Performance with Different Model Sizes 25 4.8 Performance with Different Batch Sizes 27 4.9 Visualization of the User-Outfit Space 28 5 Conclusion 30 Bibliography 31 A Appendix 37 A.1 Enhancing the Performance of a Teacher Model 37 A.1.1 Teacher-CL 38 A.1.2 Employing Teacher-CL: FND-CL* 39 Abstract (In Korean) 40석

Disentangled Variational Auto-encoder Enhanced by Counterfactual Data for Debiasing Recommendation

Recommender system always suffers from various recommendation biases, seriously hindering its development. In this light, a series of debias methods have been proposed in the recommender system, especially for two most common biases, i.e., popularity bias and amplified subjective bias. However, exsisting debias methods usually concentrate on correcting a single bias. Such single-functionality debiases neglect the bias-coupling issue in which the recommended items are collectively attributed to multiple biases. Besides, previous work cannot tackle the lacking supervised signals brought by sparse data, yet which has become a commonplace in the recommender system. In this work, we introduce a disentangled debias variational auto-encoder framework(DB-VAE) to address the single-functionality issue as well as a counterfactual data enhancement method to mitigate the adverse effect due to the data sparsity. In specific, DB-VAE first extracts two types of extreme items only affected by a single bias based on the collier theory, which are respectively employed to learn the latent representation of corresponding biases, thereby realizing the bias decoupling. In this way, the exact unbiased user representation can be learned by these decoupled bias representations. Furthermore, the data generation module employs Pearl's framework to produce massive counterfactual data, making up the lacking supervised signals due to the sparse data. Extensive experiments on three real-world datasets demonstrate the effectiveness of our proposed model. Besides, the counterfactual data can further improve DB-VAE, especially on the dataset with low sparsity