Self-Supervised Reinforcement Learning for Recommender Systems

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail to model them appropriately. Casting sequential recommendation task as a reinforcement learning (RL) problem is a promising direction. A major component of RL approaches is to train the agent through interactions with the environment. However, it is often problematic to train a recommender in an on-line fashion due to the requirement to expose users to irrelevant recommendations. As a result, learning the policy from logged implicit feedback is of vital importance, which is challenging due to the pure off-policy setting and lack of negative rewards (feedback). In this paper, we propose self-supervised reinforcement learning for sequential recommendation tasks. Our approach augments standard recommendation models with two output layers: one for self-supervised learning and the other for RL. The RL part acts as a regularizer to drive the supervised layer focusing on specific rewards(e.g., recommending items which may lead to purchases rather than clicks) while the self-supervised layer with cross-entropy loss provides strong gradient signals for parameter updates. Based on such an approach, we propose two frameworks namely Self-Supervised Q-learning(SQN) and Self-Supervised Actor-Critic(SAC). We integrate the proposed frameworks with four state-of-the-art recommendation models. Experimental results on two real-world datasets demonstrate the effectiveness of our approach.Comment: SIGIR202

Deep learning-based implicit feedback recommendation

Recommender systems are of vital importance, in the era of the Web, to address the problem of information overload. It can benefit both users by recommending personalized interesting items and service providers by increasing their site traffic. Plenty of use cases have emerged as applied recommender systems, including but not limited to multimedia recommendation (e.g., news, movies, music, and videos) and e-commerce recommendation. A recommendation agent can be trained from user-item interaction data which can be categorized as explicit feedback and implicit feedback. Compared with explicit ratings which depict the user preference explicitly, implicit feedback data like clicks, purchases, and dwell time is more prevalent in the real-world scenario. On the other hand, deep learning has achieved great success recently due to the high model expressiveness and fidelity. In this thesis, we investigate deep learning techniques for recommendation from implicit feedback data. We focus on two learning perspectives: deep supervised learning and deep reinforcement learning. Supervised learning tries to infer knowledge from implicit historical interactions. From this perspective, two models namely Convolutional Factorization Machines (CFM) and Relational Collaborative Filtering (RCF) are proposed. CFM tackles the implicit user-item interactions with side information as feature vectors and utilizes convolutional neural networks to learn high-order interaction signals. RCF considers multiple item relations into the recommendation model and tackles the implicit feedback as relation-enriched data. The two models investigate deep learning techniques for recommendation by tackling the data as two different structures: feature vectors and relations. Experimental results demonstrate that the proposed deep learning models are effective to improve the recommendation accuracy. Besides, RCF also helps to provide explainable recommendation and get a better comprehension of user behaviors. Reinforcement learning is reward-driven and focuses on long-term optimization in a whole interaction session, which conforms more with the objective of recommender systems. From this perspective, we first formulate the next-item recommendation task from implicit feedback data as a Markov Decision Process (MDP). Then we analyzed that directly utilizing reinforcement learning algorithms for recommendation is infeasible due to the challenge of pure off-policy setting and lack of negative reward signals. To address the problems, we proposed Self-Supervised Q-learning (SQN) and Self-Supervised Actor-Critic (SAC). The key insight is to combine reinforcement learning with supervised learning and perform knowledge transfer between the two components. Based on SQN and SAC, we further proposed Self-Supervised Negative Q-learning (SNQN) and Self-Supervised Advantage Actor-Critic (SA2C) to introduce the negative sampling strategy to enhance the learning of the reinforcement component. Experimental results demonstrate that the proposed learning frameworks are effective when integrated with different existing base models. Moreover, we show that combining supervised learning and reinforcement learning is a promising direction for future recommender systems. In that case, reinforcement learning introduces reward-driven objectives and long-term optimization perspectives into supervised learning while supervised learning helps to improve the data efficiency for reinforcement learning

Deep Learning based Recommender System: A Survey and New Perspectives

With the ever-growing volume of online information, recommender systems have been an effective strategy to overcome such information overload. The utility of recommender systems cannot be overstated, given its widespread adoption in many web applications, along with its potential impact to ameliorate many problems related to over-choice. In recent years, deep learning has garnered considerable interest in many research fields such as computer vision and natural language processing, owing not only to stellar performance but also the attractive property of learning feature representations from scratch. The influence of deep learning is also pervasive, recently demonstrating its effectiveness when applied to information retrieval and recommender systems research. Evidently, the field of deep learning in recommender system is flourishing. This article aims to provide a comprehensive review of recent research efforts on deep learning based recommender systems. More concretely, we provide and devise a taxonomy of deep learning based recommendation models, along with providing a comprehensive summary of the state-of-the-art. Finally, we expand on current trends and provide new perspectives pertaining to this new exciting development of the field.Comment: The paper has been accepted by ACM Computing Surveys. https://doi.acm.org/10.1145/328502

mARC: Memory by Association and Reinforcement of Contexts

This paper introduces the memory by Association and Reinforcement of Contexts (mARC). mARC is a novel data modeling technology rooted in the second quantization formulation of quantum mechanics. It is an all-purpose incremental and unsupervised data storage and retrieval system which can be applied to all types of signal or data, structured or unstructured, textual or not. mARC can be applied to a wide range of information clas-sification and retrieval problems like e-Discovery or contextual navigation. It can also for-mulated in the artificial life framework a.k.a Conway "Game Of Life" Theory. In contrast to Conway approach, the objects evolve in a massively multidimensional space. In order to start evaluating the potential of mARC we have built a mARC-based Internet search en-gine demonstrator with contextual functionality. We compare the behavior of the mARC demonstrator with Google search both in terms of performance and relevance. In the study we find that the mARC search engine demonstrator outperforms Google search by an order of magnitude in response time while providing more relevant results for some classes of queries