Graph Exploration Matters: Improving both individual-level and system-level diversity in WeChat Feed Recommender

There are roughly three stages in real industrial recommendation systems, candidates generation (retrieval), ranking and reranking. Individual-level diversity and system-level diversity are both important for industrial recommender systems. The former focus on each single user's experience, while the latter focus on the difference among users. Graph-based retrieval strategies are inevitably hijacked by heavy users and popular items, leading to the convergence of candidates for users and the lack of system-level diversity. Meanwhile, in the reranking phase, Determinantal Point Process (DPP) is deployed to increase individual-level diverisity. Heavily relying on the semantic information of items, DPP suffers from clickbait and inaccurate attributes. Besides, most studies only focus on one of the two levels of diversity, and ignore the mutual influence among different stages in real recommender systems. We argue that individual-level diversity and system-level diversity should be viewed as an integrated problem, and we provide an efficient and deployable solution for web-scale recommenders. Generally, we propose to employ the retrieval graph information in diversity-based reranking, by which to weaken the hidden similarity of items exposed to users, and consequently gain more graph explorations to improve the system-level diveristy. Besides, we argue that users' propensity for diversity changes over time in content feed recommendation. Therefore, with the explored graph, we also propose to capture the user's real-time personalized propensity to the diversity. We implement and deploy the combined system in WeChat App's Top Stories used by hundreds of millions of users. Offline simulations and online A/B tests show our solution can effectively improve both user engagement and system revenue

Discrete Conditional Diffusion for Reranking in Recommendation

Reranking plays a crucial role in modern multi-stage recommender systems by rearranging the initial ranking list to model interplay between items. Considering the inherent challenges of reranking such as combinatorial searching space, some previous studies have adopted the evaluator-generator paradigm, with a generator producing feasible sequences and a evaluator selecting the best one based on estimated listwise utility. Inspired by the remarkable success of diffusion generative models, this paper explores the potential of diffusion models for generating high-quality sequences in reranking. However, we argue that it is nontrivial to take diffusion models as the generator in the context of recommendation. Firstly, diffusion models primarily operate in continuous data space, differing from the discrete data space of item permutations. Secondly, the recommendation task is different from conventional generation tasks as the purpose of recommender systems is to fulfill user interests. Lastly, real-life recommender systems require efficiency, posing challenges for the inference of diffusion models. To overcome these challenges, we propose a novel Discrete Conditional Diffusion Reranking (DCDR) framework for recommendation. DCDR extends traditional diffusion models by introducing a discrete forward process with tractable posteriors, which adds noise to item sequences through step-wise discrete operations (e.g., swapping). Additionally, DCDR incorporates a conditional reverse process that generates item sequences conditioned on expected user responses. Extensive offline experiments conducted on public datasets demonstrate that DCDR outperforms state-of-the-art reranking methods. Furthermore, DCDR has been deployed in a real-world video app with over 300 million daily active users, significantly enhancing online recommendation quality

Evaluating Recommender Systems Qualitatively: A survey and Comparative Analysis

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Business AnalyticsRecommender systems have improved users' online quality of life by helping them find interesting and valuable items within a large item set. Most recommender system validation research has focused on accuracy metrics, studying the differences between the predicted and actual user ratings. However, recent research has found accuracy to underperform when systems go live, mainly due to accuracy’s inability to validate recommendation lists as a single entity, and shifted to evaluating recommender systems using "beyond-accuracy" metrics, like novelty and diversity. In this dissertation, we summarize and organize the leading research regarding the definitions and objectives of the beyond-accuracy metrics. Such metrics include coverage, diversity, novelty, serendipity, unexpectedness, utility, and fairness. The behaviors and relationships of these metrics are analyzed using four different models, two concerning the items characteristics (item-based) and two regarding the user behaviors (user-based). Furthermore, a new metric is proposed that allows the comparison of different models considering their overall beyond-accuracy performance. Using this metric, a reraking approach is designed to improve the performance of a system, aiming to achieve better recommendations. The impact of the reranking technique on each metric and algorithm is studied, and the accuracy and non-accuracy performance of each system is compared. We realized that, although the reranking technique can increase most beyond-accuracy metrics, the accuracy of that system starts to worsen due to the negative correlation between these two dimensions. We also found that item-based models tend to achieve much lower values of coverage and diversity than userbased models