User effort vs. accuracy in rating-based elicitation

One of the unresolved issues when designing a recommender system is the number of ratings -- i.e., the profile length -- that should be collected from a new user before providing recommendations. A design tension exists, induced by two conflicting requirements. On the one hand, the system must collect "enough"ratings from the user in order to learn her/his preferences and improve the accuracy of recommendations. On the other hand, gathering more ratings adds a burden on the user, which may negatively affect the user experience. Our research investigates the effects of profile length from both a subjective (user-centric) point of view and an objective (accuracy-based) perspective. We carried on an offline simulation with three algorithms, and a set of online experiments involving overall 960 users and four recommender algorithms, to measure which of the two contrasting forces influenced by the number of collected ratings -- recommendations relevance and burden of the rating process -- has stronger effects on the perceived quality of the user experience. Moreover, our study identifies the potentially optimal profile length for an explicit, rating based, and human controlled elicitation strategy

Overview of CLEF NEWSREEL 2014: News Recommendations Evaluation Labs

This paper summarises objectives, organisation, and results of the first news recommendation evaluation lab (NEWSREEL 2014). NEWSREEL targeted the evaluation of news recommendation algorithms in the form of a campaignstyle evaluation lab. Participants had the chance to apply two types of evaluation schemes. On the one hand, participants could apply their algorithms onto a data set. We refer to this setting as off-line evaluation. On the other hand, participants could deploy their algorithms on a server to interactively receive recommendation requests. We refer to this setting as on-line evaluation. This setting ought to reveal the actual performance of recommendation methods. The competition strived to illustrate differences between evaluation with historical data and actual users. The on-line evaluation does reflect all requirements which active recommender systems face in practise. These requirements include real-time responses and large-scale data volumes. We present the competition’s results and discuss commonalities regarding participants’ approaches

Eigenvalue analogy for confidence estimation in item-based recommender systems

Item-item collaborative filtering (CF) models are a well known and studied family of recommender systems, however current literature does not provide any theoretical explanation of the conditions under which item-based recommendations will succeed or fail. We investigate the existence of an ideal item-based CF method able to make perfect recommendations. This CF model is formalized as an eigenvalue problem, where estimated ratings are equivalent to the true (unknown) ratings multiplied by a user-specific eigenvalue of the similarity matrix. Preliminary experiments show that the magnitude of the eigenvalue is proportional to the accuracy of recommendations for that user and therefore it can provide reliable measure of confidence

High Order Profile Expansion to tackle the new user problem on recommender systems

Data Availability: The complete dataset for the High Order Profile Expansion experiments has been published in the public repository: https://doi.org/10.6084/m9.figshare.9798155.[Abstract] Collaborative Filtering algorithms provide users with recommendations based on their opinions, that is, on the ratings given by the user for some items. They are the most popular and widely implemented algorithms in Recommender Systems, especially in e-commerce, considering their good results. However, when the information is extremely sparse, independently of the domain nature, they do not present such good results. In particular, it is difficult to offer recommendations which are accurate enough to a user who has just arrived to a system or who has rated few items. This is the well-known new user problem, a type of cold-start. Profile Expansion techniques had been already presented as a method to alleviate this situation. These techniques increase the size of the user profile, by obtaining information about user tastes in distinct ways. Therefore, recommender algorithms have more information at their disposal, and results improve. In this paper, we present the High Order Profile Expansion techniques, which combine in different ways the Profile Expansion methods. The results show 110% improvement in precision over the algorithm without Profile Expansion, and 10% improvement over Profile Expansion techniques.Ministerio de Economía y Competitividad; TIN2015-70648-PXunta de Galicia; ED431G/01 2016-201

User interface patterns in recommendation-empowered content intensive multimedia applications

Design Patterns (DPs) are acknowledged as powerful conceptual tools to improve design quality and to reduce time and cost of the development process by effect of the reuse of “good” design solutions. In many fields (e.g., software engineering, web engineering, interface design) patterns are widely used by practitioners and are also investigated from a research perspective. Still, they have been seldom explored in the arena of Recommender Systems (RSs). RSs provide suggestions (“recommendations”) for items that are likely to be appropriate for the user profile, and are increasingly adopted in content-intensive multimedia applications to complement traditional forms of search in large information spaces. This paper explores RSs through the lens of User Interface (UI) Design Patterns. We have performed a systematic analysis of 54 recommendation-empowered content-intensive multimedia applications, in order to: (i) discover the occurrences of existing domain independent UI patterns; (ii) identify frequently adopted UI solutions that are not modelled by existing patterns, and define a set of new UI patterns, some of which are specific of the interfaces for recommendation features while others can be useful also in a broader context. The results of our inspection have been discussed with and evaluated by a team of experts, leading to a consolidated set of 14 new patterns that are reported in the paper. Reusing pattern-based design solutions instead of building new solutions from scratch enables novice and expert designers to build good UIs for Recommendation-empowered content intensive multimedia applications more effectively, and ultimately can improve the UX experience in this class of systems. From a broader perspective, our work can stimulate future research bridging Recommender Systems, Web Engineering and Interface Design by means of Design Patterns, and highlights new research directions also discussed in the paper