A Distributed and Accountable Approach to Offline Recommender Systems Evaluation

Different software tools have been developed with the purpose of performing offline evaluations of recommender systems. However, the results obtained with these tools may be not directly comparable because of subtle differences in the experimental protocols and metrics. Furthermore, it is difficult to analyze in the same experimental conditions several algorithms without disclosing their implementation details. For these reasons, we introduce RecLab, an open source software for evaluating recommender systems in a distributed fashion. By relying on consolidated web protocols, we created RESTful APIs for training and querying recommenders remotely. In this way, it is possible to easily integrate into the same toolkit algorithms realized with different technologies. In details, the experimenter can perform an evaluation by simply visiting a web interface provided by RecLab. The framework will then interact with all the selected recommenders and it will compute and display a comprehensive set of measures, each representing a different metric. The results of all experiments are permanently stored and publicly available in order to support accountability and comparative analyses.Comment: REVEAL 2018 Workshop on Offline Evaluation for Recommender System

Goal-based structuring in a recommender systems

Recommender systems help people to find information that is interesting to them. However, current recommendation techniques only address the user's short-term and long-term interests, not their immediate interests. This paper describes a method to structure information (with or without using recommendations) taking into account the users' immediate interests: a goal-based structuring method. Goal-based structuring is based on the fact that people experience certain gratifications from using information, which should match with their goals. An experiment using an electronic TV guide shows that structuring information using a goal-based structure makes it easier for users to find interesting information, especially if the goals are used explicitly; this is independent of whether recommendations are used or not. It also shows that goal-based structuring has more influence on how easy it is for users to find interesting information than recommendations

Algorithms Aside: Recommendation as the Lens of Life

In this position paper, we take the experimental approach of putting algorithms aside, and reflect on what recommenders would be for people if they were not tied to technology. By looking at some of the shortcomings that current recommenders have fallen into and discussing their limitations from a human point of view, we ask the question: if freed from all limitations, what should, and what could, RecSys be? We then turn to the idea that life itself is the best recommender system, and that people themselves are the query. By looking at how life brings people in contact with options that suit their needs or match their preferences, we hope to shed further light on what current RecSys could be doing better. Finally, we look at the forms that RecSys could take in the future. By formulating our vision beyond the reach of usual considerations and current limitations, including business models, algorithms, data sets, and evaluation methodologies, we attempt to arrive at fresh conclusions that may inspire the next steps taken by the community of researchers working on RecSys

Benchmarking News Recommendations in a Living Lab

Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSys’13 and then as campaign-style evaluation lab NEWSREEL at CLEF’14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems

Trust based collaborative filtering

k-nearest neighbour (kNN) collaborative filtering (CF), the widely successful algorithm supporting recommender systems, attempts to relieve the problem of information overload by generating predicted ratings for items users have not expressed their opinions about; to do so, each predicted rating is computed based on ratings given by like-minded individuals. Like-mindedness, or similarity-based recommendation, is the cause of a variety of problems that plague recommender systems. An alternative view of the problem, based on trust, offers the potential to address many of the previous limiations in CF. In this work we present a varation of kNN, the trusted k-nearest recommenders (or kNR) algorithm, which allows users to learn who and how much to trust one another by evaluating the utility of the rating information they receive. This method redefines the way CF is performed, and while avoiding some of the pitfalls that similarity-based CF is prone to, outperforms the basic similarity-based methods in terms of prediction accuracy

Privacy-preserving scheme for mobile ad hoc networks.

This paper proposes a decentralized trust establishment protocol for mobile ad hoc networks (MANETs), where nodes establish security associations. In order to achieve privacy and security, we use homomorphic encryption and polynomial intersection so as to find the intersection of two sets. The first set represents a list of recommenders of the initiator and the second set is a list of trusted recommenders of the responder. The intersection of the sets represents a list of nodes that recommend the first node and their recommendations are trusted by the second node. In our experimental results we show that our scheme is effective even if there are 30 trusted nodes

People-Powered Music: Using User-Generated Tags and Structure in Recommendations

Music recommenders often rely on experts to classify song facets like genre and mood, but user-generated folksonomies hold some advantages over expert classifications—folksonomies can reflect the same real-world vocabularies and categorizations that end users employ. We present an approach for using crowd-sourced common sense knowledge to structure user-generated music tags into a folksonomy, and describe how to use this approach to make music recommendations. We then empirically evaluate our “people-powered” structured content recommender against a more traditional recommender. Our results show that participants slightly preferred the unstructured recommender, rating more of its recommendations as “perfect” than they did for our approach. An exploration of the reasons behind participants’ ratings revealed that users behaved differently when tagging songs than when evaluating recommendations, and we discuss the implications of our results for future tagging and recommendation approaches