8,968 research outputs found
Benchmarking News Recommendations in a Living Lab
Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSys’13 and then as campaign-style evaluation lab NEWSREEL at CLEF’14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems
CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems
The CLEF NewsREEL challenge allows researchers to evaluate news
recommendation algorithms both online (NewsREEL Live) and offline (News-
REEL Replay). Compared with the previous year NewsREEL challenged participants
with a higher volume of messages and new news portals. In the 2017
edition of the CLEF NewsREEL challenge a wide variety of new approaches have
been implemented ranging from the use of existing machine learning frameworks,
to ensemble methods to the use of deep neural networks. This paper gives an
overview over the implemented approaches and discusses the evaluation results.
In addition, the main results of Living Lab and the Replay task are explained
CLEF NewsREEL 2016: Comparing Multi-Dimensional Offline and Online Evaluation of News Recommender Systems
Running in its third year at CLEF, NewsREEL challenged participants
to develop news recommendation algorithms and have them benchmarked in
an online (Task 1) and offline setting (Task 2), respectively. This paper provides
an overview of the NewsREEL scenario, outlines this year’s campaign, presents
results of both tasks, and discusses the approaches of participating teams. Moreover,
it overviews ideas on living lab evaluation that have been presented as part
of a “New Ideas” track at the conference in Portugal. Presented results illustrate
potentials for multi-dimensional evaluation of recommendation algorithms in
a living lab and simulation based evaluation setting
Recommender Systems
The ongoing rapid expansion of the Internet greatly increases the necessity
of effective recommender systems for filtering the abundant information.
Extensive research for recommender systems is conducted by a broad range of
communities including social and computer scientists, physicists, and
interdisciplinary researchers. Despite substantial theoretical and practical
achievements, unification and comparison of different approaches are lacking,
which impedes further advances. In this article, we review recent developments
in recommender systems and discuss the major challenges. We compare and
evaluate available algorithms and examine their roles in the future
developments. In addition to algorithms, physical aspects are described to
illustrate macroscopic behavior of recommender systems. Potential impacts and
future directions are discussed. We emphasize that recommendation has a great
scientific depth and combines diverse research fields which makes it of
interests for physicists as well as interdisciplinary researchers.Comment: 97 pages, 20 figures (To appear in Physics Reports
Recommender systems and their ethical challenges
This article presents the first, systematic analysis of the ethical challenges posed by recommender systems through a literature review. The article identifies six areas of concern, and maps them onto a proposed taxonomy of different kinds of ethical impact. The analysis uncovers a gap in the literature: currently user-centred approaches do not consider the interests of a variety of other stakeholders—as opposed to just the receivers of a recommendation—in assessing the ethical impacts of a recommender system
- …