9,934 research outputs found
Benchmarking News Recommendations in a Living Lab
Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSys’13 and then as campaign-style evaluation lab NEWSREEL at CLEF’14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems
Benchmarking news recommendations: the CLEF NewsREEL use case
The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to evaluate and optimize news recommender algorithms. The goal is to create an algorithm that is able to generate news items that users would click, respecting a strict time constraint. The lab challenges participants to compete in either a "living lab" (Task 1) or perform an evaluation that replays recorded streams (Task 2). In this report, we discuss the objectives and challenges of the NewsREEL lab, summarize last year's campaign and outline the main research challenges that can be addressed by participating in NewsREEL 2016
CLEF NewsREEL 2016: Comparing Multi-Dimensional Offline and Online Evaluation of News Recommender Systems
Running in its third year at CLEF, NewsREEL challenged participants
to develop news recommendation algorithms and have them benchmarked in
an online (Task 1) and offline setting (Task 2), respectively. This paper provides
an overview of the NewsREEL scenario, outlines this year’s campaign, presents
results of both tasks, and discusses the approaches of participating teams. Moreover,
it overviews ideas on living lab evaluation that have been presented as part
of a “New Ideas” track at the conference in Portugal. Presented results illustrate
potentials for multi-dimensional evaluation of recommendation algorithms in
a living lab and simulation based evaluation setting
CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems
The CLEF NewsREEL challenge allows researchers to evaluate news
recommendation algorithms both online (NewsREEL Live) and offline (News-
REEL Replay). Compared with the previous year NewsREEL challenged participants
with a higher volume of messages and new news portals. In the 2017
edition of the CLEF NewsREEL challenge a wide variety of new approaches have
been implemented ranging from the use of existing machine learning frameworks,
to ensemble methods to the use of deep neural networks. This paper gives an
overview over the implemented approaches and discusses the evaluation results.
In addition, the main results of Living Lab and the Replay task are explained
Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform
In the CLEF NEWSREEL lab, participants are invited to evaluate news recommendation techniques in real-time by providing news recommendations to actual users that visit commercial news portals to satisfy their information needs. A central role within this lab is the communication between participants and the users. This is enabled by The Open Recommendation Platform (ORP), a web-based platform which distributes users' impressions of news articles to the participants and returns their recommendations to the readers. In this demo, we illustrate the platform and show how requests are handled to provide relevant news articles in real-time
Real-Time Recommendation of Streamed Data
This tutorial addressed two trending topics in the field of recommender systems research, namely A/B testing and real-time recommendations of streamed data. Focusing on the news domain, participants learned how to benchmark the performance of stream-based recommendation algorithms in a live recommender system and in a simulated environment
Capturing User Interests for Content-based Recommendations
Nowadays, most recommender systems provide recommendations
by either exploiting feedback given by similar users, referred to as
collaborative filtering, or by identifying items with similar properties,
referred to as content-based recommendation. Focusing on the
latter, this keynote presents various examples and case studies that
illustrate both strengths and weaknesses of content-based recommendatio
Users' reading habits in online news portals
The aim of this study is to survey reading habits of users of an online news portal. The assumption motivating this study is that insight into the reading habits of users can be helpful to design better news recommendation systems. We estimated the transition probabilities that users who read an article of one news category will move to read an article of another (not necessarily distinct) news category. For this, we analyzed the users' click behavior within plista data set. Key findings are the popularity of category local, loyalty of readers to the same category, observing similar results when addressing enforced click streams, and the case that click behavior is highly influenced by the news category
The Big Picture on Small Screens Delivering Acceptable Video Quality in Mobile TV
Mobile TV viewers can change the viewing distance and (on some devices) scale the picture to their preferred viewing ratio, trading off size for angular resolution. We investigated optimal trade-offs between size and resolution through a series of studies. Participants selected their preferred size and rated the acceptability of the visual experience on a 200ppi device at a 4: 3 aspect ratio. They preferred viewing ratios similar to living room TV setups regardless of the much lower resolution: at a minimum 14 pixels per degree. While traveling on trains people required videos with a height larger than 35mm
- …