11,211 research outputs found
Reducing Offline Evaluation Bias in Recommendation Systems
Recommendation systems have been integrated into the majority of large online
systems. They tailor those systems to individual users by filtering and ranking
information according to user profiles. This adaptation process influences the
way users interact with the system and, as a consequence, increases the
difficulty of evaluating a recommendation algorithm with historical data (via
offline evaluation). This paper analyses this evaluation bias and proposes a
simple item weighting solution that reduces its impact. The efficiency of the
proposed solution is evaluated on real world data extracted from Viadeo
professional social network.Comment: 23rd annual Belgian-Dutch Conference on Machine Learning (Benelearn
2014), Bruxelles : Belgium (2014
Reducing offline evaluation bias of collaborative filtering algorithms
Recommendation systems have been integrated into the majority of large online
systems to filter and rank information according to user profiles. It thus
influences the way users interact with the system and, as a consequence, bias
the evaluation of the performance of a recommendation algorithm computed using
historical data (via offline evaluation). This paper presents a new application
of a weighted offline evaluation to reduce this bias for collaborative
filtering algorithms.Comment: European Symposium on Artificial Neural Networks, Computational
Intelligence and Machine Learning (ESANN), Apr 2015, Bruges, Belgium.
pp.137-142, 2015, Proceedings of the 23-th European Symposium on Artificial
Neural Networks, Computational Intelligence and Machine Learning (ESANN 2015
Improving offline evaluation of contextual bandit algorithms via bootstrapping techniques
In many recommendation applications such as news recommendation, the items
that can be rec- ommended come and go at a very fast pace. This is a challenge
for recommender systems (RS) to face this setting. Online learning algorithms
seem to be the most straight forward solution. The contextual bandit framework
was introduced for that very purpose. In general the evaluation of a RS is a
critical issue. Live evaluation is of- ten avoided due to the potential loss of
revenue, hence the need for offline evaluation methods. Two options are
available. Model based meth- ods are biased by nature and are thus difficult to
trust when used alone. Data driven methods are therefore what we consider here.
Evaluat- ing online learning algorithms with past data is not simple but some
methods exist in the litera- ture. Nonetheless their accuracy is not satisfac-
tory mainly due to their mechanism of data re- jection that only allow the
exploitation of a small fraction of the data. We precisely address this issue
in this paper. After highlighting the limita- tions of the previous methods, we
present a new method, based on bootstrapping techniques. This new method comes
with two important improve- ments: it is much more accurate and it provides a
measure of quality of its estimation. The latter is a highly desirable property
in order to minimize the risks entailed by putting online a RS for the first
time. We provide both theoretical and ex- perimental proofs of its superiority
compared to state-of-the-art methods, as well as an analysis of the convergence
of the measure of quality
Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
Contextual bandit algorithms have become popular for online recommendation
systems such as Digg, Yahoo! Buzz, and news recommendation in general.
\emph{Offline} evaluation of the effectiveness of new algorithms in these
applications is critical for protecting online user experiences but very
challenging due to their "partial-label" nature. Common practice is to create a
simulator which simulates the online environment for the problem at hand and
then run an algorithm against this simulator. However, creating simulator
itself is often difficult and modeling bias is usually unavoidably introduced.
In this paper, we introduce a \emph{replay} methodology for contextual bandit
algorithm evaluation. Different from simulator-based approaches, our method is
completely data-driven and very easy to adapt to different applications. More
importantly, our method can provide provably unbiased evaluations. Our
empirical results on a large-scale news article recommendation dataset
collected from Yahoo! Front Page conform well with our theoretical results.
Furthermore, comparisons between our offline replay and online bucket
evaluation of several contextual bandit algorithms show accuracy and
effectiveness of our offline evaluation method.Comment: 10 pages, 7 figures, revised from the published version at the WSDM
2011 conferenc
Evaluation of recommender systems in streaming environments
Evaluation of recommender systems is typically done with finite datasets.
This means that conventional evaluation methodologies are only applicable in
offline experiments, where data and models are stationary. However, in real
world systems, user feedback is continuously generated, at unpredictable rates.
Given this setting, one important issue is how to evaluate algorithms in such a
streaming data environment. In this paper we propose a prequential evaluation
protocol for recommender systems, suitable for streaming data environments, but
also applicable in stationary settings. Using this protocol we are able to
monitor the evolution of algorithms' accuracy over time. Furthermore, we are
able to perform reliable comparative assessments of algorithms by computing
significance tests over a sliding window. We argue that besides being suitable
for streaming data, prequential evaluation allows the detection of phenomena
that would otherwise remain unnoticed in the evaluation of both offline and
online recommender systems.Comment: Workshop on 'Recommender Systems Evaluation: Dimensions and Design'
(REDD 2014), held in conjunction with RecSys 2014. October 10, 2014, Silicon
Valley, United State
How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility
Recommendation systems are ubiquitous and impact many domains; they have the
potential to influence product consumption, individuals' perceptions of the
world, and life-altering decisions. These systems are often evaluated or
trained with data from users already exposed to algorithmic recommendations;
this creates a pernicious feedback loop. Using simulations, we demonstrate how
using data confounded in this way homogenizes user behavior without increasing
utility
- …