28 research outputs found
Evaluation of recommender systems in streaming environments
Evaluation of recommender systems is typically done with finite datasets.
This means that conventional evaluation methodologies are only applicable in
offline experiments, where data and models are stationary. However, in real
world systems, user feedback is continuously generated, at unpredictable rates.
Given this setting, one important issue is how to evaluate algorithms in such a
streaming data environment. In this paper we propose a prequential evaluation
protocol for recommender systems, suitable for streaming data environments, but
also applicable in stationary settings. Using this protocol we are able to
monitor the evolution of algorithms' accuracy over time. Furthermore, we are
able to perform reliable comparative assessments of algorithms by computing
significance tests over a sliding window. We argue that besides being suitable
for streaming data, prequential evaluation allows the detection of phenomena
that would otherwise remain unnoticed in the evaluation of both offline and
online recommender systems.Comment: Workshop on 'Recommender Systems Evaluation: Dimensions and Design'
(REDD 2014), held in conjunction with RecSys 2014. October 10, 2014, Silicon
Valley, United State
A Batch Learning Framework for Scalable Personalized Ranking
In designing personalized ranking algorithms, it is desirable to encourage a
high precision at the top of the ranked list. Existing methods either seek a
smooth convex surrogate for a non-smooth ranking metric or directly modify
updating procedures to encourage top accuracy. In this work we point out that
these methods do not scale well to a large-scale setting, and this is partly
due to the inaccurate pointwise or pairwise rank estimation. We propose a new
framework for personalized ranking. It uses batch-based rank estimators and
smooth rank-sensitive loss functions. This new batch learning framework leads
to more stable and accurate rank approximations compared to previous work.
Moreover, it enables explicit use of parallel computation to speed up training.
We conduct empirical evaluation on three item recommendation tasks. Our method
shows consistent accuracy improvements over state-of-the-art methods.
Additionally, we observe time efficiency advantages when data scale increases.Comment: AAAI 2018, Feb 2-7, New Orleans, US
Slow Learners are Fast
Online learning algorithms have impressive convergence properties when it
comes to risk minimization and convex games on very large problems. However,
they are inherently sequential in their design which prevents them from taking
advantage of modern multi-core architectures. In this paper we prove that
online learning with delayed updates converges well, thereby facilitating
parallel online learning.Comment: Extended version of conference paper - NIPS 200