2,148 research outputs found
From Counter-intuitive Observations to a Fresh Look at Recommender System
Recently, a few papers report counter-intuitive observations made from
experiments on recommender system (RecSys). One observation is that users who
spend more time and users who have many interactions with a recommendation
system receive poorer recommendations. Another observation is that models
trained by using only the more recent parts of a dataset show significant
performance improvement. In this opinion paper, we interpret these
counter-intuitive observations from two perspectives. First, the observations
are made with respect to the global timeline of user-item interactions. Second,
the observations are considered counter-intuitive because they contradict our
expectation on a recommender: the more interactions a user has, the higher
chance that the recommender better learns the user preference. For the first
perspective, we discuss the importance of the global timeline by using the
simplest baseline Popularity as a starting point. We answer two questions: (i)
why the simplest model popularity is often ill-defined in academic research?
and (ii) why the popularity baseline is evaluated in this way? The questions
lead to a detailed discussion on the data leakage issue in many offline
evaluations. As the result, model accuracies reported in many academic papers
are less meaningful and incomparable. For the second perspective, we try to
answer two more questions: (i) why models trained by using only the more recent
parts of data demonstrate better performance? and (ii) why more interactions
from users lead to poorer recommendations? The key to both questions is user
preference modeling. We then propose to have a fresh look at RecSys. We discuss
how to conduct more practical offline evaluations and possible ways to
effectively model user preferences. The discussion and opinions in this paper
are on top-N recommendation only, not on rating prediction.Comment: 11 pages, 5 figure
- …