CORE
🇺🇦
make metadata, not war
Services
Services overview
Explore all CORE services
Access to raw data
API
Dataset
FastSync
Content discovery
Recommender
Discovery
OAI identifiers
OAI Resolver
Managing content
Dashboard
Bespoke contracts
Consultancy services
Support us
Support us
Membership
Sponsorship
Community governance
Advisory Board
Board of supporters
Research network
About
About us
Our mission
Team
Blog
FAQs
Contact us
Rating consistency is consistently underrated : An exploratory analysis of movie-tag rating inconsistency
Authors
Dorota Glowacka
Denis Kotkov
+4 more
Alexandr Maslov
Alan Medlar
Mats Neovius
Umesh Raj Satyal
Publication date
25 April 2022
Publisher
Association for Computing Machinery
Doi
Cite
Abstract
Publisher Copyright: © 2022 ACM.Content-based and hybrid recommender systems rely on item-tag ratings to make recommendations. An example of an item-tag rating is the degree to which the tag "comedy"applies to the movie "Back to the Future (1985)". Ratings are often generated by human annotators who can be inconsistent with one another. However, many recommender systems take item-tag ratings at face value, assuming them all to be equally valid. In this paper, we investigate the inconsistency of item-tag ratings together with contextual factors that could affect consistency in the movie domain. We conducted semi-structured interviews to identify potential reasons for rating inconsistency. Next, we used these reasons to design a survey, which we ran on Amazon Mechanical Turk. We collected 6,070 ratings from 665 annotators across 142 movies and 80 tags. Our analysis shows that ∼45% of ratings are inconsistent with the mode rating for a given movie-tag pair. We found that the single most important factor for rating inconsistency is the annotator's perceived ease of rating, suggesting that annotators are at least tacitly aware of the quality of their own ratings. We also found that subjective tags (e.g. "funny", "boring") are more inconsistent than objective tags (e.g. "robots", "aliens"), and are associated with lower tag familiarity and lower perceived ease of rating.Peer reviewe
Similar works
Full text
Open in the Core reader
Download PDF
Available Versions
Helsingin yliopiston digitaalinen arkisto
See this paper in CORE
Go to the repository landing page
Download from data provider
oai:helda.helsinki.fi:10138/35...
Last time updated on 12/03/2023