Study of Relevance and Effort across Devices

Abstract

Relevance judgements are essential for designing information retrieval systems. Traditionally, judgements have been judgements have been gathered via desktop interfaces. However, with the rise in popularity of smaller devices for information access, it has become imperative to investigate whether desktop based judgements are different from judgements gathered using mobiles. Recently, user effort and document usefulness have also emerged as important dimensions to optimize and evaluate information retrieval systems. Since existing work is limited to desktops, it remains to be seen how these judgements are affected by user’s search device. In this paper, we address these shortcomings by collecting and analyzing relevance, usefulness and effort judgements on mobiles and desktops. Analysis of these judgements indicates that high agreement rate between desktop and mobile judges for relevance, followed by usefulness and findability. We also found that desktop judges are likely to spend more time and examine documents in greater depth on non-relevant/notuseful/difficult documents compared to mobile judges. Based on our findings, we suggest that relevance judgements should be gathered via desktops and effort judgements should be collected on each device independently