1 research outputs found
Challenges for Measuring Usefulness of Interactive IR Systems with Log-based Approaches
The usefulness evaluation model proposed by Cole et al. in 2009 [2] focuses
on the evaluation of interactive IR systems by their support towards the user's
overall goal, sub goals and tasks. This is a more human focus of the IR
evaluation process than with classical TREC-oriented studies and gives a more
holistic view on the IR evaluation process. However, yet there is no formal
framework how the usefulness model can be operationalized. Additionally, a lot
of information needed for the operationalization is only available in explicit
user studies where for example the overall goal and the tasks are prompted from
the users or are predefined. Measuring the usefulness of IR systems outside the
laboratory is a challenging task as most often only log data of user
interaction is available. But, an operationalization of the usefulness model
based on interaction data could be applied to diverse systems and evaluation
results would be comparable. In this article we discuss the challenges for
measuring the usefulness of IIR systems with log-based approaches.Comment: Proceedings of 1st International Workshop on System And User Centered
Evaluation Approaches in Interactive Information Retrieval at CHIIR 201