Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform

In the CLEF NEWSREEL lab, participants are invited to evaluate news recommendation techniques in real-time by providing news recommendations to actual users that visit commercial news portals to satisfy their information needs. A central role within this lab is the communication between participants and the users. This is enabled by The Open Recommendation Platform (ORP), a web-based platform which distributes users' impressions of news articles to the participants and returns their recommendations to the readers. In this demo, we illustrate the platform and show how requests are handled to provide relevant news articles in real-time

Investigating Correlations of Automatically Extracted Multimodal Features and Lecture Video Quality

Ranking and recommendation of multimedia content such as videos is usually realized with respect to the relevance to a user query. However, for lecture videos and MOOCs (Massive Open Online Courses) it is not only required to retrieve relevant videos, but particularly to find lecture videos of high quality that facilitate learning, for instance, independent of the video's or speaker's popularity. Thus, metadata about a lecture video's quality are crucial features for learning contexts, e.g., lecture video recommendation in search as learning scenarios. In this paper, we investigate whether automatically extracted features are correlated to quality aspects of a video. A set of scholarly videos from a Mass Open Online Course (MOOC) is analyzed regarding audio, linguistic, and visual features. Furthermore, a set of cross-modal features is proposed which are derived by combining transcripts, audio, video, and slide content. A user study is conducted to investigate the correlations between the automatically collected features and human ratings of quality aspects of a lecture video. Finally, the impact of our features on the knowledge gain of the participants is discussed

Assessing learning outcomes in web searching: A comparison of tasks and query strategies

Users make frequent use of Web search for learning-related tasks, but little is known about how different Web search interaction strategies affect outcomes for learning-oriented tasks, or what implicit or explicit indicators could reliably be used to assess search-related learning on the Web. We describe a lab-based user study in which we investigated potential indicators of learning in web searching, effective query strategies for learning, and the relationship between search behavior and learning outcomes. Using questionnaires, analysis of written responses to knowledge prompts, and search log data, we found that searchers’ perceived learning outcomes closely matched their actual learning outcomes; that the amount searchers wrote in post-search questionnaire responses was highly correlated with their cognitive learning scores; and that the time searchers spent per document while searching was also highly and consistently correlated with higher-level cognitive learning scores. We also found that of the three query interaction conditions we applied, an intrinsically diverse presentation of results was associated with the highest percentage of users achieving combined factual and conceptual knowledge gains. Our study provides deeper insight into which aspects of search interaction are most effective for supporting superior learning outcomes, and the difficult problem of how learning may be assessed effectively during Web search.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/145733/1/Collins-Thompson Rieh CHIIR 2016.pd

Evaluating Generative Ad Hoc Information Retrieval

Recent advances in large language models have enabled the development of viable generative information retrieval systems. A generative retrieval system returns a grounded generated text in response to an information need instead of the traditional document ranking. Quantifying the utility of these types of responses is essential for evaluating generative retrieval systems. As the established evaluation methodology for ranking-based ad hoc retrieval may seem unsuitable for generative retrieval, new approaches for reliable, repeatable, and reproducible experimentation are required. In this paper, we survey the relevant information retrieval and natural language processing literature, identify search tasks and system architectures in generative retrieval, develop a corresponding user model, and study its operationalization. This theoretical analysis provides a foundation and new insights for the evaluation of generative ad hoc retrieval systems.Comment: 14 pages, 5 figures, 1 tabl

A comparison of primary and secondary relevance judgements for real-life topics

The notion of relevance is fundamental to the field of Information Retrieval. Within the field a generally accepted conception of relevance as inherently subjective has emerged, with an individual’s assessment of relevance influenced by numerous contextual factors. In this paper we present a user study that examines in detail the differences between primary and secondary assessors on a set of “real-world” topics which were gathered specifically for the work. By gathering topics which are representative of the staff and students at a major university, at a particular point in time, we aim to explore differences between primary and secondary relevance judgements for real-life search tasks. Findings suggest that while secondary assessors may find the assessment task challenging in various ways (they generally possess less interest and knowledge in secondary topics and take longer to assess documents), agreement between primary and secondary assessors is high

A framework for evaluating automatic indexing or classification in the context of retrieval

Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. While some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real-life systems and situations. The paper reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single “gold standard” method when evaluating indexing and retrieval and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on indexing, classification and approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard; evaluating the quality of computer-assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance

Health consumers' knowledge learning in online health information seeking

With the increasing awareness of health consumers as active information seekers, the past decade has witnessed a shifting research interest from a physician-centered paradigm to a consumer-centered paradigm. Online health information seeking (OHIS) has become pervasive, with critical impacts on consumers' health. However, the inherent complexity and the uniqueness of health tasks pose new challenges to consumers in OHIS, such as a lack of adequate knowledge to formulate queries and evaluate the online resources with various qualities. OHIS is, by nature, a learning-oriented behavior, and knowledge learning is a critical component and outcome of consumers' OHIS. On the other hand, studies in the area of search as learning (SAL) have demonstrated that learning is a common phenomenon in the information-seeking process. However, the existing studies in OHIS mainly concentrated on viewing consumers' domain knowledge as a fixed value, even though consumers are involved in the knowledge learning in the OHIS. Therefore, this dissertation proposes a conceptual framework of health information search as learning (HearSAL) by linking the related models and prior studies from the two areas — OHIS and SAL — and conducts a systematic study to understand what, how, and how well health consumers can search and learn in online health information seeking, particularly for three increasing levels of learning objectives: Understand, Analyze and Evaluate. Two representative health consumer groups, laypeople and cancer patients, are targeted in this dissertation study because they share the common issue of facing barriers in searching and learning in OHIS, yet they are different due to prior topic knowledge, learning duration, and learning expectation. Following the conceptual framework HearSAL, four sub-studies are conducted with emphasis on different dimensions of health consumers' search as learning in OHIS, including the following: Study 1: a user study with laypeople that examines the method dimension (e.g., search behaviors and source selections); Study 2: an analysis of an ovarian cancer online health community that reveals the information dimension (e.g., types and amount of information); Study 3: interviews with laypeople; and Study 4: interviews with ovarian cancer patients and caregivers. The two complementary interviews highlight the outcomes of OHIS. Major results demonstrate that, (1) health consumers’ SAL behaviors and sources vary by different levels of learning objectives, and the variation is affected by the severity of health conditions; (2) Analyze is the most prevalent learning objective in the online health community, while the amount of informational support is the highest in the Evaluate level; (3) Though consumers’ prior knowledge of the Understand level is the highest, compared to higher levels, consumers still tend to achieve the most knowledge increase in the Understand level of learning; and (4) Receiving more informational support drives consumers to increase the level of learning objectives. This dissertation makes empirical, practical, theoretical and methodological contributions. The empirical studies of laypeople and ovarian cancer patients provide a deeper insight into health consumers' SAL behavior and performance in today's web environment. Based on the empirical results, practical implications are proposed for designing consumer-centered health information systems, which facilitate seeking and enhance learning. Finally, the HearSAL framework and its application in this study can serve as a theoretical and methodological basis for future explorations

Curiosity driven search experiences

Casual-Leisure Search describes any behaviour that allows people to express and satisfy hedonistic needs rather than information needs as part of the information-seeking process. For example, individuals who search their social media universe for hours after a long day at work may do so out of curiosity, to relax or for fun (e.g. exploring for the experience). Studies have shown that classical information seeking (IS) and interactive information retrieval models (IIR) have failed to represent them because they were created observing people in work related scenarios, and assuming that search is always a rational decision making process and with an extrinsic utilitarian value. The research described in this PhD work investigates IIR from the perspective of the psychological curiosity and leisure information seeking behaviour. Traditional search engines focus the user experience on satisfying users with topically relevant information (i.e. quick lookup search and then moving on), but they are limited supporting the discovery of unknown information because they fail to entice and engage users exploration as proxy to seek enjoyment both in leisure and work scenarios. The research described increases understanding of the role that curiosity plays in IIR and investigates the merits of incorporating the characteristics and function of human curiosity in the design of IIR systems. The research is grounded by the theoretical understanding of how human curiosity works. A review of appropriate psychological curiosity literature offers a means to critique existing IIR tools and a basis from which to start designing novel curiosity driven search tools. In the first experimental work, this research compared IIR behaviour between a standard query response paradigm and a curiosity driven search map prototype using social media content, and attempts to learn lessons from the behaviour that people show in everyday casual-leisure search scenarios. In the second experiment, this research contrast IIR behaviour between standard query-response paradigm and a curious adaptation of query-response paradigm using search notifications or recommendations for news reading in a social media leisure search scenario. The tools are evaluated to determine the usefulness of incorporating curiosity in the design of IIR systems, to learn about the effect in user engagement, how users exploration is increase when motivated by a hedonistic need, and then elaborate a set of design recommendations to enhance the search experience in leisure scenarios

Evaluation Methodologies in Information Retrieval Dagstuhl Seminar 13441

This paper reports on the Evaluation Methodologies in Information Retrieval Seminar1 held from 27 October to 1 November 2013 at the Schloss Dagstuhl - Leibniz Center for Informatics that is a world-wide renowned venue for informatics where scientists come together to exchange their knowledge and to discuss their research findings. The seminar was attended by 42 participants from thirteen different countries, including a large number of established researchers as well as some some promising young researchers, and also practitioners from industry