Learning by example : training users with high-quality query suggestions

The queries submitted by users to search engines often poorly describe their information needs and represent a potential bottleneck in the system. In this paper we investigate to what extent it is possible to aid users in learning how to formulate better queries by providing examples of high-quality queries interactively during a number of search sessions. By means of several controlled user studies we collect quantitative and qualitative evidence that shows: (1) study participants are able to identify and abstract qualities of queries that make them highly effective, (2) after seeing high-quality example queries participants are able to themselves create queries that are highly effective, and, (3) those queries look similar to expert queries as defined in the literature. We conclude by discussing what the findings mean in the context of the design of interactive search systems

Recall, Robustness, and Lexicographic Evaluation

Researchers use recall to evaluate rankings across a variety of retrieval, recommendation, and machine learning tasks. While there is a colloquial interpretation of recall in set-based evaluation, the research community is far from a principled understanding of recall metrics for rankings. The lack of principled understanding of or motivation for recall has resulted in criticism amongst the retrieval community that recall is useful as a measure at all. In this light, we reflect on the measurement of recall in rankings from a formal perspective. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation. First, we formally define `recall-orientation' as sensitivity to movement of the bottom-ranked relevant item. Second, we analyze our concept of recall orientation from the perspective of robustness with respect to possible searchers and content providers. Finally, we extend this conceptual and theoretical treatment of recall by developing a practical preference-based evaluation method based on lexicographic comparison. Through extensive empirical analysis across 17 TREC tracks, we establish that our new evaluation method, lexirecall, is correlated with existing recall metrics and exhibits substantially higher discriminative power and stability in the presence of missing labels. Our conceptual, theoretical, and empirical analysis substantially deepens our understanding of recall and motivates its adoption through connections to robustness and fairness.Comment: Under revie

Resonance and the experience of relevance

In this article, I propose the concept of resonance as a useful one for describing what it means to experience relevance. Based on an extensive interdisciplinary review, I provide a novel framework that presents resonance as a spectrum of experience with a multitude of outcomes ranging from a sense of harmony and coherence to life transformation. I argue that resonance has different properties to the more traditional interpretation of relevance and provides a better system of explanation of what it means to experience relevance. I show how traditional approaches to relevance and resonance work in a complementary fashion and outline how resonance may present distinct new lines of research into relevance theory

Moderating effects of self-perceived knowledge in a relevance assessment task : an EEG study

Relevance assessment, a crucial Human-computer Information Retrieval (HCIR) aspect, denotes how well retrieved information meets the user’s information need (IN). Recently, user-centred research benefited from the employment of brain imaging, which contributed to our understanding of relevance assessment and associated cognitive processes. However, the effect of contextual aspects, such as the searcher’s self-perceived knowledge (SPK) on relevance assessment and its underlying neurocognitive processes, has not been studied. This work investigates the impact of users’ SPK about a topic (i.e. ‘knowledgeable’ vs. ‘not knowledgeable’) on relevance assessments (i.e. ‘relevant’ vs. ‘non-relevant’). To do so, using electroencephalography (EEG), we measured the neural activity of twenty-five participants while they provided relevance assessments during the Question and Answering (Q/A) Task. In the analysis, we considered the effects of SPK and specifically how it modulates the brain activity underpinning relevance judgements. Data-driven analysis revealed significant event-related potential differences (P300/CPP, N400, LPC), which were modulated by searchers’ SPK in the context of relevance assessment. We speculate that SPK affects distinct cognitive processes associated with attention, semantic integration and categorisation, memory, and decision formation that underpin relevance assessment formation. Our findings are an important step toward a better understanding of the role users’ SPK plays during relevance assessment

Evaluación de la actividad científica en ciencia de la información a partir de indicadores bibliométricos y altmétricos

La presente investigación es un análisis de la producción científica en Ciencia de la Informacion (CI), fundamentada en el contexto epistemológico e histórico de la disciplina, para identificar las tendencias de uso de la información en plataformas de publicación formales e informales. A partir de la implementación de indicadores bibliométricos e indicadores alternativos, se pretende establecer. ¿Como la integración de indicadores altimétricos en la evaluación científica, posibilita la identificación de tendencias en la investigación disciplinar? Y si es valido afirmar, que la altmetría es una herramienta confiable y útil para la evaluación de los dominios científicos. Se toma como referente la producción visible en Web of Science durante el periodo 2012- 2016, para identificar las dinámicas científicas de investigación en la CI, a partir de una muestra de 1224 registros en los cuales se utilizan indicadores bibliometricos de producción, citación o impacto e indicadores altimétricos recuperados de las plataformas ResearchGate (RG) y Plum Analytics (PlumX). Los resultados evidencian que los indicadores alternativos aun están en periodo de desarrollo y necesitan normalización; de lo cual se concluye, que la evaluación científica requiere la complementación de modelos métricos clásicos junto a métricas alternativas que permitan identificar las dinámicas sociales y de comunicación que se generan en la comunidad científica más allá del impacto y la citación.This research is an analysis of the scientific activity in Information Science (CI), based on the epistemological and historical context of the discipline, to identify trends in the use of information in formal and informal publishing platforms. Based on the implementation of bibliometric iand alternative indicators, it is intended to establish: How does the integration of altmetric indicators in scientific evaluation make it possible to identify trends in disciplinary research? And, if it is valid to say that altmetrics is a reliable and useful tool for the scientific evaluation of scientific domains. Visible production in Web of Science during the 2012-2016 period is taken as a reference to identify the scientific dynamics of research in the CI, from a sample of 1224 records in which bibliometric indicators of production, citation or impact and altmetric indicators recovered from the ResearchGate (RG) and Plum Analytics (PlumX) platforms are used. The results show that the alternative indicators are still under development and need to be standardized; from which it is concluded that scientific evaluation requires the complementing of classical metric models with alternative metrics that allow identifying the social and communication dynamics generated in the scientific community beyond the impact and citation.Profesional en Ciencia de la Información - Bibliotecólogo (a)Pregrad

Exploiting user signals and stochastic models to improve information retrieval systems and evaluation

The leitmotiv throughout this thesis is represented by IR evaluation. We discuss different issues related to effectiveness measures and novel solutions that we propose to address these challenges. We start by providing a formal definition of utility-oriented measurement of retrieval effectiveness, based on the representational theory of measurement. The proposed theoretical framework contributes to a better understanding of the problem complexities, separating those due to the inherent problems in comparing systems, from those due to the expected numerical properties of measures. We then propose AWARE, a probabilistic framework for dealing with the noise and inconsistencies introduced when relevance labels are gathered with multiple crowd assessors. By modeling relevance judgements and crowd assessors as sources of uncertainty, we directly combine the performance measures computed on the ground-truth generated by each crowd assessor, instead of adopting a classification technique to merge the labels at pool level. Finally, we investigate evaluation measures able to account for user signals. We propose a new user model based on Markov chains, that allows the user to scan the result list with many degrees of freedom. We exploit this Markovian model in order to inject user models into precision, defining a new family of evaluation measures, and we embed this model as objective function of an LtR algorithm to improve system performances

Popularität und Relevanz in der Suche

In diesem Open-Access-Buch wird mithilfe eines großangelegten Online-Experiments untersucht, wie sich die Anzeige von Zitationen oder Downloads auf die Relevanzbewertung in akademischen Suchsystemenauswirkt. Bei der Suche nach Informationen verwenden Menschen diverse Kriterien, anhand derer sie die Relevanz der Suchergebnisse bewerten. In diesem Buch wird erstmals eine systematische Übersicht über die Einflüsse im Prozess der Relevanzbewertung von Suchergebnissen in akademischen Suchsystemen aufgezeigt. Zudem wird ein anspruchsvolles und komplexes Methodenframework zur experimentellen Untersuchung von Relevanzkriterien vorgestellt. Dieses eignet sich für die weitergehende Erforschung von Relevanzkriterien im informationswissenschaftlichen Bereich

Relevance behaviour in TREC

The purpose of this paper is to examine how various types of TREC data can be used to better understand relevance and serve as test-bed for exploring relevance. The author proposes that there are many interesting studies that can be performed on the TREC data collections that are not directly related to evaluating systems but to learning more about human judgements of information and relevance and that these studies can provide useful research questions for other types of investigation