Finding relevant documents using top ranking sentences: an evaluation of two alternative schemes

In this paper we present an evaluation of techniques that are designed to encourage web searchers to interact more with the results of a web search. Two specific techniques are examined: the presentation of sentences that highly match the searcher's query and the use of implicit evidence. Implicit evidence is evidence captured from the searcher's interaction with the retrieval results and is used to automatically update the display. Our evaluation concentrates on the effectiveness and subject perception of these techniques. The results show, with statistical significance, that the techniques are effective and efficient for information seeking

A study of interface support mechanisms for interactive information retrieval

Advances in search technology have meant that search systems can now offer assistance to users beyond simply retrieving a set of documents. For example, search systems are now capable of inferring user interests by observing their interaction, offering suggestions about what terms could be used in a query, or reorganizing search results to make exploration of retrieved material more effective. When providing new search functionality, system designers must decide how the new functionality should be offered to users. One major choice is between (a) offering automatic features that require little human input, but give little human control; or (b) interactive features which allow human control over how the feature is used, but often give little guidance over how the feature should be best used. This article presents a study in which we empirically investigate the issue of control by presenting an experiment in which participants were asked to interact with three experimental systems that vary the degree of control they had in creating queries, indicating which results are relevant in making search decisions. We use our findings to discuss why and how the control users want over search decisions can vary depending on the nature of the decisions and the impact of those decisions on the user's search

A study of factors affecting the utility of implicit relevance feedback

Implicit relevance feedback (IRF) is the process by which a search system unobtrusively gathers evidence on searcher interests from their interaction with the system. IRF is a new method of gathering information on user interest and, if IRF is to be used in operational IR systems, it is important to establish when it performs well and when it performs poorly. In this paper we investigate how the use and effectiveness of IRF is affected by three factors: search task complexity, the search experience of the user and the stage in the search. Our findings suggest that all three of these factors contribute to the utility of IRF

Evaluating implicit feedback models using searcher simulations

In this article we describe an evaluation of relevance feedback (RF) algorithms using searcher simulations. Since these algorithms select additional terms for query modification based on inferences made from searcher interaction, not on relevance information searchers explicitly provide (as in traditional RF), we refer to them as implicit feedback models. We introduce six different models that base their decisions on the interactions of searchers and use different approaches to rank query modification terms. The aim of this article is to determine which of these models should be used to assist searchers in the systems we develop. To evaluate these models we used searcher simulations that afforded us more control over the experimental conditions than experiments with human subjects and allowed complex interaction to be modeled without the need for costly human experimentation. The simulation-based evaluation methodology measures how well the models learn the distribution of terms across relevant documents (i.e., learn what information is relevant) and how well they improve search effectiveness (i.e., create effective search queries). Our findings show that an implicit feedback model based on Jeffrey's rule of conditioning outperformed other models under investigation

A simulated study of implicit feedback models

In this paper we report on a study of implicit feedback models for unobtrusively tracking the information needs of searchers. Such models use relevance information gathered from searcher interaction and can be a potential substitute for explicit relevance feedback. We introduce a variety of implicit feedback models designed to enhance an Information Retrieval (IR) system's representation of searchers' information needs. To benchmark their performance we use a simulation-centric evaluation methodology that measures how well each model learns relevance and improves search effectiveness. The results show that a heuristic-based binary voting model and one based on Jeffrey's rule of conditioning [5] outperform the other models under investigation

Going Beyond Relevance: Role of effort in Information Retrieval

The primary focus of Information Retrieval (IR) systems has been to optimize for Relevance. Existing approaches to rank documents or evaluate IR systems does not account for “user effort”. Currently, judges only determine whether the information provided in a given document would satisfy the underlying information need in a query. The current mechanism of obtaining relevance judgments does not account for time and effort that an end user must put forth to consume its content. While a judge may spend a lot of time assessing a document, an impatient user may not devote the same amount of time and effort to consume its content. This problem is exacerbated on smaller devices like mobile. While on mobile or tablets, with limited interaction, users may not put in too much effort in finding information. This thesis characterizes and incorporates effort in Information Retrieval. Comparison of explicit and implicit relevance judgments across several datasets reveals that certain documents are marked relevant by the judges but are of low utility to an end user. Experiments indicate that document-level effort features can reliably predict the mismatch between dwell time and judging time of documents. Explicit and preference-based judgments were collected to determine which factors associated with effort agreed the most with user satisfaction. The ability to locate relevant information or findability was found to be in highest agreement with preference judgments. Findability judgments were also gathered to study the association of different annotator, query or document related properties with effort judgments. We also investigate how can existing systems be optimized for relevance and effort. Finally, we investigate the role of effort on smaller devices with the help of cost-benefit models