Query-biased web page summarisation: a task-oriented evaluation

We present a system that offers a new way of assessing web document relevance and new approach to the web-based evaluation of such a system. Provisionally named WebDocSum, the system is a query-biased web page summariser that aims to provide an alternative to the short, irrelevant abstracts typical of many web search result lists. Based on an initial evaluation the system appears to be more useful in helping users gauge document relevance than the traditional ranked titles/abstracts approach

Retrieval through explanation : an abductive inference approach to relevance feedback

Relevance feedback techniques are designed to automatically improve a system's representation of a query by using documents the user has marked as relevant. However, traditional relevance feedback models suffer from a number of limitations that restrict their potential in supporting information seeking. One of the major limitations of relevance feedback is that it does not incorporate behavioural aspects of information seeking - how and why users assess relevance. We propose that relevance feedback should be viewed as a process of explanation and demonstrate how this limitation of relevance feedback techniques can be overcome by a theory of relevance feedback based on abductive inference

Complex search task: how to make a phone safe for a child

There are many factors in task design that might make it ‘complex’: having multiple components, having multiple cross-dependent components, tasks that involve comparison, evaluation, estimation, or learning. In this paper, we discuss a case study of a complex task we may consider to be highly natural, a common concern for many people, and one that ‘should’ have a clear answer, but doesn’t: how do you make a phone safe for a child. For this question, there is a lot of opinion online, many possibilities for actions, many variations in hardware and software, but ultimately no one clear and correct answer for everyday phone users. We found very little objective behaviours that separated people in terms of performance but instead have begun to identify some successful tactics that are not directly linked to domain knowledge

Realism in empirical Interactive Information Retrieval (IIR) studies: A systematic review protocol

This is the initial protocol for a systematic review. Our systematic review aims to identify empirical studies of people conducting interactive searches in which they initiate a request for information from one or more information retrieval systems to complete researcher-assigned tasks. A subset of these studies that explicitly discuss the realism of element aspect of the study (e.g., realism of researcher-assigned tasks, IR systems, etc.) will be analyzed in-depth

Evaluating the usability and usefulness of a digital library

Purpose - System usability and system usefulness are interdependent properties of system interaction, which in combination, determine system satisfaction and usage. Often approached separately, or in the case of digital libraries, often focused upon usability, there is emerging consensus among the research community for their unified treatment and research attention. However, a key challenge is to identify, both respectively and relatively, what to measure and how, compounded by concerns regarding common understanding of usability measures, and associated calls for more valid and complete measures within integrated and comprehensive models. The purpose of this paper is to address this challenge. Design/methodology/approach - Identified key usability and usefulness attributes and associated measures, compiled an integrated measurement framework, identified a suitable methodological approach for application of the framework, and conducted a pilot study on an interactive search system developed by a Health Service as part of their e-library service. Findings - Effectiveness, efficiency, aesthetic appearance, terminology, navigation, and learnability are key attributes of system usability; and relevance, reliability, and currency key attributes of system usefulness. There are shared aspects to several of these attributes, but each is also sufficiently unique to preserve its respective validity. They can be combined as part of a multi-method approach to system evaluation. Research limitations/implications - Pilot study has demonstrated that usability and usefulness can be readily combined, and that questionnaire and observation are valid multi-method approaches, but further research is called for under a variety of conditions, with further combinations of methods, and larger samples. Originality/value - This paper provides an integrated measurement framework, derived from the goal, question, metric paradigm, which provides a relatively comprehensive and representative set of system usability and system usefulness attributes and associated measures, which could be adapted and further refined on a case-by-case basis

Geographic information retrieval in a mobile environment: evaluating the needs of mobile individuals

This paper describes research that aims to define the information needs of mobile individuals, to implement a mobile information system that can satisfy those needs, and finally to evaluate the performance of that system with end-users. First a review of the emerging discipline of geographic information retrieval (GIR) is presented as background to the more specific issue of mobile information retrieval. Following this, a user needs study is described evaluating the requirements of potential users of a mobile information system; the study finds that there is a strong geographic component to users' information needs. Next, four geographic post-query filters are described which attempt to represent the region of space associated with an individual's query made at some specific spatial location. These filters are spatial proximity (distance in space), temporal proximity (travel time), speed-heading prediction surfaces (likelihood of visiting locations) and visibility (locations that can be seen). Two of these filters — spatial proximity and speed-heading prediction surfaces — are implemented in a mobile information system and subsequently evaluated with users in an outdoor setting. The results of evaluation suggest that retrieved information to which post-query geographic filters have been applied is considered more relevant than unfiltered information, and that users find information sorted by spatial proximity to be more relevant than that sorted by a prediction surface of likely future locations. The paper closes with a discussion of the wider implications of these results for developers of mobile information systems and location-based services