How users assess web pages for information-seeking

In this paper, we investigate the criteria used by online searchers when assessing the relevance of web pages for information-seeking tasks. Twenty four participants were given three tasks each, and indicated the features of web pages which they employed when deciding about the usefulness of the pages in relation to the tasks. These tasks were presented within the context of a simulated work-task situation. We investigated the relative utility of features identified by participants (web page content,structure and quality), and how the importance of these features is affected by the type of information-seeking task performed and the stage of the search. The results of this study provide a set of criteria used by searchers to decide about the utility of web pages for different types of tasks. Such criteria can have implications for the design of systems that use or recommend web pages

Time Pressure and System Delays in Information Search

We report preliminary results of the impact of time pres- sure and system delays on search behavior from a laboratory study with forty-three participants. To induce time pres- sure, we randomly assigned half of our study participants to a treatment condition where they were only allowed five minutes to search for each of four ad-hoc search topics. The other half of the participants were given no task time limits. For half of participants’ search tasks (n=2), five second de- lays were introduced after queries were submitted and SERP results were clicked. Results showed that participants in the time pressure condition queried at a significantly higher rate, viewed significantly fewer documents per query, had significantly shallower hover and view depths, and spent sig- nificantly less time examining documents and SERPs. We found few significant differences in search behavior for sys- tem delay or interaction effects between time pressure and system delay. These initial results show time pressure has a significant impact on search behavior and suggest the de- sign of search interfaces and features that support people who are searching under time pressure

Seven years of INEX interactive retrieval experiments – lessons and challenges

This paper summarizes a major effort in interactive search investigation, the INEX i-track, a collective effort run over a seven-year period. We present the experimental conditions, report some of the findings of the participating groups, and examine the challenges posed by this kind of collective experimental effort

Document Word Clouds: Visualising Web Documents as Tag Clouds to Aid Users in Relevance Decisions

Περιέχει το πλήρες κείμενοInformation Retrieval systems spend a great effort on determining the significant terms in a document. When, instead, a user is looking at a document he cannot benefit from such information. He has to read the text to understand which words are important. In this paper we take a look at the idea of enhancing the perception of web documents with visualisation techniques borrowed from the tag clouds of Web 2.0. Highlighting the important words in a document by using a larger font size allows to get a quick impression of the relevant concepts in a text. As this process does not depend on a user query it can also be used for explorative search. A user study showed, that already simple TF-IDF values used as notion of word importance helped the users to decide quicker, whether or not a document is relevant to a topic

Users and Assessors in the Context of INEX: Are Relevance Dimensions Relevant?

The main aspects of XML retrieval are identified by analysing and comparing the following two behaviours: the behaviour of the assessor when judging the relevance of returned document components; and the behaviour of users when interacting with components of XML documents. We argue that the two INEX relevance dimensions, Exhaustivity and Specificity, are not orthogonal dimensions; indeed, an empirical analysis of each dimension reveals that the grades of the two dimensions are correlated to each other. By analysing the level of agreement between the assessor and the users, we aim at identifying the best units of retrieval. The results of our analysis show that the highest level of agreement is on highly relevant and on non-relevant document components, suggesting that only the end points of the INEX 10-point relevance scale are perceived in the same way by both the assessor and the users. We propose a new definition of relevance for XML retrieval and argue that its corresponding relevance scale would be a better choice for INEX

Evaluating the usability and usefulness of a digital library

Purpose - System usability and system usefulness are interdependent properties of system interaction, which in combination, determine system satisfaction and usage. Often approached separately, or in the case of digital libraries, often focused upon usability, there is emerging consensus among the research community for their unified treatment and research attention. However, a key challenge is to identify, both respectively and relatively, what to measure and how, compounded by concerns regarding common understanding of usability measures, and associated calls for more valid and complete measures within integrated and comprehensive models. The purpose of this paper is to address this challenge. Design/methodology/approach - Identified key usability and usefulness attributes and associated measures, compiled an integrated measurement framework, identified a suitable methodological approach for application of the framework, and conducted a pilot study on an interactive search system developed by a Health Service as part of their e-library service. Findings - Effectiveness, efficiency, aesthetic appearance, terminology, navigation, and learnability are key attributes of system usability; and relevance, reliability, and currency key attributes of system usefulness. There are shared aspects to several of these attributes, but each is also sufficiently unique to preserve its respective validity. They can be combined as part of a multi-method approach to system evaluation. Research limitations/implications - Pilot study has demonstrated that usability and usefulness can be readily combined, and that questionnaire and observation are valid multi-method approaches, but further research is called for under a variety of conditions, with further combinations of methods, and larger samples. Originality/value - This paper provides an integrated measurement framework, derived from the goal, question, metric paradigm, which provides a relatively comprehensive and representative set of system usability and system usefulness attributes and associated measures, which could be adapted and further refined on a case-by-case basis

Source Evaluation and Information Literacy: Findings from a Study on Science Websites

An essential component of information literacy is the evaluation of information resources. Integral to evaluation are users’ judgments about which web sources might prove reliable when learning about a particular topic. Past website quality studies have used research methods that involved asking participants to recall quality factors without the benefit of concurrent web searching. Users in this study evaluated websites during live searching on the “open” web to determine the quality factors they valued and how these relate to gaining knowledge about a particular topic – genetically modified (GM) food. Two weeks later, participants answered questions about the websites they visited and what they had learned via an email survey. The participants then reported factors that allowed them to remember a website or the information contained within it. The effect of the quality evaluation on memory for a particular resource is examined and its relationship to information literacy is explored

Cognitive and Social Factors Affecting the Use of Wikipedia and Information Seeking

Wikipedia, the free online encyclopedia, is the preferred choice among resources used by college students to meet their research needs. However, Wikipedia has been criticized for its low information quality, lack of accountability, inconsistency, and vulnerability to vandalism. Despite the warnings and concerns voiced by academia, online learning tools such as Wikipedia will continue their rise as major learning resource in today's classroom. Using a sample of 184 college students, the study proposed theoretical models to test the effects of internal beliefs, motivations, and social influences on Wikipedia use and information-seeking, and further empirically tested those models. The findings of this study suggested that Wikipedia use is driven by internal belief and peer pressure, whereas information-seeking is influenced by belief, motivation, and subjective norms. The implications of the findings for the research and practice are discussed. Wikipédia, l'encyclopédie gratuite en ligne, est le choix préféré des étudiants ayant besoin de faire une recherche. Cependant, Wikipédia a été critiqué pour la faible qualité de ses informations, son manque d’imputabilité, son incohérence et sa vulnérabilité au vandalisme. En dépit des mises en garde et des inquiétudes exprimées par le milieu universitaire, l’importance en salle de classe des outils d'apprentissage en ligne tels que Wikipédia continuera de croître. Sur la base d’un échantillon de 184 étudiants de niveau collégial, l'étude a proposé des modèles théoriques pour tester les effets des croyances personnelles, des motivations et des influences sociales sur l'utilisation de Wikipédia et sur la recherche d'information. Ces modèles ont ensuite été testés empiriquement. Les résultats de cette étude suggèrent que l'utilisation de Wikipédia est motivée par les croyances personnelles et la pression exercée par les camarades, alors que la recherche d'information est influencée par les croyances, la motivation et des normes subjectives. L’article se termine par une discussion des implications de ces résultats pour la recherche et la pratique