Kamps, J.

English

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)The significance of search, support for complex tasks, and searcher- awareinformation access systemsKamps, J.DOI10.4230/DagRep.3.10.92Publication date2013Document VersionFinal published versionPublished inDagstuhl ReportsLicenseCC BYLink to publicationCitation for published version (APA):Kamps, J. (2013). The significance of search, support for complex tasks, and searcher- awareinformation access systems. Dagstuhl Reports, 3(10), 122-123.https://doi.org/10.4230/DagRep.3.10.92General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 2023122 13441 – Evaluation Methodologies in Information Retrieval4.8 The Significance of Search, Support for Complex Tasks, andSearcher–aware Information Access SystemsJaap Kamps (University of Amsterdam, NL)License Creative Commons BY 3.0 Unported license© Jaap KampsThis abstract documents three loosely related challenges. The first challenge is the role andsignificance of the field in general. There are massive challenges in the way the informationavailable is changing in quantity and in character, and in the ways we create, publish, share,and use information in the always-online world. This urges us to keep ’reinventing search’and redefine the field of information retrieval and its key research problems and researchmethods. How do these changes affect the core questions we address in the field of IR andwhat sort of evidence do we need for addressing these questions? How can we factor thelarger scope and context into IR evaluation? It is interesting to consider a publication likeSalton’s “Developments in automatic text retrieval” published in Science in 1991. Salton(1991) is from before the Web happened and discusses all the basic IR aspects: retrievalmodels, indexing structures, but also hypertext, knowledge resources and semantic search.Articles like Salton (1991) still look surprisingly modern! This raises two question that areperhaps not unrelated: First, why hasn’t our research field changed in a dramatic way tosuit the revolutionary changes in the information environment. Second, why isn’t our fieldmaking a larger impact outside our field (Salton published 2 Science articles in 1991) giventhe dramatic increased role and importance of “search” nowadays.The second challenge is to work on information access tools that support complex tasks.That is, to build and evaluate information access tools that actively supports a searcher toarticulate a whole search task, and to interactively explore the results of every stage of theprocess. In the prolonged search session, how should we evaluate the overall effectivenessas well as the success at various stages? How can evaluation reflect the different goals ofeach stage? There is a striking difference in how we ask a person for information, givingcontext and articulating what we want and why, and how we communicate with currentsearch engines. Current search technology requires us to slice-and-dice our problem intoseveral queries and sub-queries, and laboriously combine the answers post hoc to solveour tasks. Combining different sources requires opening multiple windows or tabs, andcutting-and-pasting information between them. Current search engines may have reached alocal optimum for answering micro information needs with lighting speed. Supporting theoverall task opens up new ways to significantly advance our information access tools, bydevelop tools that are adapted to our overall tasks rather than have searchers adapt theirsearch tactics to the “things that work.”The third challenge is to make information access systems more informed about thesearcher. Can we make a retrieval system aware of the searcher’s stage in the informationseeking process, tailor the results to each stage, and guide the searcher through the overallprocess? How to evaluate the utility of this (accuracy of the prediction, usefulness of thesupport, etc)? Can we equate evaluation with observing preferred information interactionpatterns? A search session for a non- trivial search task consists of stages with differentsub-goals (e.g., problem identification) and specific search tactics (e.g., reading introductorytexts, familiarizing with terminology). Making a system aware of a searcher’s informationseeking stage has the potential to significantly improve the search experience. Searchersare stimulated to actively engage with the material, to get a grasp on the information needand articulate effective queries, to critically evaluate retrieved results, and to construct aMaristella Agosti, Norbert Fuhr, Elaine Toms, and Pertti Vakkari 123comprehensive answer. This may be of particularly great help for those searchers havingpoor information or media literacy. This is of obvious importance in many situations: e.g.,education, medical information, and search for topics “that matter”. Some special domains,such as patent search and evidence based practices in medicine, have clearly prescribed aparticular information seeking process in great detail. Here building a systems to support(and enforce) this process is of obvious value.4.9 Interaction, Measures and ModelsGianmaria Silvello (University of Padova, IT), Leif Azzopardi (University of Glasgow, GB),Charlie Clarke (University of Waterloo, CA), Matthias Hagen (Bauhaus–Universität Weimar,DE), Robert Villa (University of Sheffield, GB)License Creative Commons BY 3.0 Unported license© Gianmaria Silvello, Leif Azzopardi, Charlie Clarke, Matthias Hagen, and Robert VillaA common framework for user interaction models and a common framework in which toplace evaluation measures (i.e., the units of measurement) should be consistent but does notyet exist. Current measures are not comparable as the units used are not clearly definedin terms of real-world outcomes, and vary between measures. Since most measures encodesome form of user behaviour as an underlying user interaction model, having measures thatuse the same unit of measure would enable comparisons between different user interactionmodels across different systems. As well as making it possible to compare between measuresthemselves (opposed to viewing them independently in different units).4.9.1 MotivationThe main goal is to enable assessment of the performances of the system as a whole orspecific components in particular. For that we need a repeatable way to say that a system isbetter than another on a gain base (utility, usefulness, happiness, ...). Ideally, the effect ofuser attributes that are not salient to the evaluation itself should be minimized (e.g. “whatthe user had for breakfast”). The measures should be comparable; that is, defined using thesame units (i.e. gain, cost, or gain/cost). We would also like to be able to determine theeffects of the interface and interaction on the actual performance.4.9.2 Proposed solutionIntegrate the interaction with an IR interface into the measures, e.g. in a TREC- styleevaluation, individual IR systems may submit conventional ranked lists. Systems can thenbe evaluated based on different models of user interfaces or interactions. To extend TREC-style evaluations to accommodate more realistic interfaces, individual systems might submitresponses to a variety of user actions, which would then be evaluated across more complexand detailed interfaces and interaction models.One possible solution would be to decompose measures into components: Interactionmodel (I) (traditionally: when the user stops) Gain model (G) (traditionally: number ofviewed relevant docs) Cost model (C) (traditionally: number of viewed docs with unit costs)An evaluation measure could then be parameterized by the components as M(I,G,C).An interaction model might be characterized by a sequence of states and for each statesome specific interaction with the system taken; potentially depending on the intent and task13441

The significance of search, support for complex tasks, and searcher- aware information access systems

https://pure.uva.nl/ws/files/26699685/Kamps_significance.pdf

The significance of search, support for complex tasks, and searcher- aware information access systems

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications