Typically, three main query reformulation types in sessions

are considered: generalization, specication, and drift. We show that given the full context of user interactions, repeat queries represent an important reformulation type which should also be addressed in session retrieval evaluation. We investigate dierent query reformulation patterns in logs from The European Library. Using an automatic classification for query reformulations, we found that the most frequent (and presumably the most important) reformulation pattern corresponds to repeat queries. We aim to nd possible explanations for repeat queries in sessions and try to uncover implications for session retrieval evaluation

Jones, Gareth J.F.

Leveling, Johannes

English

DCU Online Research Access Service

Same Query – Different Results?A Study of Repeat Queries in Search SessionsJohannes Leveling and Gareth J. F. JonesCentre for Next Generation Localisation (CNGL), School of ComputingDublin City University, Dublin 9, Ireland{jleveling, gjones}@computing.dcu.ieAbstract. Typically, three main query reformulation types in sessionsare considered: generalization, specification, and drift. We show thatgiven the full context of user interactions, repeat queries represent animportant reformulation type which should also be addressed in sessionretrieval evaluation. We investigate different query reformulation pat-terns in logs from The European Library. Using an automatic classifi-cation for query reformulations, we found that the most frequent (andpresumably the most important) reformulation pattern corresponds torepeat queries. We aim to find possible explanations for repeat queries insessions and try to uncover implications for session retrieval evaluation.1 IntroductionThere are two different approaches to information retrieval (IR) evaluation. Oneis the traditional Cranfield paradigm, where IR systems are evaluated indepen-dent of any search context, i.e. the same query will yield the same, reproducibleresults, regardless of individual users or search context. The other approach orig-inates from studies in human computer interaction, where evaluation is basedon user interactions with a system, which poses the challenge to design thesestudies so that reproducible results can be obtained. There is growing interest inbridging the gap between these approaches by investigating query reformulationsand evaluating IR in context (see, for example, the TREC session track [1]).In this paper we show that repeat queries should be considered as an impor-tant query reformulation pattern in session retrieval evaluation. Typically, themain categories of query reformulations include generalization, specification, anddrift [2, 1, 3]. To investigate the importance of different reformulation patterns,we perform an analysis of reformulation types in sessions. We base our analysis onuser interaction logs from The European Library (TEL). He et al. [3] and Jansenet al. [4] both propose similar algorithms for automatic classification of queryreformulations. We classify query reformulations following the latter algorithm,which involves examining common, added, deleted, and modified terms for twosuccessive query formulations. To our surprise, the automatic classification ofreformulations revealed that repeat queries are the most frequent reformulationpattern in sessions. Similar to the Excite logs [2], navigating multi-page resultlists appears to generate repeat queries in the TEL log. However, even if theseare discarded – as in our additional analysis – repeat queries still make up themost frequent reformulation pattern. This poses a major problem for session re-trieval evaluation because, to the best of our knowledge, evaluation experimentsand metrics do not explicitly address repeat queries. For example, the issue ofrepeat queries raises such questions as: should the same result list be returned tothe user for two identical successive queries in the same session? We aim to findpossible explanations for repeat queries in sessions and implications for sessionretrieval evaluation.2 Related WorkMost publications on repeat queries are concerned with reformulations in websearch and do not concern repeat queries within sessions. For example, Teevanet al. [5] found in an analysis of web log data that 40% of all queries are re-findqueries. He et al. [3] observe that the second most frequent type of reformulationin sessions is repeat queries, but they do not provide an explanation. (Note thatwe do not consider the most frequent type in their study, browsing, as queryreformulation). Sanderson and Dumais [6] report that 80% of repeat queries inweb search are navigational queries. They find that repetition of queries often oc-curs on the same day. The smallest time interval examined in their investigation(except for periodicity studies) is a single day.Repeat queries may have different causes: repeat queries in different sessionstypically correspond to a known-item search and occur often in web search [5].Repeat queries in sessions are unlikely to represent known-item search, due tothe different time frame (i.e. less than 30 minutes between actions), and areissued by the same user. Some log entries with repeated queries are generatedautomatically by a system when the user navigates the multi-page search results(e.g. Excite, see [2]) and does not actually enter a query, but clicks on a link.Huang and Efthimidiadis give an excellent overview of different query refor-mulation patterns in the literature [7]. In the session track at TREC 2010, thefocus was on three major patterns of reformulation [1]: 1. generalization (wherethe original results were too narrow and the reformulation into a more generalquery can be achieved by deleting a term), 2. specification (where the originalresults are too broad and the reformulation into a more specific query can beachieved by adding a term), and 3. drifting (where the reformulation is anotherquery aiming at a different aspect of the information need). So far, repeat querieshave not been considered in the session track.However, most of the related work does not focus on repeat queries in thesame session, but on repeated queries in different sessions (e.g. in web search)and by different users [5, 6].1 None provides possible explanations as to why thesemay occur within a session.1 Note that Spink et al. defined repeat queries as all multiple occurrences of the samequery that represent requests for multi-page viewing [2].Table 1. TEL 2009 action log statistics.# actions (total): 1866330# queries (total): 86981# sessions (total): 20325avg. # actions per session: 15.68avg. # queries per session: 4.283 Session AnalysisTEL Data. The analysis described in this paper is based on the TEL 2009 actionlogs (queries and user interactions), which were employed at LogCLEF 20092, atask for analyzing query logs. Table 1 shows statistics on the TEL log data.Session reconstruction. The TEL logs were preprocessed to reconstruct usersessions by grouping all actions with the same (unique) session ID together andsorting them by timestamp. We define a session as a consecutive sequence of alluser interactions, presuming the start of a new session if the time between con-secutive actions exceeds 30 minutes. The 30 minute time interval originates fromthe working definition of session reconstruction in previous editions of LogCLEF.Thus, sessions comprise actions such as changing the interface languages,selecting the target collection, submitting a query, and viewing results. Sessionsusing the TEL advanced search3, sessions containing non-English queries, andsessions with less than two queries were filtered out because we did not want toover-emphasize the importance of context changes (since switches between theadvanced and simple search are frequent) or language-specific aspects.Analysis of reformulations In the context of session retrieval, we only considerdirectly consecutive queries as potential repeat queries and disregard all othersimilar queries within a session as possible repeat queries. We applied the auto-matic classification approach described in Jansen et al. [4] to identify the mostimportant query reformulation patterns in sessions. We made only minor modifi-cations to the algorithm (see [4] for details), i.e. using the edit distance for singleword queries to determine if they are related and applying the classification onnormalized queries, after case folding, stopword removal, and stemming. Thesechanges were made to include spelling corrections, case folding, and morpholog-ical normalizations as query reformulations. Synonyms and abbreviations havenot been explicitly considered as reformulations due to the open domain of theTEL target collections. Table 2 shows results for the reformulation classification.The most surprising result is that repeat queries correspond to the most frequentquery reformulation pattern in sessions (i.e. not changing the query). So far, re-peat queries in session evaluations have generally been ignored, but they raisethe question as to why users should repeat queries within sessions.2 http://www.uni-hildesheim.de/logclef/3 TEL supports simple (keyword) search and advanced (structured) search, whichaims at different metadata fields (e.g. author, title) and employs Boolean operators.Table 2. Frequency of query reformulation patterns in sessions.Reformulation pattern Frequency# specifications: 1410# specifications+reformulation: 584# generalizations: 1196# generalizations+reformulation: 590# reformulations: 2072# content changes: 4608# repeat queries (with diff. context): 8942# empty queries: 386Σ 19788# repeat queries (including navigation): 50963Explaining repeat queries. One reason why repeat queries have been ignored inevaluation is the assumption that the same query should return the same results.However, if the full search context is considered, there are actually repeat queriesfor which different results are expected by the user, for example when a differenttarget collection is selected or language-specific settings are changed.There are several explanations why users repeat queries. Often, a part of thesearch context, which includes the viewed documents and the selected targetlanguage or collection, is changed. In web search, users typically do not changesearch options or choose different target collections. In general, repeat queries inweb search mostly occur in different sessions, where submitting identical queriesrepresents a known-item search [5], i.e. re-finding a result with a previous query.One explanation for repeat queries in the same session is that the user’s stateof mind changes after viewing result documents and he assumes that the systemkeeps track of his search findings and will return different results for the samequery. At least some users might expect that results change for the same query,because they are unfamiliar with a search system and their mental model of theinformation need changes after viewing some results. For example, log analysison the TEL logs has shown that users often modify queries regardless of queryprocessing steps, for example by adding or deleting stopwords, which are removedby the TEL system [8]. If this mismatch between actual and expected systembehavior exists – even for a small group of users – session retrieval evaluationmetrics should take the search context into account (e.g. by employing usermodels). For example, users would expect to find the same (or very similar)results when they submit the same query on the next day to re-find previousresults. When they submit the same query in the same session, they may bemore interested in result diversity and expect different results. Current sessionevaluation does not reflect this and ignores search context.One possible explanation for repeat queries is that users submit a query againsimply by mistake. However, the number of repeat queries is much too high toaccount for all of them by mistakes. An explanation found in web search researchis that users use previous queries as an “anchor point” to jump back to the firstresult page, before exploring other aspects of the topic. This explanation impliesa subsequent reformulation of the query so that each repeat should be followedby at least one reformulation on another aspect of the topic. This reformulation“meta-pattern” was not observed in the TEL logs. However, many users mightnavigate results and browse the same page or intermediate results several timesin a session.4 Implications for Session Retrieval EvaluationWe think that to build reusable test collections for session IR, more contextof queries in sessions has to be captured. The search context includes languagesettings, the selected target collection, viewed documents, and user interactionsother than queries (e.g. changing search options). The TEL log data alreadycaptures this information and provides additional metadata for evaluation (e.g.timestamps).We also think that realistic data is a necessary prerequisite for research onsession retrieval. So far, most (to our best knowledge: all) session retrieval modelsignore repeat queries in sessions, although they are the most frequent reformu-lation pattern in sessions.As part of the future work, we intend to carry out user studies to createreal logs and experiments on different log data, to see if the observed results arespecific to the log data used in this paper or can be observed in web search orin other domains such as enterprise search.AcknowledgementsThis research is supported by the Science Foundation Ireland (grant 07/CE/I1142)as part of the Centre for Next Generation Localisation (http://www.cngl.ie).References1. Kanoulas, E., Clough, P., Carterette, B., Sanderson, M.: Session track at TREC2010. In: SIGIR 2010 Workshop on Simulation of Interaction. (2010)2. Spink, A., Wolfram, D., Jansen, B.J., Saracevic, T.: Searching the web: The publicand their queries. JASIST 52(3) (2001) 226–2343. He, D., Go¨ker, A., Harper, D.J.: Combining evidence for automatic web sessionidentification. Information Processing & Management 38(5) (2002) 727–7424. Jansen, B.J., Booth, D., Spink, A.: Patterns of query reformulation during websearching. ASIS&T 60(7) (2009) 1358–13715. Teevan, J., Adar, E., Jones, R., Potts, M.A.S.: Information re-retrieval: Repeatquerie in Yahoo’s logs. In: SIGIR 2007. (2007) 151–1586. Sanderson, M., Dumais, S.T.: Examining repetition in user search behavior. In:ECIR 2007. (2007) 597–6047. Huang, J., Efthimiadis, E.N.: Analyzing and evaluating query reformulation strate-gies in web search logs. In: CIKM 2009. (2009) 77–868. Ghorab, M.R., Leveling, J., Zhou, D., Jones, G.J.F., Wade, V.: Identifying commonuser behaviour in multilingual search logs. In: CLEF 2009. (2010) 518–525

A.: Patterns of query reformulation during web searching.

Combining evidence for automatic web session identi

Efthimiadis, E.N.: Analyzing and evaluating query reformulation strategies in web search logs. In: CIKM

Identifying common user behaviour in multilingual search logs. In: CLEF

Information re-retrieval: Repeat querie in Yahoo's logs. In: SIGIR

S.T.: Examining repetition in user search behavior. In: ECIR

Searching the web: The public and their queries.

Session track at TREC

Same query - different results? A study of repeat queries in search sessions

Typically, three main query reformulation types in sessions\ud
are considered: generalization, specication, and drift. We show that given the full context of user interactions, repeat queries represent an important reformulation type which should also be addressed in session retrieval evaluation. We investigate dierent query reformulation patterns in logs from The European Library. Using an automatic classification for query reformulations, we found that the most frequent (and presumably the most important) reformulation pattern corresponds to repeat queries. We aim to nd possible explanations for repeat queries in sessions and try to uncover implications for session retrieval evaluation

Name not available

Typically, three main query reformulation types in sessions
are considered: generalization, specication, and drift. We show that given the full context of user interactions, repeat queries represent an important reformulation type which should also be addressed in session retrieval evaluation. We investigate dierent query reformulation patterns in logs from The European Library. Using an automatic classification for query reformulations, we found that the most frequent (and presumably the most important) reformulation pattern corresponds to repeat queries. We aim to nd possible explanations for repeat queries in sessions and try to uncover implications for session retrieval evaluation

Irish Universities

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.218.6428

Same query - different results? A study of repeat queries in search sessions

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Name not available

Irish Universities