Ordinary Search Engine Users Carrying Out Complex Search Tasks

Web search engines have become the dominant tools for finding information on the Internet. Due to their popularity, users apply them to a wide range of search needs, from simple look-ups to rather complex information tasks. This paper presents the results of a study to investigate the characteristics of these complex information needs in the context of Web search engines. The aim of the study is to find out more about (1) what makes complex search tasks distinct from simple tasks and if it is possible to find simple measures for describing their complexity, (2) if search success for a task can be predicted by means of unique measures, and (3) if successful searchers show a different behavior than unsuccessful ones. The study includes 60 people who carried out a set of 12 search tasks with current commercial search engines. Their behavior was logged with the Search-Logger tool. The results confirm that complex tasks show significantly different characteristics than simple tasks. Yet it seems to be difficult to distinguish successful from unsuccessful search behaviors. Good searchers can be differentiated from bad searchers by means of measurable parameters. The implications of these findings for search engine vendors are discussed.Comment: 60 page

On the adequacy of current empirical evaluations of formal models of categorization

Categorization is one of the fundamental building blocks of cognition, and the study of categorization is notable for the extent to which formal modeling has been a central and influential component of research. However, the field has seen a proliferation of noncomplementary models with little consensus on the relative adequacy of these accounts. Progress in assessing the relative adequacy of formal categorization models has, to date, been limited because (a) formal model comparisons are narrow in the number of models and phenomena considered and (b) models do not often clearly define their explanatory scope. Progress is further hampered by the practice of fitting models with arbitrarily variable parameters to each data set independently. Reviewing examples of good practice in the literature, we conclude that model comparisons are most fruitful when relative adequacy is assessed by comparing well-defined models on the basis of the number and proportion of irreversible, ordinal, penetrable successes (principles of minimal flexibility, breadth, good-enough precision, maximal simplicity, and psychological focus)

Understanding “influence”: An exploratory study of academics’ process of knowledge construction through iterative and interactive information seeking

The motivation for this study is to better understand the searching and sensemaking processes undertaken to solve exploratory tasks for which people lack pre-existing frames. To investigate people’s strategies for that type of task, we focused on “influence” tasks because, although they appear to be unfamiliar, they arise in much academic discourse, at least tacitly. This qualitative study reports the process undertaken by academics of different levels of seniority to complete exploratory search tasks that involved identifying influential members of their academic community and “rising stars, ” and to identify similar roles in an unfamiliar academic community. 11 think-aloud sessions followed by semi-structured interviews were conducted to investigate the role of specific and general domain expertise in the process of information seeking and knowledge construction. Academics defined and completed the task through an iterative and interactive process of seeking and sensemaking, during which they constructed an understanding of their communities and determined qualities of “being influential”. Elements of the Data/Frame Theory of Sensemaking (Klein et al., 2007) were used as sensitising theoretical constructs. The study shows that both external and internal knowledge resources are essential to define a starting point or frame, make and support decisions, and experience satisfaction. Ill-defined or non-existent initial frames may cause unsubstantial or arbitrary decisions, and feelings of uncertainty and lack of confidence

Information Search Patterns in Complex Tasks

acceptedVersionPeer reviewe

Evaluation of Rich and Explicit Feedback for Exploratory Search

A user’s goals and interests during an exploratory search task are often ambiguous and complex. When engaging with new documents, people regularly use annotations to help better understand them and make them their own. These annotations can also pro- vide rich information to gain insight into a reader’s interests. In particular, it is possible to use highlights as a richer form of feed- back compared to traditional document-level relevance feedback. We first show that this form of feedback leads to improvements in document retrieval in exploratory search tasks with simulated users when compared to relevance feedback. We then present an evaluation platform which will allow us to understand the retrieval performance, user experience, and behavioral characteristics of human subjects using highlights as feedback. Finally, we propose an experimental design with human subjects. We hope that our experimental findings will help improve current simulated user evaluations for such systems

Investigating the Influence of Subgoals on Learning During Search

Search-as-learning research has emphasized the need to better support searchers when learning about complex topics online. Prior work in the learning sciences has shown that effective self-regulated learning (SRL), in which goals are a central function, is critical to improving learning outcomes. This dissertation investigates the influence of subgoals on learning during search. Two conditions were investigated: Subgoals and NoSubgoals. In the Subgoals condition, a tool called the Subgoal Manager was used to help searchers to develop specific subgoals associated with an overall learning-oriented search task. The influence of subgoals is explored along four dimensions: (1) learning outcomes; (2) searcher perceptions; (3) search behaviors; and (4) SRL processes. Learning outcomes were measured with two assessments, an established multiple-choice conceptual knowledge test and an open-ended summary of learning. Learning assessments were administered immediately after search and one week after search to capture learning retention. A qualitative analysis was conducted to identify the percentage of true statements on open-ended learning assessments. A think-aloud protocol was used to capture SRL processes. A second qualitative analysis was conducted to categorize SRL processes from think-aloud comments and behaviors during the search session. Findings from the dissertation suggest that subgoals improved learning during search. Additionally, it seems that subgoals helped participants to better retain what was learned one week later. Findings also suggest that SRL processes of participants in the Subgoals condition were more frequent and more diverse. SRL processes that were explicitly supported by the Subgoal Manager seemed to be more frequent in the Subgoals condition as well as SRL processes that were not explicitly supported.Doctor of Philosoph

Improving the effectiveness and efficiency of web-based search tasks for policy workers

We adapt previous literature on search tasks for developing a domain-specific search engine that supports the search tasks of policy workers. To characterise the search tasks we conducted two rounds of interviews with policy workers at the municipality of Utrecht, and found that they face different challenges depending on the complexity of the task. During simple tasks, policy workers face information overload and time pressures, especially during web-based searches. For complex tasks, users prefer finding domain experts within their organisation to obtain the necessary information, which requires a different type of search functionality. To support simple tasks, we developed a web search engine that indexes web pages from authoritative sources only. We tested the hypothesis that users prefer expert search over web search for complex tasks and found that supporting complex tasks requires integrating functionality that enables finding internal experts within the broader web search engine. We constructed representative tasks to evaluate the proposed system’s effectiveness and efficiency, and found that it improved user performance. The search functionality developed could be standardised for use by policy workers in different municipalities within the Netherlands

Task complexity affects information use : a questionnaire study in city administration

Journal has a creative commons license: CC BY-NC-ND 3.0