Evaluating sources of implicit feedback for web search

This dissertation investigated several important issues in using implicit feedback techniques to assist searchers with difficulties in formulating effective search strategies. The study focused on examining the relationship between types of behavioral evidence that can be captured from Web searches and searchers’ interests. Web search cases which involved underspecification of information needs at the beginning and modification of search strategies during the search process were collected and reviewed by human analysts (reference librarians) who tried to infer searchers’ interests from behavioral traces. Analysts’ rationales for making the inferences were elicited and analyzed with the focus on understanding what evidence was used to support the inferences and how it was used. The analysis revealed the complexities and nuances in using behavioral evidence for implicit feedback and led to the proposal of an implicit feedback model for Web search that bridged previous studies on behavioral evidence and implicit feedback measures. A new level of analysis termed an analytical lens emerged from the data and provides a road map for future research on this topic. The study also put forward design recommendations for implicit feedback systems based on the signals that analysts identified and the rules that they used in making inferences

Models and Algorithms for Understanding and Supporting Learning Goals in Information Retrieval

While search technology is widely used for learning-oriented information needs, the results provided by popular services such as Web search engines are optimized primarily for generic relevance, not effective learning outcomes. As a result, the typical information trail that a user must follow while searching to achieve a learning goal may be an inefficient one, possibly involving unnecessarily difficult content, or material that is irrelevant to actual learning progress relative to a user's existing knowledge. My work addresses these problems through multiple studies where various models and frameworks are developed and tested to support particular dimensions of search as learning. Empirical analysis of these studies through user studies demonstrate promising results and provide a solid foundation for further work. The earliest work we focused on centered on developing a framework and algorithms to support vocabulary learning objectives in a Web document context. The proposed framework incorporates user information, topic information and effort constraints to provide a desirable combination of personalized and efficient (by word length) learning experience. Our user studies demonstrate the effectiveness of our framework against a strong commercial baseline's (Google search) results in both short- and long-term assessment. While topic-specific content features (such as frequency of subtopic occurrences) naturally play a role in influencing learning outcomes, stylistic and structural features of the documents themselves may also play a role. Using such features we construct robust regression models that show strong predictive strength for multiple measures of learning outcomes. We also show early evidence that regression models trained on one dataset of search as learning can show strong test-set predictions on an independent dataset of search as learning, suggesting a certain degree of generalizability of stylistic content features. The models developed in my work are designed to be as generalizable, scalable and efficient as possible to make it easier for practitioners in the field to improve how people use search engines for learning. Finally, we investigate how gaze-tracking and automatic question generation could be used to scale a form of active learning to arbitrary text material. Our results show promising potential for incorporating interactive learning experiences in arbitrary text documents on the Web. A major theme in these studies centers on understanding and improving how people learn when using Web search engines. We also put specific emphasis on long-term learning outcomes and demonstrate that our models and frameworks actually yield sustainable knowledge gains, both for passive and interactive learning. Taken together, these research studies provide a solid foundation for multiple promising directions in exploring search as learning.PHDInformationUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155065/1/rmsyed_1.pd

Comparing methods for finding search sessions on a specified topic: A double case study

Users searching for different topics in a collection may show distinct search patterns. To analyze search behavior of users searching for a specific topic, we need to retrieve the sessions containing this topic. In this paper, we compare different topic representations and approaches to find topic-specific sessions. We conduct our research in a double case study of two topics, World War II and feminism, using search logs of a historical newspaper collection. We evaluate the results using manually created ground truths of over 600 sessions per topic. The two case studies show similar results: The query-based methods yield high precision, at the expense of recall. The document-based methods find more sessions, at the expense of precision. In both approaches, precision improves significantly by manually curating the topic representations. This study demonstrates how different methods to find sessions containing specific topics can be applied by digital humanities scholars and practitioners

Inferring implicit relevance from physiological signals

Ongoing growth in data availability and consumption has meant users are increasingly faced with the challenge of distilling relevant information from an abundance of noise. Overcoming this information overload can be particularly difficult in situations such as intelligence analysis, which involves subjectivity, ambiguity, or risky social implications. Highly automated solutions are often inadequate, therefore new methods are needed for augmenting existing analysis techniques to support user decision making. This project investigated the potential for deep learning to infer the occurrence of implicit relevance assessments from users' biometrics. Internal cognitive processes manifest involuntarily within physiological signals, and are often accompanied by 'gut feelings' of intuition. Quantifying unconscious mental processes during relevance appraisal may be a useful tool during decision making by offering an element of objectivity to an inherently subjective situation. Advances in wearable or non-contact sensors have made recording these signals more accessible, whilst advances in artificial intelligence and deep learning have enhanced the discovery of latent patterns within complex data. Together, these techniques might make it possible to transform tacit knowledge into codified knowledge which can be shared. A series of user studies recorded eye gaze movements, pupillary responses, electrodermal activity, heart rate variability, and skin temperature data from participants as they completed a binary relevance assessment task. Participants were asked to explicitly identify which of 40 short-text documents were relevant to an assigned topic. Investigations found this physiological data to contain detectable cues corresponding with relevance judgements. Random forests and artificial neural networks trained on features derived from the signals were able to produce inferences with moderate correlations with the participants' explicit relevance decisions. Several deep learning algorithms trained on the entire physiological time series data were generally unable to surpass the performance of feature-based methods, and instead produced inferences with low correlations with participants' explicit personal truths. Overall, pupillary responses, eye gaze movements, and electrodermal activity offered the most discriminative power, with additional physiological data providing diminishing or adverse returns. Finally, a conceptual design for a decision support system is used to discuss social implications and practicalities of quantifying implicit relevance using deep learning techniques. Potential benefits included assisting with introspection and collaborative assessment, however quantifying intrinsically unknowable concepts using personal data and abstruse artificial intelligence techniques were argued to pose incommensurate risks and challenges. Deep learning techniques therefore have the potential for inferring implicit relevance in information-rich environments, but are not yet fit for purpose. Several avenues worthy of further research are outlined

Just-in-time Information Interfaces: A new Paradigm for Information Discovery and Exploration

We live in a time of increasing information overload. Described as “a byproduct of the lack of maturity of the information age” (Spira & Goldes, 2007), information overload can be painful, and harm our concentration - the resulting choice overload impacts out decision-making abilities. Given the problem of information overload, and the unsatisfying nature of human-information interaction using traditional browsing or keyword-based search, this research investigates how the design of just-in-time information services can improve the user experience of goal-driven interactions with information. This thesis explores the design of just-in-time information services through the iterative development of two strands of high-level prototypes (FMI and KnowDis). I custombuilt both prototype systems for the respective evaluations, which have then been conducted as part of a series of lab-based eye-tracking studies (FMI) as well as two field studies (KnowDis). The lab-based eye-tracking studies were conducted with 100 participants measuring task performance, user satisfaction, and gaze behaviour. The lab studies found that the FMI prototype design did improve the performance aspect of the user experience for all participants and improved the usability aspect of the user experience for novice participants. However, the FMI prototype design seemed to be less effective and usable for expert participants. Two field studies were conducted as part of a two-year research collaboration, which lasted a total of 10 weeks and involved approximately 70 knowledge workers overall from across the globe. As part of those field studies, 46 semi-structured interviews were also conducted. The field studies found that the KnowDis prototype design did improve the user experience for participants overall by making work-related information search more efficient. However, while the KnowDis prototype design was useful for some knowledge workers and even integrated seamlessly into their day-to-day work, it appeared to be less useful and even distracting to others

Ephemeral relevance and user activities in a search session

We study relevance judgment and user activities in a search session. We focus on ephemeral relevance—a contextual measurement regarding the amount of useful information a searcher acquired from a clicked result at a particular time—and two primary types of search activities—query reformulation and click. The purpose of the study is both explanatory and practical. First, we examine the influence of different factors on ephemeral relevance and user activities in a search session. Second, we leverage short-term search history and implicit feedback in a session to predict ephemeral relevance and future search activities. The main findings include: 1. As a contextual usefulness measurement, ephemeral relevance differs from both topical relevance judgment and context-independent usefulness assessment. We show ephemeral relevance significantly relates to a wide range of factors, including topical relevance, novelty, understandability, reliability, effort spent, and search task. The difference between ephemeral relevance and context-independent usefulness assessment is linked to judgment criteria, novelty, effort spent, and changes in user’s perceptions of a search result. 2. Ephemeral relevance can be predicted accurately using implicit feedback signals without any manual explicit judgments. We generalize existing implicit feedback methods from using information related to a single result to those based on user activities in a whole session, achieving a correlation as high as 0.5 between the predicted and real judgments. 3. We show choices of word changes in query reformulation and click decisions significantly relate to recent search history, such as the contents and effectiveness of previous search queries, the contents of the results viewed and clicked in previous searches, etc. 4. Leveraging short-term search history in a session and other information, we can predict word changes in query reformulation and click decisions with different levels of accuracies. These findings help disclose and explain the dynamics of relevance and user activities in a search session. The developed techniques provide effective support for developing interactive IR systems