Predicting Causes of Reformulation in Intelligent Assistants

Intelligent assistants (IAs) such as Siri and Cortana conversationally interact with users and execute a wide range of actions (e.g., searching the Web, setting alarms, and chatting). IAs can support these actions through the combination of various components such as automatic speech recognition, natural language understanding, and language generation. However, the complexity of these components hinders developers from determining which component causes an error. To remove this hindrance, we focus on reformulation, which is a useful signal of user dissatisfaction, and propose a method to predict the reformulation causes. We evaluate the method using the user logs of a commercial IA. The experimental results have demonstrated that features designed to detect the error of a specific component improve the performance of reformulation cause detection.Comment: 11 pages, 2 figures, accepted as a long paper for SIGDIAL 201

Auditing Search Engines for Differential Satisfaction Across Demographics

Many online services, such as search engines, social media platforms, and digital marketplaces, are advertised as being available to any user, regardless of their age, gender, or other demographic factors. However, there are growing concerns that these services may systematically underserve some groups of users. In this paper, we present a framework for internally auditing such services for differences in user satisfaction across demographic groups, using search engines as a case study. We first explain the pitfalls of na\"ively comparing the behavioral metrics that are commonly used to evaluate search engines. We then propose three methods for measuring latent differences in user satisfaction from observed differences in evaluation metrics. To develop these methods, we drew on ideas from the causal inference literature and the multilevel modeling literature. Our framework is broadly applicable to other online services, and provides general insight into interpreting their evaluation metrics.Comment: 8 pages Accepted at WWW 201

Why We Read Wikipedia

Wikipedia is one of the most popular sites on the Web, with millions of users relying on it to satisfy a broad range of information needs every day. Although it is crucial to understand what exactly these needs are in order to be able to meet them, little is currently known about why users visit Wikipedia. The goal of this paper is to fill this gap by combining a survey of Wikipedia readers with a log-based analysis of user activity. Based on an initial series of user surveys, we build a taxonomy of Wikipedia use cases along several dimensions, capturing users' motivations to visit Wikipedia, the depth of knowledge they are seeking, and their knowledge of the topic of interest prior to visiting Wikipedia. Then, we quantify the prevalence of these use cases via a large-scale user survey conducted on live Wikipedia with almost 30,000 responses. Our analyses highlight the variety of factors driving users to Wikipedia, such as current events, media coverage of a topic, personal curiosity, work or school assignments, or boredom. Finally, we match survey responses to the respondents' digital traces in Wikipedia's server logs, enabling the discovery of behavioral patterns associated with specific use cases. For instance, we observe long and fast-paced page sequences across topics for users who are bored or exploring randomly, whereas those using Wikipedia for work or school spend more time on individual articles focused on topics such as science. Our findings advance our understanding of reader motivations and behavior on Wikipedia and can have implications for developers aiming to improve Wikipedia's user experience, editors striving to cater to their readers' needs, third-party services (such as search engines) providing access to Wikipedia content, and researchers aiming to build tools such as recommendation engines.Comment: Published in WWW'17; v2 fixes caption of Table

Deep Sequential Models for Task Satisfaction Prediction

Detecting and understanding implicit signals of user satisfaction are essential for experimentation aimed at predicting searcher satisfaction. As retrieval systems have advanced, search tasks have steadily emerged as accurate units not only to capture searcher's goals but also in understanding how well a system is able to help the user achieve that goal. However, a major portion of existing work on modeling searcher satisfaction has focused on query level satisfaction. The few existing approaches for task satisfaction prediction have narrowly focused on simple tasks aimed at solving atomic information needs. In this work we go beyond such atomic tasks and consider the problem of predicting user's satisfaction when engaged in complex search tasks composed of many different queries and subtasks. We begin by considering holistic view of user interactions with the search engine result page (SERP) and extract detailed interaction sequences of their activity. We then look at query level abstraction and propose a novel deep sequential architecture which leverages the extracted interaction sequences to predict query level satisfaction. Further, we enrich this model with auxiliary features which have been traditionally used for satisfaction prediction and propose a unified multi-view model which combines the benefit of user interaction sequences with auxiliary features. Finally, we go beyond query level abstraction and consider query sequences issued by the user in order to complete a complex task, to make task level satisfaction predictions. We propose a number of functional composition techniques which take into account query level satisfaction estimates along with the query sequence to predict task level satisfaction. Through rigorous experiments, we demonstrate that the proposed deep sequential models significantly outperform established baselines at both query and task satisfaction prediction. Our findings have implications on metric development for gauging user satisfaction and on designing systems which help users accomplish complex search tasks

Distinguishing a ‘hit’ from a ‘view’: Using the access durations of lecture recordings to tell whether learning might have happened

Audiovisual recordings of lectures are available to many students in all disciplines. The use of lecture recordings has been studied extensively, but it is still not clear how, or how much, they are actually used. Previous analysis of their use has been based on either survey data or computer logs of access. In the latter case, measurements of actual use have usually been based on counts of the number of times recordings have been accessed. This does not distinguish those that happen accidentally (‘hits’), from those that might permit learning (‘views’). This distinction is essential to the meaningful analysis of the log of the actual use of recorded lectures. Using the access logs of undergraduate science students, we show that the distribution of the durations of the access of recordings of scheduled lectures has two distinct components. The most rapid of these is complete within three minutes and we infer that it reflects the behaviour of students searching among recordings. This inference is based on a comparison of these distributions with those of (i) recordings made automatically during a non-teaching period and (ii) individual users. This is also consistent with the pattern of usage by students searching for a specific recording

Affective Signals as Implicit Indicators of Information Relevancy and Information Processing Strategies

Search engines have become better in providing information to users, however, they still face major challenges such as determining how searchers process information, how they make relevance judgments, and how their cognitive or emotional state affect their search progress. We address these challenges by exploring searchers' affective dimension. In particular, we investigate how feelings, facial expressions, and electrodermal activity (EDA) could help to understand information relevancy, search progress, and information processing strategies (IPSs). To meet this goal, we designed an experiment with 45 participants exposed to affective stimuli prior solving a precision-oriented search task. Results indicate that initial affective states are linked to IPSs. In addition, we found that smiles act as implicit indicators of information relevancy and IPSs. Moreover, results convey that both smiles and EDA may serve as implicit indicators of progress and completion of search tasks. Findings from this work have practical implications in areas such as personalization and relevance feedback.ye