13 research outputs found
Understanding Mobile Search Task Relevance and User Behaviour in Context
Improvements in mobile technologies have led to a dramatic change in how and
when people access and use information, and is having a profound impact on how
users address their daily information needs. Smart phones are rapidly becoming
our main method of accessing information and are frequently used to perform
`on-the-go' search tasks. As research into information retrieval continues to
evolve, evaluating search behaviour in context is relatively new. Previous
research has studied the effects of context through either self-reported diary
studies or quantitative log analysis; however, neither approach is able to
accurately capture context of use at the time of searching. In this study, we
aim to gain a better understanding of task relevance and search behaviour via a
task-based user study (n=31) employing a bespoke Android app. The app allowed
us to accurately capture the user's context when completing tasks at different
times of the day over the period of a week. Through analysis of the collected
data, we gain a better understanding of how using smart phones on the go
impacts search behaviour, search performance and task relevance and whether or
not the actual context is an important factor.Comment: To appear in CHIIR 2019 in Glasgow, U
Target Apps Selection: Towards a Unified Search Framework for Mobile Devices
With the recent growth of conversational systems and intelligent assistants
such as Apple Siri and Google Assistant, mobile devices are becoming even more
pervasive in our lives. As a consequence, users are getting engaged with the
mobile apps and frequently search for an information need in their apps.
However, users cannot search within their apps through their intelligent
assistants. This requires a unified mobile search framework that identifies the
target app(s) for the user's query, submits the query to the app(s), and
presents the results to the user. In this paper, we take the first step forward
towards developing unified mobile search. In more detail, we introduce and
study the task of target apps selection, which has various potential real-world
applications. To this aim, we analyze attributes of search queries as well as
user behaviors, while searching with different mobile apps. The analyses are
done based on thousands of queries that we collected through crowdsourcing. We
finally study the performance of state-of-the-art retrieval models for this
task and propose two simple yet effective neural models that significantly
outperform the baselines. Our neural approaches are based on learning
high-dimensional representations for mobile apps. Our analyses and experiments
suggest specific future directions in this research area.Comment: To appear at SIGIR 201
Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems
Predicting user satisfaction in conversational systems has become critical,
as spoken conversational assistants operate in increasingly complex domains.
Online satisfaction prediction (i.e., predicting satisfaction of the user with
the system after each turn) could be used as a new proxy for implicit user
feedback, and offers promising opportunities to create more responsive and
effective conversational agents, which adapt to the user's engagement with the
agent. To accomplish this goal, we propose a conversational satisfaction
prediction model specifically designed for open-domain spoken conversational
agents, called ConvSAT. To operate robustly across domains, ConvSAT aggregates
multiple representations of the conversation, namely the conversation history,
utterance and response content, and system- and user-oriented behavioral
signals. We first calibrate ConvSAT performance against state of the art
methods on a standard dataset (Dialogue Breakdown Detection Challenge) in an
online regime, and then evaluate ConvSAT on a large dataset of conversations
with real users, collected as part of the Alexa Prize competition. Our
experimental results show that ConvSAT significantly improves satisfaction
prediction for both offline and online setting on both datasets, compared to
the previously reported state-of-the-art approaches. The insights from our
study can enable more intelligent conversational systems, which could adapt in
real-time to the inferred user satisfaction and engagement.Comment: Published in CIKM '19, 10 page
Optimizing Interactive Systems via Data-Driven Objectives
Effective optimization is essential for real-world interactive systems to
provide a satisfactory user experience in response to changing user behavior.
However, it is often challenging to find an objective to optimize for
interactive systems (e.g., policy learning in task-oriented dialog systems).
Generally, such objectives are manually crafted and rarely capture complex user
needs in an accurate manner. We propose an approach that infers the objective
directly from observed user interactions. These inferences can be made
regardless of prior knowledge and across different types of user behavior. We
introduce Interactive System Optimizer (ISO), a novel algorithm that uses these
inferred objectives for optimization. Our main contribution is a new general
principled approach to optimizing interactive systems using data-driven
objectives. We demonstrate the high effectiveness of ISO over several
simulations.Comment: 30 pages, 12 figures. arXiv admin note: text overlap with
arXiv:1802.0630
Deep Sequential Models for Task Satisfaction Prediction
Detecting and understanding implicit signals of user satisfaction are essential for experimentation aimed at predicting searcher satisfaction. As retrieval systems have advanced, search tasks have steadily emerged as accurate units not only to capture searcher's goals but also in understanding how well a system is able to help the user achieve that goal. However, a major portion of existing work on modeling searcher satisfaction has focused on query level satisfaction. The few existing approaches for task satisfaction prediction have narrowly focused on simple tasks aimed at solving atomic information needs.
In this work we go beyond such atomic tasks and consider the problem of predicting user's satisfaction when engaged in complex search tasks composed of many different queries and subtasks. We begin by considering holistic view of user interactions with the search engine result page (SERP) and extract detailed interaction sequences of their activity. We then look at query level abstraction and propose a novel deep sequential architecture which leverages the extracted interaction sequences to predict query level satisfaction. Further, we enrich this model with auxiliary features which have been traditionally used for satisfaction prediction and propose a unified multi-view model which combines the benefit of user interaction sequences with auxiliary features.
Finally, we go beyond query level abstraction and consider query sequences issued by the user in order to complete a complex task, to make task level satisfaction predictions. We propose a number of functional composition techniques which take into account query level satisfaction estimates along with the query sequence to predict task level satisfaction. Through rigorous experiments, we demonstrate that the proposed deep sequential models significantly outperform established baselines at both query and task satisfaction prediction. Our findings have implications on metric development for gauging user satisfaction and on designing systems which help users accomplish complex search tasks
Using contextual information to understand searching and browsing behavior
There is great imbalance in the richness of information on the web and the succinctness and poverty of search requests of web users, making their queries only a partial description of the underlying complex information needs. Finding ways to better leverage contextual information and make search context-aware holds the promise to dramatically improve the search experience of users. We conducted a series of studies to discover, model and utilize contextual information in order to understand and improve users' searching and browsing behavior on the web. Our results capture important aspects of context under the realistic conditions of different online search services, aiming to ensure that our scientific insights and solutions transfer to the operational settings of real world applications