122 research outputs found
Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems
Predicting user satisfaction in conversational systems has become critical,
as spoken conversational assistants operate in increasingly complex domains.
Online satisfaction prediction (i.e., predicting satisfaction of the user with
the system after each turn) could be used as a new proxy for implicit user
feedback, and offers promising opportunities to create more responsive and
effective conversational agents, which adapt to the user's engagement with the
agent. To accomplish this goal, we propose a conversational satisfaction
prediction model specifically designed for open-domain spoken conversational
agents, called ConvSAT. To operate robustly across domains, ConvSAT aggregates
multiple representations of the conversation, namely the conversation history,
utterance and response content, and system- and user-oriented behavioral
signals. We first calibrate ConvSAT performance against state of the art
methods on a standard dataset (Dialogue Breakdown Detection Challenge) in an
online regime, and then evaluate ConvSAT on a large dataset of conversations
with real users, collected as part of the Alexa Prize competition. Our
experimental results show that ConvSAT significantly improves satisfaction
prediction for both offline and online setting on both datasets, compared to
the previously reported state-of-the-art approaches. The insights from our
study can enable more intelligent conversational systems, which could adapt in
real-time to the inferred user satisfaction and engagement.Comment: Published in CIKM '19, 10 page
Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Features
Predicting the success of Conversational Task Assistants (CTA) can be
critical to understand user behavior and act accordingly. In this paper, we
propose TB-Rater, a Transformer model which combines conversational-flow
features with user behavior features for predicting user ratings in a CTA
scenario. In particular, we use real human-agent conversations and ratings
collected in the Alexa TaskBot challenge, a novel multimodal and multi-turn
conversational context. Our results show the advantages of modeling both the
conversational-flow and behavioral aspects of the conversation in a single
model for offline rating prediction. Additionally, an analysis of the
CTA-specific behavioral features brings insights into this setting and can be
used to bootstrap future systems
Quantifying the Effects of Prosody Modulation on User Engagement and Satisfaction in Conversational Systems
As voice-based assistants such as Alexa, Siri, and Google Assistant become
ubiquitous, users increasingly expect to maintain natural and informative
conversations with such systems. However, for an open-domain conversational
system to be coherent and engaging, it must be able to maintain the user's
interest for extended periods, without sounding boring or annoying. In this
paper, we investigate one natural approach to this problem, of modulating
response prosody, i.e., changing the pitch and cadence of the response to
indicate delight, sadness or other common emotions, as well as using
pre-recorded interjections. Intuitively, this approach should improve the
naturalness of the conversation, but attempts to quantify the effects of
prosodic modulation on user satisfaction and engagement remain challenging. To
accomplish this, we report results obtained from a large-scale empirical study
that measures the effects of prosodic modulation on user behavior and
engagement across multiple conversation domains, both immediately after each
turn, and at the overall conversation level. Our results indicate that the
prosody modulation significantly increases both immediate and overall user
satisfaction. However, since the effects vary across different domains, we
verify that prosody modulations do not substitute for coherent, informative
content of the responses. Together, our results provide useful tools and
insights for improving the naturalness of responses in conversational systems.Comment: Published in CHIIR 2020, 4 page
Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval
With the improvements in speech recognition and voice generation technologies
over the last years, a lot of companies have sought to develop conversation
understanding systems that run on mobile phones or smart home devices through
natural language interfaces. Conversational assistants, such as Google
Assistant and Microsoft Cortana, can help users to complete various types of
tasks. This requires an accurate understanding of the user's information need
as the conversation evolves into multiple turns. Finding relevant context in a
conversation's history is challenging because of the complexity of natural
language and the evolution of a user's information need. In this work, we
present an extensive analysis of language, relevance, dependency of user
utterances in a multi-turn information-seeking conversation. To this aim, we
have annotated relevant utterances in the conversations released by the TREC
CaST 2019 track. The annotation labels determine which of the previous
utterances in a conversation can be used to improve the current one.
Furthermore, we propose a neural utterance relevance model based on BERT
fine-tuning, outperforming competitive baselines. We study and compare the
performance of multiple retrieval models, utilizing different strategies to
incorporate the user's context. The experimental results on both classification
and retrieval tasks show that our proposed approach can effectively identify
and incorporate the conversation context. We show that processing the current
utterance using the predicted relevant utterance leads to a 38% relative
improvement in terms of nDCG@20. Finally, to foster research in this area, we
have released the dataset of the annotations.Comment: To appear in ACM CHIIR 2020, Vancouver, BC, Canad
Using contextual information to understand searching and browsing behavior
There is great imbalance in the richness of information on the web and the succinctness and poverty of search requests of web users, making their queries only a partial description of the underlying complex information needs. Finding ways to better leverage contextual information and make search context-aware holds the promise to dramatically improve the search experience of users. We conducted a series of studies to discover, model and utilize contextual information in order to understand and improve users' searching and browsing behavior on the web. Our results capture important aspects of context under the realistic conditions of different online search services, aiming to ensure that our scientific insights and solutions transfer to the operational settings of real world applications
- …