Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

Predicting user satisfaction in conversational systems has become critical, as spoken conversational assistants operate in increasingly complex domains. Online satisfaction prediction (i.e., predicting satisfaction of the user with the system after each turn) could be used as a new proxy for implicit user feedback, and offers promising opportunities to create more responsive and effective conversational agents, which adapt to the user's engagement with the agent. To accomplish this goal, we propose a conversational satisfaction prediction model specifically designed for open-domain spoken conversational agents, called ConvSAT. To operate robustly across domains, ConvSAT aggregates multiple representations of the conversation, namely the conversation history, utterance and response content, and system- and user-oriented behavioral signals. We first calibrate ConvSAT performance against state of the art methods on a standard dataset (Dialogue Breakdown Detection Challenge) in an online regime, and then evaluate ConvSAT on a large dataset of conversations with real users, collected as part of the Alexa Prize competition. Our experimental results show that ConvSAT significantly improves satisfaction prediction for both offline and online setting on both datasets, compared to the previously reported state-of-the-art approaches. The insights from our study can enable more intelligent conversational systems, which could adapt in real-time to the inferred user satisfaction and engagement.Comment: Published in CIKM '19, 10 page

Predicting Causes of Reformulation in Intelligent Assistants

Intelligent assistants (IAs) such as Siri and Cortana conversationally interact with users and execute a wide range of actions (e.g., searching the Web, setting alarms, and chatting). IAs can support these actions through the combination of various components such as automatic speech recognition, natural language understanding, and language generation. However, the complexity of these components hinders developers from determining which component causes an error. To remove this hindrance, we focus on reformulation, which is a useful signal of user dissatisfaction, and propose a method to predict the reformulation causes. We evaluate the method using the user logs of a commercial IA. The experimental results have demonstrated that features designed to detect the error of a specific component improve the performance of reformulation cause detection.Comment: 11 pages, 2 figures, accepted as a long paper for SIGDIAL 201

Modeling user return time using inhomogeneous poisson process

For Intelligent Assistants (IA), user activity is often used as a lag metric for user satisfaction or engagement. Conversely, predictive leading metrics for engagement can be helpful with decision making and evaluating changes in satisfaction caused by new features. In this paper, we propose User Return Time (URT), a fine grain metric for gauging user engagement. To compute URT, we model continuous inter-arrival times between users’ use of service via a log Gaussian Cox process (LGCP), a form of inhomogeneous Poisson process which captures the irregular variations in user usage rate and personal preferences typical of an IA. We show the effectiveness of the proposed approaches on predicting the return time of users on real-world data collected from an IA. Experimental results demonstrate that our model is able to predict user return times reasonably well and considerably better than strong baselines that make the prediction based on past utterance frequency

“Hey Siri, how much do you know about me?”: Intelligent Virtual Assistants and the dilemma between commodity and privacy

Dissertation presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Marketing IntelligenceArtificial Intelligence has been gaining ground over time, and Intelligent Virtual Assistants (IVAs) are no exception, as people realize that they can be using the time and effort spend on daily tasks, more efficiently, by trusting them to these technological auxiliaries. IVAs are being used by people all over the world to change channels, play songs, turn up the volume, reading text messages and emails, calling someone or even grocery shopping when something’s missing, among many other purposes. However, previous studies show that the concerns with data privacy from using these emerging technologies is growing, since in order to work, these AI assistants need constant access to the devices’ microphones, cameras or even locations. Faced with this dilemma, what weights the most on the scale: The users’ commodity, or their information’s privacy and security? In this research, we used PLS-SEM in order to analyze the barriers and drivers that people take into consideration when having to choose if they would use or not Intelligent Virtual Assistants, and what influences this decision, based on four variables: Familiarity, Trust, Satisfaction and Privacy. Our findings conclude that consumers decidedly value their commodity, having familiarity and satisfaction influencing positively the intentions of use, and having satisfaction being highly influenced by trust. At the same time, it also shows that privacy is an inhibitor to many consumers, affecting negatively the usage perception, as expected

Using contextual information to understand searching and browsing behavior

There is great imbalance in the richness of information on the web and the succinctness and poverty of search requests of web users, making their queries only a partial description of the underlying complex information needs. Finding ways to better leverage contextual information and make search context-aware holds the promise to dramatically improve the search experience of users. We conducted a series of studies to discover, model and utilize contextual information in order to understand and improve users' searching and browsing behavior on the web. Our results capture important aspects of context under the realistic conditions of different online search services, aiming to ensure that our scientific insights and solutions transfer to the operational settings of real world applications