Quantifying the Effects of Prosody Modulation on User Engagement and Satisfaction in Conversational Systems

As voice-based assistants such as Alexa, Siri, and Google Assistant become ubiquitous, users increasingly expect to maintain natural and informative conversations with such systems. However, for an open-domain conversational system to be coherent and engaging, it must be able to maintain the user's interest for extended periods, without sounding boring or annoying. In this paper, we investigate one natural approach to this problem, of modulating response prosody, i.e., changing the pitch and cadence of the response to indicate delight, sadness or other common emotions, as well as using pre-recorded interjections. Intuitively, this approach should improve the naturalness of the conversation, but attempts to quantify the effects of prosodic modulation on user satisfaction and engagement remain challenging. To accomplish this, we report results obtained from a large-scale empirical study that measures the effects of prosodic modulation on user behavior and engagement across multiple conversation domains, both immediately after each turn, and at the overall conversation level. Our results indicate that the prosody modulation significantly increases both immediate and overall user satisfaction. However, since the effects vary across different domains, we verify that prosody modulations do not substitute for coherent, informative content of the responses. Together, our results provide useful tools and insights for improving the naturalness of responses in conversational systems.Comment: Published in CHIIR 2020, 4 page

Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

Predicting user satisfaction in conversational systems has become critical, as spoken conversational assistants operate in increasingly complex domains. Online satisfaction prediction (i.e., predicting satisfaction of the user with the system after each turn) could be used as a new proxy for implicit user feedback, and offers promising opportunities to create more responsive and effective conversational agents, which adapt to the user's engagement with the agent. To accomplish this goal, we propose a conversational satisfaction prediction model specifically designed for open-domain spoken conversational agents, called ConvSAT. To operate robustly across domains, ConvSAT aggregates multiple representations of the conversation, namely the conversation history, utterance and response content, and system- and user-oriented behavioral signals. We first calibrate ConvSAT performance against state of the art methods on a standard dataset (Dialogue Breakdown Detection Challenge) in an online regime, and then evaluate ConvSAT on a large dataset of conversations with real users, collected as part of the Alexa Prize competition. Our experimental results show that ConvSAT significantly improves satisfaction prediction for both offline and online setting on both datasets, compared to the previously reported state-of-the-art approaches. The insights from our study can enable more intelligent conversational systems, which could adapt in real-time to the inferred user satisfaction and engagement.Comment: Published in CIKM '19, 10 page

A financial crime analysis methodology for financial discussion boards using information extraction techniques

Financial discussion boards (FDBs) have been widely used for a variety of financial knowledge exchange activities through the posting of comments. Popular public FDBs are prone to be used as a medium for spreading misleading financial information due to having larger audience groups. Moderation of posted content heavily relies on manual tasks. Unfortunately, the daily comments volume received on popular FDBs realistically prevents human moderators or relevant authorities from proactively monitoring and moderating possibly fraudulent FDB content as it is extremely time-consuming and expensive to manually read all the content. This thesis presents a financial crime analysis methodology (which is comprised of novel forward analysis and novel backward analysis methodologies) implemented in a template-based Information Extraction (IE) prototype system, namely FDBs Miner (FDBM). The methodologies aim to detect potentially illegal Pump and Dump (P&D) activities on FDBs with the integration of per minute share prices in the detection process. This integration can reduce false positives during the detection as it categorises the potentially illegal comments into different risk levels for investigation purposes. P&D is a well-known financial crime that happens through different methods including FDBs. P&D happens when fraudsters deceive investors into buying stocks by spreading misleading information. FDBM extracts a company’s ticker symbol (i.e. a unique symbol that represents and identifies each listed company on the stock market), comments and share prices from FDBs based in the UK for experimental purposes. Results from both forward and backward analysis experiments show that the two novel methodologies can aid relevant authorities in the detection of potentially illegal activities on FDBs. Semantic Textual Similarity (STS) experiments have also shown that the approach could be adopted in the process of detecting potentially illegal activities on FDBs