An Analysis of Mixed Initiative and Collaboration in Information-Seeking Dialogues

The ability to engage in mixed-initiative interaction is one of the core requirements for a conversational search system. How to achieve this is poorly understood. We propose a set of unsupervised metrics, termed ConversationShape, that highlights the role each of the conversation participants plays by comparing the distribution of vocabulary and utterance types. Using ConversationShape as a lens, we take a closer look at several conversational search datasets and compare them with other dialogue datasets to better understand the types of dialogue interaction they represent, either driven by the information seeker or the assistant. We discover that deviations from the ConversationShape of a human-human dialogue of the same type is predictive of the quality of a human-machine dialogue.Comment: SIGIR 2020 short conference pape

Chatbots for learning: A review of educational chatbots for the Facebook Messenger

With the exponential growth in the mobile device market over the last decade, chatbots are becoming an increasingly popular option to interact with users, and their popularity and adoption are rapidly spreading. These mobile devices change the way we communicate and allow ever-present learning in various environments. This study examined educational chatbots for Facebook Messenger to support learning. The independent web directory was screened to assess chatbots for this study resulting in the identification of 89 unique chatbots. Each chatbot was classified by language, subject matter and developer's platform. Finally, we evaluated 47 educational chatbots using the Facebook Messenger platform based on the analytic hierarchy process against the quality attributes of teaching, humanity, affect, and accessibility. We found that educational chatbots on the Facebook Messenger platform vary from the basic level of sending personalized messages to recommending learning content. Results show that chatbots which are part of the instant messaging application are still in its early stages to become artificial intelligence teaching assistants. The findings provide tips for teachers to integrate chatbots into classroom practice and advice what types of chatbots they can try out.Web of Science151art. no. 10386

Entertaining and Opinionated but Too Controlling: A Large-Scale User Study of an Open Domain Alexa Prize System

Conversational systems typically focus on functional tasks such as scheduling appointments or creating todo lists. Instead we design and evaluate SlugBot (SB), one of 8 semifinalists in the 2018 AlexaPrize, whose goal is to support casual open-domain social inter-action. This novel application requires both broad topic coverage and engaging interactive skills. We developed a new technical approach to meet this demanding situation by crowd-sourcing novel content and introducing playful conversational strategies based on storytelling and games. We collected over 10,000 conversations during August 2018 as part of the Alexa Prize competition. We also conducted an in-lab follow-up qualitative evaluation. Over-all users found SB moderately engaging; conversations averaged 3.6 minutes and involved 26 user turns. However, users reacted very differently to different conversation subtypes. Storytelling and games were evaluated positively; these were seen as entertaining with predictable interactive structure. They also led users to impute personality and intelligence to SB. In contrast, search and general Chit-Chat induced coverage problems; here users found it hard to infer what topics SB could understand, with these conversations seen as being too system-driven. Theoretical and design implications suggest a move away from conversational systems that simply provide factual information. Future systems should be designed to have their own opinions with personal stories to share, and SB provides an example of how we might achieve this.Comment: To appear in 1st International Conference on Conversational User Interfaces (CUI 2019

Rewarding Chatbots for Real-World Engagement with Millions of Users

The emergence of pretrained large language models has led to the deployment of a range of social chatbots for chitchat. Although these chatbots demonstrate language ability and fluency, they are not guaranteed to be engaging and can struggle to retain users. This work investigates the development of social chatbots that prioritize user engagement to enhance retention, specifically examining the use of human feedback to efficiently develop highly engaging chatbots. The proposed approach uses automatic pseudo-labels collected from user interactions to train a reward model that can be used to reject low-scoring sample responses generated by the chatbot model at inference time. Intuitive evaluation metrics, such as mean conversation length (MCL), are introduced as proxies to measure the level of engagement of deployed chatbots. A/B testing on groups of 10,000 new daily chatbot users on the Chai Research platform shows that this approach increases the MCL by up to 70%, which translates to a more than 30% increase in user retention for a GPT-J 6B model. Future work aims to use the reward model to realise a data fly-wheel, where the latest user conversations can be used to alternately fine-tune the language model and the reward model

Conversational agents with personality

Conversational agents (CAs) such as voice assistants and chatbots have permeated people's everyday lives. When interacting with these CAs, people automatically attribute a personality to them regardless of whether the CA designer intended it or not. This personality attribution fundamentally influences people's interaction behaviour and attitude towards the CA. By deliberately shaping the CA personality, designers have the opportunity to steer these automatic personality attributions in a desired direction. However, little information is available on how to design such a desired personality impression for a CA. Furthermore, in inter-human interaction, there is no such thing as a perfect personality. Nonetheless, today's commercial CAs have adopted a one-size-fits-all approach to their personality design, ignoring the potential benefits of adaptation. These two insights, namely (1) that users assign a personality to CAs and (2) that there is no such thing as a perfect personality, motivate the vision of this thesis: To improve the interaction between users and CAs by deliberately imbuing CAs with personality and tailoring them to user preferences. This dissertation pursues two primary goals to realise this vision: (1) to develop methods to imbue CAs with personality systematically and (2) to examine user preferences for CA personalities. To achieve the first goal, I introduce two approaches to imbue CAs with personality based on two underlying personality descriptions. The first approach adopts the human Big Five personality model as the theoretical basis for describing CA personality. This adoption allows me to transfer behaviour cues associated with human personality traits compiled from the psycholinguistic literature and my work to synthesise three levels of Agreeableness and Extraversion implemented in fully functional text-based CAs. An empirical evaluation of users' perceptions of these CAs after interacting with them demonstrates that human behaviour cues may be used to synthesise Agreeableness. However, they are insufficient to elicit the impression of low Extraversion or paint a complete picture of CA personality. Due to this insufficiency, I develop a second approach in which I explore whether the human Big Five model can be used to describe CA personality. To this end, I apply the psycholexical approach, which yields ten personality dimensions that do not correspond with the human Big Five model. Consequently, I propose these ten dimensions as an alternative comprehensive way to describe CA personality and introduce a new method, Enactment-based Dialogue Design, to synthesise personality based on these ten dimensions. To achieve the second goal, I present two approaches to examine user preferences for CA personality. Using a deductive approach, I investigate whether users prefer low, average, or high levels of four different personality dimensions in a CA in the context of different use cases. These investigations show that users have very individual preferences for the dimensions Extraversion and Social-Entertaining, whereas the majority prefer CAs that have a medium or high level of Agreeableness and a low level of Confrontational. I find the deductive approach to be useful for capturing users' evaluation of a personality-imbued CA, but it is not effective in collecting user requirements and visions of a perfect CA. The second inductive approach, however, furnishes a novel pragmatic method to better engage users in developing CA personalities. In this context, I also examine the influence of users’ personalities on their preferences for CA personality, but the effects are minimal. In summary, this thesis makes the following contributions to imbuing CAs with personality: (1) theoretical clarity on the necessity of dedicated personality descriptions for CAs, (2) a set of verbal cues associated with human personality implemented in fully functional text-based CA artefacts, (3) an exploration of two methods for synthesising personality in CAs, and (4) a new method for eliciting users' vision of the perfect CA. I consolidate these methods into a user-centred design process for developing CAs with personality. Furthermore, I provide empirical evidence of diverging user preferences and discuss overarching patterns which CA designers may use to tailor their CA personalities to individual users. Finally, this thesis proposes a research agenda for future work, which addresses the challenges that emerged from the presented work.Conversational Agents (CAs) wie Sprachassistenten und Chatbots sind aus dem Alltag der Menschen nicht mehr wegzudenken. In der Interaktion mit CAs schreiben Benutzer:innen ihnen automatisch eine Persönlichkeit zu, unabhängig davon, ob die CA-Designer:innen dies beabsichtigten oder nicht. Diese Persönlichkeitszuschreibung beeinflusst grundlegend das Interaktionsverhalten und die Einstellung der Benutzer:innen gegenüber den CAs. Eine bewusste Gestaltung der CA-Persönlichkeit erlaubt Designer:innen, diese automatischen Persönlichkeitszuschreibungen in eine gewünschte Richtung zu lenken. Jedoch gibt es nur wenige Informationen darüber, wie eine solche gewünschte Persönlichkeit für einen CA gestaltet werden kann. Darüber hinaus gibt es in der zwischenmenschlichen Interaktion nicht die eine perfekte CA-Persönlichkeit, die allen Benutzer:innen gleichermaßen gefällt. Nichtsdestotrotz sind heutige kommerzielle CAs lediglich mit einer Persönlichkeit für alle Benutzer:innen ausgestattet und lassen somit die potenziellen Vorteile einer Anpassung an individuelle Präferenzen außer Acht. Diese beiden Erkenntnisse, (1) dass Benutzer:innen CAs eine Persönlichkeit zuweisen und (2) dass es die eine perfekte Persönlichkeit nicht gibt, motivieren die Vision dieser Arbeit: Die Interaktion zwischen Benutzer:innen und CAs zu verbessern, indem CAs gezielt mit einer Persönlichkeit ausgestattet und an die Präferenzen der Benutzer:innen angepasst werden. Um diese Vision zu realisieren, verfolgt die vorliegende Dissertation zwei primäre Ziele: (1) die Entwicklung von Methoden, um CAs systematisch eine Persönlichkeit zu verleihen und (2) die Untersuchung von Präferenzen der Benutzer:innen für CA-Persönlichkeiten. Um das erste Ziel zu erreichen, stelle ich zwei Ansätze zur Ausstattung von CAs mit Persönlichkeit vor, die auf der jeweiligen zugrunde liegenden Persönlichkeitsbeschreibung basieren. In dem ersten Ansatz verwende ich das menschliche Big Five Persönlichkeitsmodell als theoretische Grundlage für die Beschreibung von CA-Persönlichkeit. Diese Annahme ermöglicht es, Verhaltenshinweise, die mit menschlichen Persönlichkeitsmerkmalen assoziiert sind, in der psycholinguistischen Literatur sowie meiner eigenen Arbeit zu identifizieren. Diese Verhaltenshinweise übertrage ich dann auf CAs, um jeweils drei Ausprägungen von Verträglichkeit und Extraversion zu synthetisieren, die in vollständig funktionsfähigen text-basierten CAs implementiert sind. Eine empirische Untersuchung der Wahrnehmung dieser text-basierten CAs deutet darauf hin, dass menschliche Verhaltenshinweise genutzt werden können, um Verträglichkeit zu synthetisieren. Sie sind jedoch unzureichend, um den Eindruck von niedriger Extraversion zu vermitteln sowie die Persönlichkeit von CAs vollständig abzubilden. Aufgrund der mangelnden Eignung der menschlichen Persönlichkeitsbeschreibung entwickle ich einen zweiten Ansatz, in dem ich untersuche, ob das menschliche Big Five Modell für die Beschreibung von CA-Persönlichkeit genutzt werden kann. Zu diesem Zweck wende ich den psycholexikalischen Ansatz an, aus dem zehn Persönlichkeitsdimensionen hervorgehen, die nicht mit dem menschlichen Big Five Modell übereinstimmen. Folglich schlage ich diese zehn Dimensionen als eine alternative und vollständige Möglichkeit zur Beschreibung von CA-Persönlichkeit vor. Außerdem führe ich eine neue Methode, genannt Inszenierung-basiertes Dialogdesign, ein, die es ermöglicht, Persönlichkeit auf Grundlage dieser zehn Dimensionen zu synthetisieren. Um das zweite Ziel zu erreichen, stelle ich zwei Ansätze zur Untersuchung der Präferenzen von Benutzer:innen für CA-Persönlichkeit vor. In einem deduktiven Ansatz untersuche ich zunächst, ob Benutzer:innen eine niedrige, durchschnittliche oder hohe Ausprägung von vier verschiedenen Persönlichkeitsdimensionen in einem CA im Kontext unterschiedlicher Anwendungsfälle bevorzugen. Diese Untersuchungen zeigen, dass die Benutzer:innen sehr individuelle Präferenzen für die Dimensionen Extraversion und Sozial-Unterhaltend haben, während die Mehrheit CAs bevorzugt, die eine mittlere oder hohe Ausprägung in Verträglichkeit sowie eine niedrige Ausprägung in Konfrontativ aufweisen. Obgleich der deduktive Ansatz nützlich für die Evaluierung von CA-Prototypen ist, ermöglicht dieser es nicht, Bedürfnisse und Vorstellungen der Benutzer:innen einzufangen. Im zweiten, induktiven Ansatz präsentiere ich daher eine neue pragmatische Methode, um die Benutzer:innen besser in die Entwicklung von CA-Persönlichkeiten einzubinden. In diesem Zusammenhang untersuche ich darüber hinaus den Einfluss der Persönlichkeit der Benutzer:innen auf ihre Präferenzen für die CA-Persönlichkeit, finde jedoch nur einen begrenzten Effekt. Zusammenfassend leistet die vorliegende Arbeit die folgenden wissenschaftlichen Beiträge zur Ausstattung von CAs mit Persönlichkeit: (1) Theoretische Klarheit über die Notwendigkeit dedizierter Persönlichkeitsbeschreibungen für CAs, (2) eine Sammlung verbaler Verhaltenshinweise, die mit menschlicher Persönlichkeit assoziiert sind und in voll funktionsfähigen CA-Artefakten implementiert sind, (3) eine Exploration von zwei Methoden zur Synthese von Persönlichkeit in CAs und (4) eine neue Methode, um die Vision eines perfekten CAs von Benutzer:innen zu eruieren. Ich führe diese Methoden in einem benutzungszentrierten Designprozess für die Entwicklung von CA-Persönlichkeiten zusammen. Darüber hinaus liefere ich empirische Belege für divergierende Präferenzen der Benutzer:innen für CA-Persönlichkeit und erörtere übergreife Muster, die CA-Designer:innen anwenden können, um ihre CA-Persönlichkeiten auf individuelle Benutzer:innen zuzuschneiden. Abschließend wird eine Forschungsagenda für zukünftige Arbeiten präsentiert, welche die Herausforderungen diskutiert, die sich aus den vorgestellten Arbeiten ergeben

User Intent Prediction in Information-seeking Conversations

Conversational assistants are being progressively adopted by the general population. However, they are not capable of handling complicated information-seeking tasks that involve multiple turns of information exchange. Due to the limited communication bandwidth in conversational search, it is important for conversational assistants to accurately detect and predict user intent in information-seeking conversations. In this paper, we investigate two aspects of user intent prediction in an information-seeking setting. First, we extract features based on the content, structural, and sentiment characteristics of a given utterance, and use classic machine learning methods to perform user intent prediction. We then conduct an in-depth feature importance analysis to identify key features in this prediction task. We find that structural features contribute most to the prediction performance. Given this finding, we construct neural classifiers to incorporate context information and achieve better performance without feature engineering. Our findings can provide insights into the important factors and effective methods of user intent prediction in information-seeking conversations.Comment: Accepted to CHIIR 201

Listening between the Lines: Learning Personal Attributes from Conversations

Open-domain dialogue agents must be able to converse about many topics while incorporating knowledge about the user into the conversation. In this work we address the acquisition of such knowledge, for personalization in downstream Web applications, by extracting personal attributes from conversations. This problem is more challenging than the established task of information extraction from scientific publications or Wikipedia articles, because dialogues often give merely implicit cues about the speaker. We propose methods for inferring personal attributes, such as profession, age or family status, from conversations using deep learning. Specifically, we propose several Hidden Attribute Models, which are neural networks leveraging attention mechanisms and embeddings. Our methods are trained on a per-predicate basis to output rankings of object values for a given subject-predicate combination (e.g., ranking the doctor and nurse professions high when speakers talk about patients, emergency rooms, etc). Experiments with various conversational texts including Reddit discussions, movie scripts and a collection of crowdsourced personal dialogues demonstrate the viability of our methods and their superior performance compared to state-of-the-art baselines.Comment: published in WWW'1