Developing a Personality Model for Speech-based Conversational Agents Using the Psycholexical Approach

We present the first systematic analysis of personality dimensions developed specifically to describe the personality of speech-based conversational agents. Following the psycholexical approach from psychology, we first report on a new multi-method approach to collect potentially descriptive adjectives from 1) a free description task in an online survey (228 unique descriptors), 2) an interaction task in the lab (176 unique descriptors), and 3) a text analysis of 30,000 online reviews of conversational agents (Alexa, Google Assistant, Cortana) (383 unique descriptors). We aggregate the results into a set of 349 adjectives, which are then rated by 744 people in an online survey. A factor analysis reveals that the commonly used Big Five model for human personality does not adequately describe agent personality. As an initial step to developing a personality model, we propose alternative dimensions and discuss implications for the design of agent personalities, personality-aware personalisation, and future research.Comment: 14 pages, 2 figures, 3 tables, CHI'2

Conversational agents with personality

Conversational agents (CAs) such as voice assistants and chatbots have permeated people's everyday lives. When interacting with these CAs, people automatically attribute a personality to them regardless of whether the CA designer intended it or not. This personality attribution fundamentally influences people's interaction behaviour and attitude towards the CA. By deliberately shaping the CA personality, designers have the opportunity to steer these automatic personality attributions in a desired direction. However, little information is available on how to design such a desired personality impression for a CA. Furthermore, in inter-human interaction, there is no such thing as a perfect personality. Nonetheless, today's commercial CAs have adopted a one-size-fits-all approach to their personality design, ignoring the potential benefits of adaptation. These two insights, namely (1) that users assign a personality to CAs and (2) that there is no such thing as a perfect personality, motivate the vision of this thesis: To improve the interaction between users and CAs by deliberately imbuing CAs with personality and tailoring them to user preferences. This dissertation pursues two primary goals to realise this vision: (1) to develop methods to imbue CAs with personality systematically and (2) to examine user preferences for CA personalities. To achieve the first goal, I introduce two approaches to imbue CAs with personality based on two underlying personality descriptions. The first approach adopts the human Big Five personality model as the theoretical basis for describing CA personality. This adoption allows me to transfer behaviour cues associated with human personality traits compiled from the psycholinguistic literature and my work to synthesise three levels of Agreeableness and Extraversion implemented in fully functional text-based CAs. An empirical evaluation of users' perceptions of these CAs after interacting with them demonstrates that human behaviour cues may be used to synthesise Agreeableness. However, they are insufficient to elicit the impression of low Extraversion or paint a complete picture of CA personality. Due to this insufficiency, I develop a second approach in which I explore whether the human Big Five model can be used to describe CA personality. To this end, I apply the psycholexical approach, which yields ten personality dimensions that do not correspond with the human Big Five model. Consequently, I propose these ten dimensions as an alternative comprehensive way to describe CA personality and introduce a new method, Enactment-based Dialogue Design, to synthesise personality based on these ten dimensions. To achieve the second goal, I present two approaches to examine user preferences for CA personality. Using a deductive approach, I investigate whether users prefer low, average, or high levels of four different personality dimensions in a CA in the context of different use cases. These investigations show that users have very individual preferences for the dimensions Extraversion and Social-Entertaining, whereas the majority prefer CAs that have a medium or high level of Agreeableness and a low level of Confrontational. I find the deductive approach to be useful for capturing users' evaluation of a personality-imbued CA, but it is not effective in collecting user requirements and visions of a perfect CA. The second inductive approach, however, furnishes a novel pragmatic method to better engage users in developing CA personalities. In this context, I also examine the influence of users’ personalities on their preferences for CA personality, but the effects are minimal. In summary, this thesis makes the following contributions to imbuing CAs with personality: (1) theoretical clarity on the necessity of dedicated personality descriptions for CAs, (2) a set of verbal cues associated with human personality implemented in fully functional text-based CA artefacts, (3) an exploration of two methods for synthesising personality in CAs, and (4) a new method for eliciting users' vision of the perfect CA. I consolidate these methods into a user-centred design process for developing CAs with personality. Furthermore, I provide empirical evidence of diverging user preferences and discuss overarching patterns which CA designers may use to tailor their CA personalities to individual users. Finally, this thesis proposes a research agenda for future work, which addresses the challenges that emerged from the presented work.Conversational Agents (CAs) wie Sprachassistenten und Chatbots sind aus dem Alltag der Menschen nicht mehr wegzudenken. In der Interaktion mit CAs schreiben Benutzer:innen ihnen automatisch eine Persönlichkeit zu, unabhängig davon, ob die CA-Designer:innen dies beabsichtigten oder nicht. Diese Persönlichkeitszuschreibung beeinflusst grundlegend das Interaktionsverhalten und die Einstellung der Benutzer:innen gegenüber den CAs. Eine bewusste Gestaltung der CA-Persönlichkeit erlaubt Designer:innen, diese automatischen Persönlichkeitszuschreibungen in eine gewünschte Richtung zu lenken. Jedoch gibt es nur wenige Informationen darüber, wie eine solche gewünschte Persönlichkeit für einen CA gestaltet werden kann. Darüber hinaus gibt es in der zwischenmenschlichen Interaktion nicht die eine perfekte CA-Persönlichkeit, die allen Benutzer:innen gleichermaßen gefällt. Nichtsdestotrotz sind heutige kommerzielle CAs lediglich mit einer Persönlichkeit für alle Benutzer:innen ausgestattet und lassen somit die potenziellen Vorteile einer Anpassung an individuelle Präferenzen außer Acht. Diese beiden Erkenntnisse, (1) dass Benutzer:innen CAs eine Persönlichkeit zuweisen und (2) dass es die eine perfekte Persönlichkeit nicht gibt, motivieren die Vision dieser Arbeit: Die Interaktion zwischen Benutzer:innen und CAs zu verbessern, indem CAs gezielt mit einer Persönlichkeit ausgestattet und an die Präferenzen der Benutzer:innen angepasst werden. Um diese Vision zu realisieren, verfolgt die vorliegende Dissertation zwei primäre Ziele: (1) die Entwicklung von Methoden, um CAs systematisch eine Persönlichkeit zu verleihen und (2) die Untersuchung von Präferenzen der Benutzer:innen für CA-Persönlichkeiten. Um das erste Ziel zu erreichen, stelle ich zwei Ansätze zur Ausstattung von CAs mit Persönlichkeit vor, die auf der jeweiligen zugrunde liegenden Persönlichkeitsbeschreibung basieren. In dem ersten Ansatz verwende ich das menschliche Big Five Persönlichkeitsmodell als theoretische Grundlage für die Beschreibung von CA-Persönlichkeit. Diese Annahme ermöglicht es, Verhaltenshinweise, die mit menschlichen Persönlichkeitsmerkmalen assoziiert sind, in der psycholinguistischen Literatur sowie meiner eigenen Arbeit zu identifizieren. Diese Verhaltenshinweise übertrage ich dann auf CAs, um jeweils drei Ausprägungen von Verträglichkeit und Extraversion zu synthetisieren, die in vollständig funktionsfähigen text-basierten CAs implementiert sind. Eine empirische Untersuchung der Wahrnehmung dieser text-basierten CAs deutet darauf hin, dass menschliche Verhaltenshinweise genutzt werden können, um Verträglichkeit zu synthetisieren. Sie sind jedoch unzureichend, um den Eindruck von niedriger Extraversion zu vermitteln sowie die Persönlichkeit von CAs vollständig abzubilden. Aufgrund der mangelnden Eignung der menschlichen Persönlichkeitsbeschreibung entwickle ich einen zweiten Ansatz, in dem ich untersuche, ob das menschliche Big Five Modell für die Beschreibung von CA-Persönlichkeit genutzt werden kann. Zu diesem Zweck wende ich den psycholexikalischen Ansatz an, aus dem zehn Persönlichkeitsdimensionen hervorgehen, die nicht mit dem menschlichen Big Five Modell übereinstimmen. Folglich schlage ich diese zehn Dimensionen als eine alternative und vollständige Möglichkeit zur Beschreibung von CA-Persönlichkeit vor. Außerdem führe ich eine neue Methode, genannt Inszenierung-basiertes Dialogdesign, ein, die es ermöglicht, Persönlichkeit auf Grundlage dieser zehn Dimensionen zu synthetisieren. Um das zweite Ziel zu erreichen, stelle ich zwei Ansätze zur Untersuchung der Präferenzen von Benutzer:innen für CA-Persönlichkeit vor. In einem deduktiven Ansatz untersuche ich zunächst, ob Benutzer:innen eine niedrige, durchschnittliche oder hohe Ausprägung von vier verschiedenen Persönlichkeitsdimensionen in einem CA im Kontext unterschiedlicher Anwendungsfälle bevorzugen. Diese Untersuchungen zeigen, dass die Benutzer:innen sehr individuelle Präferenzen für die Dimensionen Extraversion und Sozial-Unterhaltend haben, während die Mehrheit CAs bevorzugt, die eine mittlere oder hohe Ausprägung in Verträglichkeit sowie eine niedrige Ausprägung in Konfrontativ aufweisen. Obgleich der deduktive Ansatz nützlich für die Evaluierung von CA-Prototypen ist, ermöglicht dieser es nicht, Bedürfnisse und Vorstellungen der Benutzer:innen einzufangen. Im zweiten, induktiven Ansatz präsentiere ich daher eine neue pragmatische Methode, um die Benutzer:innen besser in die Entwicklung von CA-Persönlichkeiten einzubinden. In diesem Zusammenhang untersuche ich darüber hinaus den Einfluss der Persönlichkeit der Benutzer:innen auf ihre Präferenzen für die CA-Persönlichkeit, finde jedoch nur einen begrenzten Effekt. Zusammenfassend leistet die vorliegende Arbeit die folgenden wissenschaftlichen Beiträge zur Ausstattung von CAs mit Persönlichkeit: (1) Theoretische Klarheit über die Notwendigkeit dedizierter Persönlichkeitsbeschreibungen für CAs, (2) eine Sammlung verbaler Verhaltenshinweise, die mit menschlicher Persönlichkeit assoziiert sind und in voll funktionsfähigen CA-Artefakten implementiert sind, (3) eine Exploration von zwei Methoden zur Synthese von Persönlichkeit in CAs und (4) eine neue Methode, um die Vision eines perfekten CAs von Benutzer:innen zu eruieren. Ich führe diese Methoden in einem benutzungszentrierten Designprozess für die Entwicklung von CA-Persönlichkeiten zusammen. Darüber hinaus liefere ich empirische Belege für divergierende Präferenzen der Benutzer:innen für CA-Persönlichkeit und erörtere übergreife Muster, die CA-Designer:innen anwenden können, um ihre CA-Persönlichkeiten auf individuelle Benutzer:innen zuzuschneiden. Abschließend wird eine Forschungsagenda für zukünftige Arbeiten präsentiert, welche die Herausforderungen diskutiert, die sich aus den vorgestellten Arbeiten ergeben

What Do We See in Them? Identifying Dimensions of Partner Models for Speech Interfaces Using a Psycholexical Approach

Perceptions of system competence and communicative ability, termed partner models, play a significant role in speech interface interaction. Yet we do not know what the core dimensions of this concept are. Taking a psycholexical approach, our paper is the first to identify the key dimensions that define partner models in speech agent interaction. Through a repertory grid study (N=21), a review of key subjective questionnaires, an expert review of resulting word pairs and an online study of 356 users of speech interfaces, we identify three key dimensions that make up a users’ partner model: 1) perceptions towards partner competence and dependability; 2) assessment of human-likeness; and 3) a system’s perceived cognitive flexibility. We discuss the implications for partner modelling as a concept, emphasising the importance of salience and the dynamic nature of these perceptions

What Do WEIRD and Non-WEIRD Conversational Agent Users Want and Perceive? Towards Transparent, Trustworthy, Democratized Agents

A majority of researchers who develop design guidelines have WEIRD, adult perspectives. This means we may not have technology developed appropriately for people from non-WEIRD countries and children. We present five design recommendations to empower designers to consider diverse users' desires and perceptions of agents. For one, designers should consider the degree of task-orientation of agents appropriate to end-users' cultural perspectives. For another, designers should consider how competence, predictability, and integrity in agent-persona affects end-users' trust of agents. We developed recommendations following our study, which analyzed children and parents from WEIRD and non-WEIRD countries' perspectives on agents as they create them. We found different subsets of participants' perceptions differed. For instance, non-WEIRD and child perspectives emphasized agent artificiality, whereas WEIRD and parent perspectives emphasized human-likeness. Children also consistently felt agents were warmer and more human-like than parents did. Finally, participants generally trusted technology, including agents, more than people.Comment: 24 pages, 13 figures, submitted to CHI 2023, for associated appendix: https://gist.github.com/jessvb/fa1d4c75910106d730d194ffd4d725d

Do users need human-like conversational agents? - Exploring conversational system design using framework of human needs

The fascinating story of human evolution can be attributed to our ability to speak, write, and communicate complex thoughts. When researchers envision a perfect, artificially intelligent conversational system, they want the system to be human-like. In other words, the system should converse with the same intellect and cognition as humans. Now, the question which we need to ask is if we need a human-like conversational system? Before we engage in the complex endeavor of implementing human-like characteristics, we should debate if the pursuit of such a system is logical and ethical. We analyze some of the system-level characteristics and discuss their merits and potential of harm. We review some of the latest work on conversational systems to understand how design features are evolving for Conversational Agents. Additionally, we look into the framework of human needs to assess how the system should assign relative importance to user requests, and prioritize user tasks. We draw on the peer work in human-computer interaction, sentiment analysis, and human psychology to provide insights into how future conversational agents should be designed for better user satisfaction

Report on the future conversations workshop at CHIIR 2021

The Future Conversations workshop at CHIIR’21 looked to the future of search, recommen- dation, and information interaction to ask: where are the opportunities for conversational interactions? What do we need to do to get there? Furthermore, who stands to benefit?The workshop was hands-on and interactive. Rather than a series of technical talks, we solicited position statements on opportunities, problems, and solutions in conversational search in all modalities (written, spoken, or multimodal). This paper –co-authored by the organisers and participants of the workshop– summarises the submitted statements and the discussions we had during the two sessions of the workshop. Statements discussed during the workshop are available at https://bit.ly/FutureConversations2021Statements

Exploring Data-Driven Components of Socially Intelligent AI through Cooperative Game Paradigms

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways

Real-World Evaluation of a University Guidance and Information Robot

We have developed a social robot to assist an existing support team in a large, recently-built university building designed for learning and teaching. Over the course of a week-long, supervised deployment, we collected long form questionnaire results (N = 59) on attitudes and feelings towards the robot from students and staff. We observed an overall positive response to the robot, but with a wide variety of specific opinions. We describe the limitations and challenges we found with the real-world deployment and outline next steps to allow an unsupervised deployment of the robot as part of the university’s wider service delivery strategy

The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners

Recent work has looked to understand user perceptions of speech agent capabilities as dialogue partners (termed partner models), and how this affects user interaction. Yet, currently partner model effects are inferred from language production as no metrics are available to quantify these subjective perceptions more directly. Through three studies, we develop and validate the Partner Modelling Questionnaire (PMQ): an 18-item self-report semantic differential scale designed to reliably measure people's partner models of non-embodied speech interfaces. Through principal component analysis and confirmatory factor analysis, we show that the PMQ scale consists of three factors: communicative competence and dependability, human-likeness in communication, and communicative flexibility. Our studies show that the measure consistently demonstrates good internal reliability, strong test-retest reliability over 12 and 4-week intervals, and predictable convergent/divergent validity. Based on our findings we discuss the multidimensional nature of partner models, whilst identifying key future research avenues that the development of the PMQ facilitates. Notably, this includes the need to identify the activation, sensitivity, and dynamism of partner models in speech interface interaction.Comment: Submitted (TOCHI