Investigating Automatic Measurements of Prosodic Accommodation and Its Dynamics in Social Interaction

Spoken dialogue systems are increasingly being used to facilitate and enhance human communication. While these interactive systems can process the linguistic aspects of human communication, they are not yet capable of processing the complex dynamics involved in social interaction, such as the adaptation on the part of interlocutors. Providing interactive systems with the capacity to process and exhibit this accommodation could however improve their efficiency and make machines more socially-competent interactants. At present, no automatic system is available to process prosodic accommodation, nor do any clear measures exist that quantify its dynamic manifestation. While it can be observed to be a monotonically manifest property, it is our hypotheses that it evolves dynamically with functional social aspects. In this paper, we propose an automatic system for its measurement and the capture of its dynamic manifestation. We investigate the evolution of prosodic accommodation in 41 Japanese dyadic telephone conversations and discuss its manifestation in relation to its functions in social interaction. Overall, our study shows that prosodic accommodation changes dynamically over the course of a conversation and across conversations, and that these dynamics inform about the naturalness of the conversation flow, the speakers’ degree of involvement and their affinity in the conversation

Investigating Fine Temporal Dynamics of Prosodic and Lexical Accommodation

Conversational interaction is a dynamic activity in which participants engage in the construction of meaning and in establishing and maintaining social relationships. Lexical and prosodic accommodation have been observed in many studies as contributing importantly to these dimensions of social interaction. However, while previous works have considered accommodation mechanisms at global levels (for whole conversations, halves and thirds of conversations), this work investigates their evolution through repeated analysis at time intervals of increasing granularity to analyze the dynamics of alignment in a spoken language corpus. Results show that the levels of both prosodic and lexical accommodation fluctuate several times over the course of a conversation

Investigating Prosodic Accommodation in Clinical Interviews with Depressed Patients

Six in-depth clinical interviews, involving six elderly female patients (aged 60+) and one female psychiatrist, were recorded and analysed for a number of prosodic accommodation variables. Our analysis focused on pitch, speaking time, and vowel-space ratio. Findings indicate that there is a dynamic manifestation of prosodic accommodation over the course of the interactions. There is clear adaptation on the part of the psychiatrist, even going so far as to have a reduced vowel-space ratio, mirroring a reduced vowel-space ratio in the depressed patients. Previous research has found a reduced vowel-space ratio to be associated with psychological distress; however, we suggest that it indicates a high level of adaptation on the part of the psychiatrist and needs to be considered when analysing psychiatric clinical interactions

Providing Objective Metrics of Team Communication Skills via Interpersonal coordination Mechanisms

Being able to communicate efficiently has been acknowledged as a vital skill in many different domains. In particular, team communication skills are of key importance in the operation of complex machinery such as aircrafts, maritime vessels and such other, highly-specialized, civilian or military vehicles, as well as the performance of complex tasks in the medical domain. In this paper, we propose to use prosodic accommodation and turn- taking organisation to provide objective metrics of communica- tion skills. To do this, human-factors evaluations, via a coordi- nation Demand Analysis (CDA), were used in conjunction with a dynamic model of prosodic accommodation and turn-taking organisation. Using conversational speech from airline pilots involved in a collaborative task (decision-making exercise), our study reveals that interpersonal coordination mechanisms are indicative of human evaluation of pilots’ communication skills. We discuss our results in terms of relevance for training simu- lation for personnel in safety or mission critical environment

A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology Applications

Inter-speaker accommodation is a well-known property of human speech and human interaction in general. Broadly it refers to the behavioural patterns of two (or more) interactants and the effect of the (verbal and non-verbal) behaviour of each to that of the other(s). Implementation of thisbehavior in spoken dialogue systems is desirable as an improvement on the naturalness of humanmachine interaction. However, traditional qualitative descriptions of accommodation phenomena do not provide sufficient information for such an implementation. Therefore, a quantitativedescription of inter-speaker accommodation is required. This thesis proposes a methodology of monitoring accommodation during a human or humancomputer dialogue, which utilizes a moving average filter over sequential frames for each speaker. These frames are time-aligned across the speakers, hence the name Time Aligned Moving Average (TAMA). Analysis of spontaneous human dialogue recordings by means of the TAMA methodology reveals ubiquitous accommodation of prosodic features (pitch, intensity and speech rate) across interlocutors, and allows for statistical (time series) modeling of the behaviour, in a way which is meaningful for implementation in spoken dialogue system (SDS) environments.In addition, a novel dialogue representation is proposed that provides an additional point of view to that of TAMA in monitoring accommodation of temporal features (inter-speaker pause length and overlap frequency). This representation is a percentage turn distribution of individual speakercontributions in a dialogue frame which circumvents strict attribution of speaker-turns, by considering both interlocutors as synchronously active. Both TAMA and turn distribution metrics indicate that correlation of average pause length and overlap frequency between speakers can be attributed to accommodation (a debated issue), and point to possible improvements in SDS “turntaking” behaviour. Although the findings of the prosodic and temporal analyses can directly inform SDS implementations, further work is required in order to describe inter-speaker accommodation sufficiently, as well as to develop an adequate testing platform for evaluating the magnitude ofperceived improvement in human-machine interaction. Therefore, this thesis constitutes a first step towards a convincingly useful implementation of accommodation in spoken dialogue systems

A Review of the Research on Linguistic Prosody in Conversation Entrainment

Having conversation is a joint action in which interacting individuals coordinate their behavior and adapt their linguistic choices to each other in order to make smooth and successful communication. This coordination or adaptation is called entrainment. In daily conversation, interlocutors entrain at linguistic or nonlinguistic levels. The present research focuses on linguistic prosody in entrainment, and reviews the theoretical and empirical studies of linguistic prosody in conversation entrainment in order to supply references for future studies in the similar fields

An investigation of therapeutic rapport through prosody in brief psychodynamic psychotherapy

Therapeutic alliance, a concept closely related to rapport, is one of the most important variables in psychotherapy. High degrees of synchrony/coordination in the therapeutic session are considered to contribute to rapport, and have received attention in the psychotherapy literature. Coordinative behaviours are observable in speech, and they manifest in phenomena such as prosodic accommodation, a dynamic phenomenon closely related to conversational success. A preliminary investigation of interpersonal prosodic dynamics in psychotherapy was performed on a database obtained in collaboration with the University of Padua, consisting of 16 recordings making up the entire course of a brief psychodynamic psychotherapy intervention for a 25 year old female volunteer and a 41 years old male psychotherapist. The data was analysed with Time Aligned Moving Averages, a method commonly used in interpersonal speech research. Issues of data sparsity are discussed, and preliminary results on the relationship between empathy and anxiety with interpersonal speech dynamics are presented

A Study of Prosodic Entrainment and Social Factors in Mandarin Conversations

In conversations, interlocutors usually adopt prosody to that of their partner, and they become similar in prosodic production for successful communication. This phenomenon of prosodic entrainment is related to complex factors. This study aims to explore the relationship between prosodic entrainment and social factors. Two analyses are accomplished: the analysis of prosodic entrainment and gender, and the analysis of prosodic entrainment and role. In terms of prosodic entrainment and gender, it is found that the most prosodic features are entrained in female-male conversations, and the least in male-male conversations. In terms of prosodic entrainment and roles, it is found that different roles have influence on the entrainment degree, and information givers entrain more to followers in conversation

The Influence of Conversation Role on Prosodic Entrainment in Mandarin Interactions

The aim of this study is to find out how conversation role affects prosodic entrainment in Mandarin interactions. Tongji Games Corpus are adopted for this study, in which two interlocutors play unequal roles in Picture Ordering Games (information giver and information follower), and equal roles in Picture Classifying Games. Based on this corpus, two tests are conducted in this study, the Role Influence Test and Role Direction Test. In the analysis of Role Influence Test, it is found that the entrainment degree in Picture Ordering Games is significantly bigger than that in the Picture Classifying Games. In the analysis of Role Direction Test, it is found that information givers entrain more to followers in conversation. These findings provide evidences that conversation roles, as one type of the social factors, have influence on the degree and direction of prosodic entrainment in Mandarin interactions