Disordered speech disrupts conversational entrainment: a study of acoustic-prosodic entrainment and communicative success in populations with communication challenges

abstract: Conversational entrainment, a pervasive communication phenomenon in which dialogue partners adapt their behaviors to align more closely with one another, is considered essential for successful spoken interaction. While well-established in other disciplines, this phenomenon has received limited attention in the field of speech pathology and the study of communication breakdowns in clinical populations. The current study examined acoustic-prosodic entrainment, as well as a measure of communicative success, in three distinctly different dialogue groups: (i) healthy native vs. healthy native speakers (Control), (ii) healthy native vs. foreign-accented speakers (Accented), and (iii) healthy native vs. dysarthric speakers (Disordered). Dialogue group comparisons revealed significant differences in how the groups entrain on particular acoustic–prosodic features, including pitch, intensity, and jitter. Most notably, the Disordered dialogues were characterized by significantly less acoustic-prosodic entrainment than the Control dialogues. Further, a positive relationship between entrainment indices and communicative success was identified. These results suggest that the study of conversational entrainment in speech pathology will have essential implications for both scientific theory and clinical application in this domain.View the article as published at http://journal.frontiersin.org/article/10.3389/fpsyg.2015.01187/ful

Disordered speech disrupts conversational entrainment: a study of acoustic-prosodic entrainment and communicative success in populations with communication challenges

Conversational entrainment, a pervasive communication phenomenon in which dialogue partners adapt their behaviors to align more closely with one another, is considered essential for successful spoken interaction. While well-established in other disciplines, this phenomenon has received limited attention in the field of speech pathology and the study of communication breakdowns in clinical populations. The current study examined acoustic-prosodic entrainment, as well as a measure of communicative success, in three distinctly different dialogue groups: (i) healthy native vs. healthy native speakers (Control), (ii) healthy native vs. foreign-accented speakers (Accented), and (iii) healthy native vs. dysarthric speakers (Disordered). Dialogue group comparisons revealed significant differences in how the groups entrain on particular acoustic–prosodic features, including pitch, intensity, and jitter. Most notably, the Disordered dialogues were characterized by significantly less acoustic-prosodic entrainment than the Control dialogues. Further, a positive relationship between entrainment indices and communicative success was identified. These results suggest that the study of conversational entrainment in speech pathology will have essential implications for both scientific theory and clinical application in this domain

Acoustic-Prosodic Entrainment in Human-Human and Human-Computer Dialogue

Entrainment (sometimes called adaptation or alignment) is the tendency of human speakers to adapt to or imitate characteristics of their interlocutors' behavior. This work focuses on entrainment on acoustic-prosodic features. Acoustic-prosodic entrainment has been extensively studied but is not well understood. In particular, it is difficult to compare the results of different studies, since entrainment is usually measured in different ways, reflect- ing disparate conceptualizations of the phenomenon. In the first part of this thesis, we look for evidence of entrainment on a variety of acoustic-prosodic features according to various conceptualizations, and show that human speakers of both Standard American English and Mandarin Chinese entrain to each other globally and locally, in synchrony, and that this entrainment can be constant or convergent. We explore the relationship between entrainment and gender and show that entrainment on some acoustic-prosodic features is related to social behavior and dialogue coordination. In addition, we show that humans entrain in a novel domain, backchannel-inviting cues, and propose and test a novel hypothesis: that entrainment will be stronger in the case of an outlier feature value. In the second part of the thesis, we describe a method for flexibly and dynamically entraining a TTS voice to multiple acoustic-prosodic features of a user's input utterances, and show in an exploratory study that users prefer an entraining avatar to one that does not entrain, are more likely to ask its advice, and choose more positive adjectives to describe its voice. This work introduces a coherent view of entrainment in both familiar and novel domains. Our results add to the body of knowledge of entrainment in human-human conversations and propose new directions for making use of that knowledge to enhance human-computer interactions

Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Investigating Fine Temporal Dynamics of Prosodic and Lexical Accommodation

Conversational interaction is a dynamic activity in which participants engage in the construction of meaning and in establishing and maintaining social relationships. Lexical and prosodic accommodation have been observed in many studies as contributing importantly to these dimensions of social interaction. However, while previous works have considered accommodation mechanisms at global levels (for whole conversations, halves and thirds of conversations), this work investigates their evolution through repeated analysis at time intervals of increasing granularity to analyze the dynamics of alignment in a spoken language corpus. Results show that the levels of both prosodic and lexical accommodation fluctuate several times over the course of a conversation

Global analysis of entrainment in dialogues

This paper performs a global analysis of entrainment between dyads in map-task dialogues in European Portuguese (EP), including 48 dialogues, between 24 speakers. Our main goals focus on the acoustic-prosodic similarities between speakers, namely if there are global entrainment cues displayed in the dialogues, if there are degrees of entrainment manifested in distinct sets of features shared amongst the speakers, if entrainment depends on the role of the speaker as either giver or follower, and also if speakers tend to entrain more with specific pairs regardless of the role. Results show global entrainment in almost all the dyads, but the degrees of entrainment (stronger within the same gender), and the role effects tend to be less striking than the interlocutors’ effect. Globally, speakers tend to be more similar to their own speech in other dialogues than to their partners. However, speakers are also more similar to their interlocutors than to speakers with whom they never spoke.info:eu-repo/semantics/publishedVersio

A Review of the Research on Linguistic Prosody in Conversation Entrainment

Having conversation is a joint action in which interacting individuals coordinate their behavior and adapt their linguistic choices to each other in order to make smooth and successful communication. This coordination or adaptation is called entrainment. In daily conversation, interlocutors entrain at linguistic or nonlinguistic levels. The present research focuses on linguistic prosody in entrainment, and reviews the theoretical and empirical studies of linguistic prosody in conversation entrainment in order to supply references for future studies in the similar fields

Conversational Alignment: A Study of Neural Coherence and Speech Entrainment

Conversational alignment refers to the tendency for communication partners to adjust their verbal and non-verbal behaviors to become more like one another during the course of human interaction. This alignment phenomenon has been observed in neural patterns, specifically in the prefrontal areas of the brain (Holper et al., 2013; Cui et al., 2012; Dommer et al., 2012; Holper et al., 2012; Funane et al., 2011; Jiang et al., 2012); verbal behaviors such acoustic speech features (e.g., Borrie & Liss, 2014; Borrie et al., 2015; Lubold & Pon-Barry, 2014), phonological features (e.g., Babel, 2012; Pardo, 2006), lexical selection (e.g., Brennan & Clark, 1996; Garrod & Anderson, 1989), syntactic structure (e.g., Branigan, Pickering, & Cleland, 2000; Reitter, Moore, & Keller, 2006); and motor behaviors including body posture, facial expressions and breathing rate (e.g., Furuyama, Hayashi, & Mishima, 2005; Louwerse, Dale, Bard, & Jeuniaux, 2012; Richardson, March, & Schmit, 2005; Shockley, Santana, & Fowler, 2003; McFarland, 2001). While conversational alignment in itself, is a largely physical phenomenon, it has been linked to significant functional value, both in the cognitive and social domains. Cognitively, conversational alignment facilitates spoken message comprehension, enabling listeners to share mental models (Garrod & Pickering, 2004) and generate temporal predictions about upcoming aspects of speech. From a social perspective, behavioral alignment has been linked with establishing turn-taking behaviors, and with increased feelings of rapport, empathy, and intimacy between conversational pairs (e.g., Lee et al. 2010; Nind, & Macrae, 2009; Smith, 2008; Bailenson & Yee, 2005; Chartrand & Barg, 1999; Miles, Putman & Street, 1984; Street & Giles, 1982). Benus (2014), for example, observed that individuals who align their speech features are perceived as more socially attractive and likeable, and have interactions that are more successful. These cognitive and social benefits, associated with conversational alignment, have been observed in both linguistic and neural data (e.g., Holper et al., 2012; 2013, Cui et al. 2012; Jiang et al., 2012; Egetemeir et al., 2011; Stephens et al. 2010). The purpose of the current study was to examine conversational alignment as a multi-level communication phenomenon, by examining the relationship between neural and speech behaviors. To assess neural alignment, we used Near-Infrared Spectroscopy (NIRS), a non-invasive neuroimaging technology that detects cortical increases and decreases in the concentration of oxygenated and deoxygenated hemoglobin at multiple measurement sites to determine the rate that oxygen is being released and absorbed (Ferrari & Quaresima, 2012). While still considered a relatively new neural imaging technique, NIRS has been well established as an efficacious and effective data collection approach, particularly appropriate for social interaction research (e.g., Holper et al., 2013; Jiang et al., 2012; Holper et al., 2012; Suda et al., 2010). We utilized hyperscanning, a technique that allows for the quantitation of two simultaneous signals, allowing us to document neural alignment between two individuals (Babiloni & Astolfi, 2012). Recent studies have revealed neural alignment between two persons in cooperative states, including alignment in the right superior frontal cortices and medial prefrontal regions (Cui et al., 2012; Dommer et al., 2012; Funane et al., 2011). This increased prefrontal interbrain alignment has also been observed in other social interactions, including joint attention tasks (Dommer et al., 2012), imitation tasks (Holper et al., 2012), competitive games (Cheng et al., 2015, Duan et al., 2013), teaching-learning interactions (Holper et al., 2013), face- to-face communication (Jiang et al., 2012), mother-child interactions (Hirata et al., 2014), and during cooperative singing tasks (Osaka et al., 2015). Interestingly, Jiang et al. (2012) showed that increased neural alignment only occurred between conversational participants when they were speaking face-to-face, but not when participants had their backs facing one another. The authors speculated that the multi-sensory information, for example motor behaviors such as gestures, was required for neural alignment to occur

Universal and language-specific processing : the case of prosody

A key question in the science of language is how speech processing can be influenced by both language-universal and language-specific mechanisms (Cutler, Klein, & Levinson, 2005). My graduate research aimed to address this question by adopting a crosslanguage approach to compare languages with different phonological systems. Of all components of linguistic structure, prosody is often considered to be one of the most language-specific dimensions of speech. This can have significant implications for our understanding of language use, because much of speech processing is specifically tailored to the structure and requirements of the native language. However, it is still unclear whether prosody may also play a universal role across languages, and very little comparative attempts have been made to explore this possibility. In this thesis, I examined both the production and perception of prosodic cues to prominence and phrasing in native speakers of English and Mandarin Chinese. In focus production, our research revealed that English and Mandarin speakers were alike in how they used prosody to encode prominence, but there were also systematic language-specific differences in the exact degree to which they enhanced the different prosodic cues (Chapter 2). This, however, was not the case in focus perception, where English and Mandarin listeners were alike in the degree to which they used prosody to predict upcoming prominence, even though the precise cues in the preceding prosody could differ (Chapter 3). Further experiments examining prosodic focus prediction in the speech of different talkers have demonstrated functional cue equivalence in prosodic focus detection (Chapter 4). Likewise, our experiments have also revealed both crosslanguage similarities and differences in the production and perception of juncture cues (Chapter 5). Overall, prosodic processing is the result of a complex but subtle interplay of universal and language-specific structure

Prosodic Entrainment in Mandarin and English: A Cross-Linguistic Comparison

Entrainment is the propensity of speakers to begin behaving like one another in conversation. We identify evidence of entrainment in a number of acoustic and prosodic dimensions in conversational speech of Standard American English speakers and Mandarin Chinese speakers. We compare entrainment in the Columbia Games Corpus and the Tongji Games Corpus and find similar patterns of global and local entrainment in both. Differences appear primarily in global convergence