Proceedings of Gesture and Speech in Interaction (GESPIN)

Wagner P, Bryhadyr N. Mutual Visibility and Information Structure Enhance Synchrony between Speech and Co-Speech Movements. Journal of Multimodal Communication Studies. 2017;4(1-2):69-74

Does information-structural acoustic prosody change under different visibility conditions?

Wagner P, Bryhadyr N, Schröer M, Ludusan B. Does information-structural acoustic prosody change under different visibility conditions? In: Proceedings of ICPhS. 2019.It is well-known that the effort invested in prosodic expression can be adjusted to the information structure in a message, but also to the characteristics of the transmission channel. To investigate wether visibly accessible cues to information structure or facial prosodic expression have a differentiated impact on acoustic prosody, we modified the visibility conditions in a spontaneous dyadic interaction task, i.e. a verbalized version of TicTacToe. The main hypothesis was that visibly accessible cues should lead to a decrease in prosodic effort. While we found that - as expected - information structure is expressed throughout a number of acoustic-prosodic cues, visible accessibility to context information makes accents shorter, while accessability to an interlocutor's facial expression slightly increases the mean F0 of an accent

Pitch Accent Trajectories across Different Conditions of Visibility and Information Structure - Evidence from Spontaneous Dyadic Interaction

Wagner P, Bryhadyr N, Schröer M. Pitch Accent Trajectories across Different Conditions of Visibility and Information Structure - Evidence from Spontaneous Dyadic Interaction. In: Proceedings of Interspeech. 2019.Previous research identified a differential contribution of information structure and the visibility of facial and contextual information to the acoustic-prosodic expression of pitch accents. However, it is unclear whether pitch accent shapes are affected by these conditions as well. To investigate whether varying context cues have a differentiated impact on pitch accent trajectories produced in conversational interaction, we modified the visibility conditions in a spontaneous dyadic interaction task, i.e. a verbalized version of TicTacToe. Besides varying visibility, the game task allows for measuring the impact of information-structure on pitch accent trajectories, differentiating important and unpredictable game moves. Using GAMMs on four speaker groups (identified by a cluster analysis), we could isolate varying strategies of prosodic adaptation to contextual change. While few speaker groups showed a reaction to the availability of visible context cues (facial prosody or executed game moves), all groups differentiated the verbalization of unpredictable and predictable game moves with a groupspecific trajectory adaptation. The importance of game moves resulted in differentiated adaptations in two out of four speaker groups. The detected strategic trajectory adaptations were characterized by different characteristics of boundary tones, adaptations of the global f0-level, or the shape of the corresponding pitch accent

image70

https://dc.cod.edu/ilccoer2018images/1066/thumbnail.jp

Degrees of agreement using Indian head nods and bobble gestures

Jabeen F, Bryhadyr N, Wagner P. Degrees of agreement using Indian head nods and bobble gestures. Presented at the International Society of Gesture Studies Conference, Chicago, IL, USA.Head bobble is stereotypically associated with the people from the Indian Subcontinent. The interpretation of this gesture as conveying degrees of agreement has led to amusing anecdotes of cultural misunderstanding. However, there is no existing analysis of this gesture and how it differs from nods used to indicate agreement. Moreover, it is not clear if bobbles and nods are used only by listeners or speakers. This research investigates the use of bobbles and nods by Hindi speakers and listeners from India to indicate explicit agreement marked by “yes” or “Ok” and implicit agreement. We analyzed YouTube videos (90 minutes) of street interviews recorded in Delhi, India. The videos were annotated to mark the role of the interlocutors (Speakers: turn holders; Listeners: provided feedback with backchannel and/or stayed silent when another person talked) and the communicative contexts for each target gesture. Overall, speakers used nods more often (66%) then listeners, whereas bobbles were used both by listeners and speakers with equal frequency. Both listeners and speakers used nods to mark explicit agreement more frequently (Listeners: 63%, Speakers: 86%) than implicit agreement. As for bobbles, listeners used them slightly more frequently to convey implicit (54%) as compared with explicit agreement. Contrarily, speakers used bobbles overwhelmingly to mark explicit agreement (92%). This leads us to claim that (1) Hindi-speaking Indian listeners and speakers prefer to use nods to convey explicit agreement and (2) bobbles are used for implicit agreement by speakers but implicit agreement by listeners. This shows that there is a functional split in the use of nods and bobbles that depends on the role of interlocutors. Moreover, there is no one-to-one relation between target gestures, interlocutors’ roles, and the communicative contexts. This supports the claim that bobbles are not as clearly associated with strong agreement as nods are

Yes or no: An analysis of Indian head gestures

Jabeen F, Bryhadyr N, Wagner P. Yes or no: An analysis of Indian head gestures. Presented at the International Society of Gesture Studies Conference, Chicago, IL, USA.Head bobble is stereotypically associated with people from the Indian Subcontinent. The apparent substitution of bobbles with nods and shakes is anecdotally reported to be problematic for non- Indian interlocutors. However, there is no existing analysis of bobbles and how they differ from nods and shakes generally believed to indicate agreement and disagreement respectively. Moreover, it is not clear if these gestures are used exclusively by listeners or speakers. This study investigates the use of nods, shakes, and bobbles by Hindi speakers and listeners from India and reports the communicative contexts used for these gestures. We analyzed YouTube videos (90 minutes) of street interviews recorded in Delhi, India. The videos were annotated to mark interlocutors’ roles (Speakers: turn holders; Listeners: provided feedback with backchannel and/or stayed silent when another person talked) and communicative contexts for each gesture (N = 755). The analysis showed that speakers used bobbles (63%) and shakes (64%) more frequently than listeners, whereas nods were used by speakers and listeners with similar frequency (49%, 51% respectively). As for the contexts, listeners overwhelmingly used nods and bobbles for agreement/backchanneling (Nods: 97%, Bobbles: 91%). Furthermore, listeners used shakes for agreement/backchanneling (71%) as well as in the context of negation (14.5%), and while being presented with new information (14.5%). However, speakers predominantly used shakes to convey negation (63%) as well as when providing information (13.5%) or elaborating on it (13.5%). Nods and bobbles were not preferred in a particular context and speakers used them for agreement/backchanneling (Nods: 38%, Bobbles: 38.5%), elaboration (Nods: 28%, 23%), providing information (Nods: 29%, Bobbles: 36%), and negation (Nods: 5%, Bobbles: 2.5%). This data indicates that shakes are clearly associated with negation as compared with nods and bobbles. The interchangeable use of nods and bobbles explains the confusion faced by non-Indians when communicating with the Indian interlocutors

A crosslinguistic study on the interplay of fillers and silences

Betz S, Bryhadyr N, Kosmala L, Schettino L. A crosslinguistic study on the interplay of fillers and silences. In: Proceedings of DiSS 2021. 2021.We present a crosslinguistic study on the interplay of hesitation silences and fillers in conversation. The research questions have been addressed for English in a previous DiSS workshop paper (Betz & Kosmala, 2019) and this study extends the analysis to German, Italian and French. The research questions are: 1) Does the type of the filler influence following silence duration 2) Does the duration of the filler correlate with silence duration 3) Does silence duration vary depending on its distance from filler. The analysis shows cross-linguistic similarities and differences, thus highlighting the role and the language- and culture-specific nature of disfluencies

Cognitive Load Increases Spoken and Gestural Hesitation Frequency

Betz S, Bryhadyr N, Türk O, Wagner P. Cognitive Load Increases Spoken and Gestural Hesitation Frequency. Languages. 2023;8(1): 71.This study investigates the interplay of spoken and gestural hesitations under varying amounts of cognitive load. We argue that not only fillers and silences, as the most common hesitations, are directly related to speech pausing behavior, but that hesitation lengthening is as well. We designed a resource-management card game as a method to elicit ecologically valid pausing behavior while being able to finely control cognitive load via card complexity. The method very successfully elicits large amounts of hesitations. Hesitation frequency increases as a function of cognitive load. This is true for both spoken and gestural hesitations. We conclude that the method presented here is a versatile tool for future research and we present foundational research on the speech-gesture link related to hesitations induced by controllable cognitive load

A Crosslinguistic Study on the Interplay of Fillers and Silences

We present a crosslinguistic study on the interplay of hesitation silences and fillers in conversation. The research questions have been addressed for English in a previous DiSS workshop paper (Betz & Kosmala, 2019) and this study extends the analysis to German, Italian and French. The research questions are: 1) Does the type of the filler influence following silence duration 2) Does the duration of the filler correlate with silence duration 3) Does silence duration vary depending on its distance from filler. The analysis shows cross-linguistic similarities and differences, thus highlighting the role and the language- and culture-specific nature of disfluencies