On what happens in gesture when communication is unsuccessful

Previous studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director–matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback

The multimodal nature of communicative efficiency in social interaction

How does communicative efficiency shape language use? We approach this question by studying it at the level of the dyad, and in terms of multimodal utterances. We investigate whether and how people minimize their joint speech and gesture efforts in face-to-face interactions, using linguistic and kinematic analyses. We zoom in on other-initiated repair—a conversational microcosm where people coordinate their utterances to solve problems with perceiving or understanding. We find that efforts in the spoken and gestural modalities are wielded in parallel across repair turns of different types, and that people repair conversational problems in the most cost-efficient way possible, minimizing the joint multimodal effort for the dyad as a whole. These results are in line with the principle of least collaborative effort in speech and with the reduction of joint costs in non-linguistic joint actions. The results extend our understanding of those coefficiency principles by revealing that they pertain to multimodal utterance design

Referring Strategies in American Sign Language and English (with Co-speech Gesture): The Role of Modality in Referring to Non-nameable Objects

American Sign Language (ASL) and English differ in linguistic resources available to express visual–spatial information. In a referential communication task, we examined the effect of language modality on the creation and mutual acceptance of reference to non-nameable figures. In both languages, description times reduced over iterations and references to the figures’ geometric properties (“shape-based reference”) declined over time in favor of expressions describing the figures’ resemblance to nameable objects (“analogy-based reference”). ASL signers maintained a preference for shape-based reference until the final (sixth) round, while English speakers transitioned toward analogy-based reference by Round 3. Analogy-based references were more time efficient (associated with shorter round description times). Round completion times were longer for ASL than for English, possibly due to gaze demands of the task and/or to more shape-based descriptions. Signers’ referring expressions remained unaffected by figure complexity while speakers preferred analogy-based expressions for complex figures and shape-based expressions for simple figures. Like speech, co-speech gestures decreased over iterations. Gestures primarily accompanied shape-based references, but listeners rarely looked at these gestures, suggesting that they were recruited to aid the speaker rather than the addressee. Overall, different linguistic resources (classifier constructions vs. geometric vocabulary) imposed distinct demands on referring strategies in ASL and English

How What We See and What We Know Influence Iconic Gesture Production

In face-to-face communication, speakers typically integrate information acquired through different sources, including what they see and what they know, into their communicative messages. In this study, we asked how these different input sources influence the frequency and type of iconic gestures produced by speakers during a communication task, under two degrees of task complexity. Specifically, we investigated whether speakers gestured differently when they had to describe an object presented to them as an image or as a written word (input modality) and, additionally, when they were allowed to explicitly name the object or not (task complexity). Our results show that speakers produced more gestures when they attended to a picture. Further, speakers more often gesturally depicted shape information when attended to an image, and they demonstrated the function of an object more often when they attended to a word. However, when we increased the complexity of the task by forbidding speakers to name the target objects, these patterns disappeared, suggesting that speakers may have strategically adapted their use of iconic strategies to better meet the task’s goals. Our study also revealed (independent) effects of object manipulability on the type of gestures produced by speakers and, in general, it highlighted a predominance of molding and handling gestures. These gestures may reflect stronger motoric and haptic simulations, lending support to activation-based gesture production accounts

On what happens in gesture when communication is unsuccessful

Previous studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director–matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback

Meaning and context in the gestural communication of wild bilia (bonobo: Pan paniscus)

Studying the communication of our closest living relatives, the great apes, can inform our understanding of language evolution. Great ape gestural communication has been well-documented in captivity, but less so in the wild, with the exception of the chimpanzee. My research on the gestural communication of wild bonobos (at Wamba, Democratic Republic of the Congo) aims to fill one of the gaps in our knowledge. In my thesis, I first describe the gestural repertoire of wild bonobos –the physical form of the gestures that they use. The Wamba communities of wild bonobos use 68 gesture types. I then look at the meaning of gestures by analysing the Apparently Satisfactory Outcome (ASO) that they achieve. Of the gesture types that are suitable for analysis, about half have only one ASO, while the other half achieve multiple ASOs. Where these meanings are ambiguous, with one gesture type achieving multiple ASOs, I look at potential modifiers: syntax-like sequence ordering, and behavioural and interpersonal context. There is no effect of sequence order on the meaning of gestures; rather, the behavioural and interpersonal context explains the apparent ambiguity. Gesture types mean different things in different contexts. Finally, I take my findings and compare them to data from wild chimpanzees at Budongo, Uganda. The gestural repertoire (the physical form of the gestures) overlaps by 88-96%, and many ASOs are achieved by the same gesture types. However, the distribution of gesture types for each ASO is different between species, possibly as a result of different contexts arising from differences in social behaviour

Gesture and prosody: cognitive and communicative effort in L1 and L2

Gestures and speech are two interconnected features of human communication. Studies have shown that they develop together and they both covey semantic and pragmatic meanings (Kendon, 2004; McNeill, 1992). Furthermore, gestures seem to be also linked to the prosodic features of speech since they also share synchronicity aspects and are often temporally aligned (McClave, 1991; Esteve-Gibert & Prieto, 2013). The co-production of gestures and speech, thus, seems to have a number of different functions both at the cognitive and at the communicative levels. On the one hand, in fact, gestures seem to help the process of speaking: they have a scaffolding function in the lexical and rhythmical organization of speech and they help information packaging (Butterworth & Beattie, 1978; Esteve-Gibert et al., 2014; Krauss et al, 1996; Kita, 2000, 2010). This is supported by the evidence that speakers gesticulate also when they don’t see their addressees (for example when they are speaking over the phone (Cohen & Harrison, 1973; de Ruiter, 2003). On the other hand, gesturing is intended to communicate and is part of the speaker’s communicative effort (Kendon, 2004). Gestures also play an important role in L2 development and communicative strategies (Gullberg, 1998) and it is possible that L1 gestures influence gestures during L2 language development, and that gestural transfer co-occurs with linguistic transfer (Brown & Gullberg, 2008; Pika, Nicoladis, Marentette, 2006). It has also been suggested that, since gestures play an important role in facilitating language access in speech production, bilingual/L2 speakers may gesture more than monolingual speakers. This would be due to the cognitive load deriving from speaking a different language from the native one (Kita, 2000; Krauss & Hadar, 1999). Until now, gestures and prosody have been studied mostly in their temporal interaction and coordination (McClave, 1991, Loehr, 2004) or in the possible similarities between their pragmatic functions (Tuite, 1993). Little attention has been paid to the relationship between speakers’ global pitch range and use of gestures (and gesture categories) in conditions of high cognitive effort, or when they increase their communicative effort in their speeches. In fact, there seem to be a lack of scientific evidence about the use of pitch and gestures variability as a communicative strategy of the speaker. The aim of this thesis is to investigate aspects of the role and functions of gestures and speech in L1 and L2 and how the cognitive and communicative efforts may influence speakers’ global pitch range and the use of gestures in a story telling task. The investigation consists of two experiments. The first one (presented in chapter 5) aims to examine if speakers’ increase in the communicative effort produces changes in pitch variation and in the use of different categories of gestures. The second one (presented in chapter 6) examines the effects of speakers’ decrease in cognitive effort in the production of speech and gestures. In the first experiment 8 Italian speakers, Italian learners of English (L2) and students of a Public Speaking class, were asked to read and tell a fable in English to their classmates (Italian was used only as a control and recorded only one time). One week later, the subjects were asked to repeat the task with the instruction to be as communicative as possible. The audiovisual material was analyzed with the software Praat (phonetic analysis) and Elan (gesture analysis). For speech, the aim was to examine possible changes in the speakers’ fluency and pitch variation after they received the instruction to be communicative; for gestures, the aim was to examine the variations in both speakers’ overall gesturing and representational gestures in the communicative task. The hypothesis tested was that the communicative task causes an increase in fluency in the L2 (shown through features like a higher speech rate and a decrease of disfluencies and pauses), and that the communicative task leads to an increased use of representational gestures, since they might be considered as helpful for the addressee to better understand what is said. The results of the first experiment led to the conclusion that if a person is asked to be communicative in the L2, they will probably increase both their pitch variation and the total number of gestures produced, with a significant increase in iconic and representational gestures. The results on pitch variation, though, do not exclude the possible effect of other variables, first of all of task repetition. In fact, it is possible that by repeating the task, the speakers became more confident and could be more focused on being more communicative in telling the story. The second experiment tests the effect of task repetition on L2 speech and gesture. In this experiment (chapter 6) only the cognitive facilitation and the effect of the decrease in cognitive load was considered. This time, in fact, the subjects (10 Italian students of English L2) were asked to watch a short cartoon and to tell the story, both in Italian and English, in front of a small audience while being video-recorded. The subjects were asked to repeat the task one week later in the two languages with the same modalities. The analyses followed the same procedure as the previous experiment (chapter 5). The hypotheses tested were that repetition itself could influence the communicativeness of the speakers in both the speech and the gesturing levels, with an increased fluency and a different use of gestures compared to the first narration attempt. This could be caused by the facilitation of the task in terms of cognitive load and better memorization. The results showed that with the decrease of the cognitive load, speakers reach a higher fluency in L2. In Italian, however, the speakers show no significant difference in fluency or liveliness (Hincks, 2005) and this might be due to the fact that repetition and memorization did not help their fluency as much as they did in the L2. As for gestures, the speakers employed a greater number overall gestures when speaking English, their L2. The use of gestures, though, did not change in spite of the repetition, with a greater use of discursive gestures even if representational gestures seem to be used more in the L2 than in Italian. Overall, the investigation shows that the results of the two experiments can be integrated and offer a wider picture on the use of gesture and the employment of representationality with the conscious intent to be communicative. When people are asked to tell a story in a more communicative way, their response is to increase the representationality carried in their gestures and consequently use more representational gestures, this does not occur whenever the subjects only repeat the narration with a lighter cognitive load

The interactive ecology of construal in gesture: a microethnographic analysis of peer learning at an EMI university in China

Depictive manual gestures do not appear in isolation, but are motivated by a complex of experiential knowledge, communicative goals, and contextual-environmental factors (Harrison 2018; Kendon 2004; Müller 2014; Streeck 1993, 1994, 2009b). However, little is known about the incremental, moment-by-moment formulation of depictions in elaborate sequences of talk. Furthermore, questions endure about depiction as a learning resource within the contingent interactivity of the foreign language academic classroom. This study explores these questions in the context of subject-related student talk at a Sino-foreign university in China by focusing on how gesturers build expositions through intercorporeal and intersubjective sense making (cf. Merleau-Ponty 1945/2012). Drawing on empirical material from the corpus of Chinese Academic Written and Spoken English (CAWSE), I aim to contribute greater understanding of the intersubjective ecology of depictive gesturing. The study builds on previous research on depictive gestures in the classroom (e.g. Rosborough 2014; Roth & Lawless 2002) by focusing on sequences of gesturing within two distinct classroom tasks: i) dialogic explanations of complex systems and ii) interactional multi-party group discussions. By converging theories of intersubjectivity drawing on Cognitive Grammar (e.g. Langacker 2008; Blomberg & Zlatev 2014) and Conversation Analysis (Heritage & Atkinson 1984; Schegloff 1992), I use microethnography for the investigation of gesture as a cognitive practice (Streeck 2009b; cf. Erickson 1995; Streeck & Mehus 2005). The analysis engages concepts in phenomenology, ecological cognition and enactivism in order to illustrate the publicly displayable achievement of enactive construal in spoken exposition. These analyses expose the ways that speakers depict for intersubjective visualization of the topic-at-hand, and anticipate and react to affordances that occur within the landscape of interaction. Speakers design their depictions, by manipulating construal dimensions in three ways: i) depictions are integrated into the exposition for projecting and delimiting epistemic arenas where construal relations are tailored for specific structural aspects of the depictions, ii) depictions invite participatory frameworks for co-analysis of the topic-at-hand, and iii) speakers refashion their depictions to anticipate previous trouble. Furthermore, the analysis of the interactional order of the tasks illustrates the intercorporeality, the pre-reflective disposition towards sense-making, of construal in the moment-by-moment construction of academic classroom talk. This study has implications that problematize the notion of the body as a communicative resource by obscuring the notions of planning and strategy. Overall, the analysis shows that explanations and discussions involve finely grained attenuation of the corporeal dimensions of spoken language