On what happens in speech and gesture when communication is unsuccessful

Previous studies found that repeated references in successful communication are often reduced, not only at the acoustic level, but also in terms of words and manual co-speech gestures. In the present study, we investigated whether repeated references are still reduced in a situation when reduction would not be beneficial for the communicative situation, namely after the speaker receives negative feedback from the addressee. In a director–matcher task (experiment I), we studied gesture rate, as well as the general form of the gestures produced in initial and repeated references. In a separate experiment (experiment II) we studied whether there might (also) be more gradual differences in gesture form between gestures in initial and repeated references, by asking human judges which of two gestures (one from an initial and one from a repeated reference following negative feedback) they considered more precise. In both experiments, mutual visibility was added as a between subjects factor. Results showed that after negative feedback, gesture rate increased in a marginally significant way. With regard to gesture form, we found little evidence for changes in gesture form after negative feedback, except for a marginally significant increase of the number of repeated strokes within a gesture. Lack of mutual visibility only had a significant reducing effect on gesture size, and did not interact with repetition in any way. However, we did find gradual differences in gesture form, with gestures produced after negative feedback being judged as marginally more precise than initial gestures. The results from the present study suggest that in the production of unsuccessful repeated references, a process different from the reduction process as found in previous studies in repeated references takes place, with speakers appearing to put more effort into their gestures after negative feedback, as suggested by the data trending towards an increased gesture rate and towards gestures being judged as more precise after feedback

Cross-modal reduction: Repetition of words and gestures

This dissertation examines speakers’ production of speech and representational gesture. It utilizes the Repetition Effect as the investigative tool. The Repetition Effect appears to vary by the tendency for some items to shorten when repeating, at least under the condition that speakers can primarily operate by their assumption of the state of knowledge of the listener. In speech, a highly conventionalized form of performance, word duration reduces within the same stretch of coherent discourse; then, it resets in the first mention of a new stretch of coherent discourse regardless of the state of knowledge to the speaker or the listener. Therefore, the Repetition Effect in speech is best analyzed as an automatic behavior triggered by discourse structure, rather than reflecting online changes in word accessibility for either interlocutor, be it for the speaker (Listener-neutral explanation) or for the listener (Listener-modeling explanation). The Repetition Effect in speech production in this dissertation will be accounted for within an exemplar model of the perception/production loop. However, in representational gestures, a much less conventionalized form of performance compared to speech, the Repetition Effect shows a different pattern. When speakers only operate by their assumption of the state of knowledge of the listener, without dynamic, appreciable listener feedback, they steadily reduce most types of representational gesture across tellings. Based on these results, it can be argued that representational gestures primarily serve as a part of speech production, rather than as communicative acts. That is, they are produced without regard to the novelty of the information to the listener, thus, consistent with the Listener-neutral explanation