The Nottingham Multi-Modal Corpus: a demonstration

This software demonstration overviews the developments made during the 3-year NCeSS funded Understanding New Forms of the Digital Record for e-Social Science project (DReSS) that was based at the University of Nottingham. The demo highlights the outcomes of a specific ‘driver project’ hosted by DReSS, which sought to combine the knowledge of linguists and the expertise of computer scientists in the construction of the multi-modal (MM hereafter) corpus software: the Digital Replay System (DRS). DRS presents ‘data’ in three different modes, as spoken (audio), video and textual records of real-life interactions, accurately aligning within a functional, searchable corpus setting (known as the Nottingham Multi-Modal Corpus: NMMC herein). The DRS environment therefore allows for the exploration of the lexical, prosodic and gestural features of conversation and how they interact in everyday speech. Further to this, the demonstration introduces a computer vision based gesture recognition system which has been constructed to allow for the detection and preliminary codification of gesture sequences. This gesture tracking system can be imported into DRS to enable an automated approach to the analysis of MM datasets

A Formal and Functional Analysis of Gaze, Gestures, and Other Body Movements in a Contemporary Dance Improvisation Performance

UID/FIL/00183/2019 PTDC/FER‐FIL/28278/2017This study presents a microanalysis of what information performers “give” and “give off” to each other via their bodies during a contemporary dance improvisation. We compare what expert performers and non-performers (sufficiently trained to successfully perform) do with their bodies during a silent, multiparty improvisation exercise, in order to identify any differences and to provide insight into nonverbal communication in a less conventional setting. The coordinated collaboration of the participants (two groups of six) was examined in a frame-by-frame analysis focusing on all body movements, including gaze shifts as well as the formal and functional movement units produced in the head–face, upper-, and lower-body regions. The Methods section describes in detail the annotation process and inter-rater agreement. The results of this study indicate that expert performers during the improvisation are in “performance mode” and have embodied other social cognitive strategies and skills (e.g., endogenous orienting, gaze avoidance, greater motor control) that the non-performers do not have available. Expert performers avoid using intentional communication, relying on information to be inferentially communicated in order to coordinate collaboratively, with silence and stillness being construed as meaningful in that social practice and context. The information that expert performers produce is quantitatively less (i.e., producing fewer body movements) and qualitatively more inferential than intentional compared to a control group of non-performers, which affects the quality of the performance.publishersversionpublishe

Prosody and Kinesics Based Co-analysis Towards Continuous Gesture Recognition

The aim of this study is to develop a multimodal co-analysis framework for continuous gesture recognition by exploiting prosodic and kinesics manifestation of natural communication. Using this framework, a co-analysis pattern between correlating components is obtained. The co-analysis pattern is clustered using K-means clustering to determine how well the pattern distinguishes the gestures. Features of the proposed approach that differentiate it from the other models are its less susceptibility to idiosyncrasies, its scalability, and simplicity. The experiment was performed on Multimodal Annotated Gesture Corpus (MAGEC) that we created for research on understanding non-verbal communication community, particularly the gestures

Distant pointing in desktop collaborative virtual environments

Deictic pointing—pointing at things during conversations—is natural and ubiquitous in human communication. Deictic pointing is important in the real world; it is also important in collaborative virtual environments (CVEs) because CVEs are 3D virtual environments that resemble the real world. CVEs connect people from different locations, allowing them to communicate and collaborate remotely. However, the interaction and communication capabilities of CVEs are not as good as those in the real world. In CVEs, people interact with each other using avatars (the visual representations of users). One problem of avatars is that they are not expressive enough when compare to what we can do in the real world. In particular, deictic pointing has many limitations and is not well supported. This dissertation focuses on improving the expressiveness of distant pointing—where referents are out of reach—in desktop CVEs. This is done by developing a framework that guides the design and development of pointing techniques; by identifying important aspects of distant pointing through observation of how people point at distant referents in the real world; by designing, implementing, and evaluating distant-pointing techniques; and by providing a set of guidelines for the design of distant pointing in desktop CVEs. The evaluations of distant-pointing techniques examine whether pointing without extra visual effects (natural pointing) has sufficient accuracy; whether people can control free arm movement (free pointing) along with other avatar actions; and whether free and natural pointing are useful and valuable in desktop CVEs. Overall, this research provides better support for deictic pointing in CVEs by improving the expressiveness of distant pointing. With better pointing support, gestural communication can be more effective and can ultimately enhance the primary function of CVEs—supporting distributed collaboration

Production and comprehension of audience design behaviours in co-speech gesture

Speakers can use gesture to depict information during conversation (Kendon, 2004). The current thesis investigates how speakers can adjust their gestures to communicate more effectively to an addressee using gesture. Furthermore, the current thesis investigates the mechanisms behind audience design behaviours. Chapter 1 introduces the topics of gestures and audience design, and outlines the structure of the thesis. Chapter 2 explores the definition and classification of gestures, and provides a review of the literature on gesture production, gesture comprehension, and audience design. Chapter 3 investigates the mechanisms responsible for producing audience design behaviours, and the competing factors affecting gesture production. The findings suggest that speakers use cue-based heuristics to design communicative behaviours. Furthermore, the findings suggest that speakers value gesture more for communication when describing spatial stimuli than abstract stimuli. Chapter 4 further investigates the mechanisms responsible for producing audience design behaviours and the factors affecting gesture production. The findings suggest that speakers can both respond to cues from the addressee using heuristics and take the perspective of the addressee. Furthermore, we found no evidence to suggest that the effect of visibility was due to the confounding of visibility and addressee responsiveness. Chapter 5 investigates how foregrounding gestures can help the gestures convey information to the addressee. The findings do not provide unequivocal evidence that foregrounding benefits the addressee’s comprehension. However, trends in the data suggest that making gestures visually prominent or referring to the gesture in speech may help the gesture to convey information to the addressee. Chapter 6 discussed and interpreted the findings from the previous Chapters. It discusses the mechanisms responsible for audience design behaviours, the factors that affect gesture production, and the effect of gestural audience design behaviours on addressee comprehension. The chapter discusses my interpretations of the findings regarding the current literature and proposes further research