How do Iconic Gestures Convey Visuo-spatial Information? Bringing together Empirical, Theoretical, and Simulation Studies

Rieser H, Bergmann K, Kopp S. How do Iconic Gestures Convey Visuo-spatial Information? Bringing together Empirical, Theoretical, and Simulation Studies. In: Efthimiou E, Kouroupetroglou G, eds. Gestures in Embodied Communication and Human-Computer Interaction. Berlin/Heidelberg, Germany: Springer; 2012: 139-150.We investigate the question of how co-speech iconic gestures are used to convey visuo-spatial information in an interdisciplinary way, starting with a corpus-based empirical and theoretical perspective on how a typology of gesture form and a partial ontology of gesture meaning are related. Results provide the basis for a computational modeling approach that allows us to simulate the production of speaker-specific gesture forms to be realized with virtual agents. An evaluation of our simulation results and our methodology shows that the model is able to successfully approximate human gestural behavior use of iconic gestures, and moreover, that gestural behavior can improve how humans rate a virtual agent in terms of eloquence, competence, human-likeness, or likeability

Cognition in action C-i-A: Rethinking gesture in neuro-atypical young people: A conceptual framework for embodied, embedded, extended and enacted intentionality

The three aims of my interdisciplinary thesis are: -To develop a conceptual framework for re-thinking the gestures of neuro-atypical young people, that is non-traditional and non-representational -To develop qualitative analytical tools for the annotation and interpretation of gesture that can be applied inclusively to both neuro-atypical and neuro-typical young people -To consider the conceptual framework in terms of its theoretical implications and practical applications Learning to communicate and work with neuro-atypical young people provides the rationale and continued impetus for my work. My approach is influenced by the limited social, physical and communicative experiences of young people with severe speech and motor impairment, due to cerebral palsy (SSMI-CP). CP is described as: a range of non-progressive syndromes of posture and motor impairment. The aetiology is thought to result from damage to the developing central nervous system during gestation or in the neonate. Brain lesions involve the basal ganglia and the cerebellum; both these sites are known to support motor control and integration. However, gaps in theoretical research and empirical data in the study of corporeal expression in young people with SSMI-CP necessitated the development of both an alternative theoretical framework and new tools. Biological Dynamic Systems Theory is proposed as the best candidate structure for the reconsideration of gesture. It encompasses the global, synthetic and embodied nature of gesture. Gesture is redefined and considered part of an emergent dynamic, complex, non-linear and self-organizing system. My construct of Cognition-in-Action (C-i-A) is derived from the notion of knowing-as-doing influenced by socio-biological paradigms; it places the Action-Ready-Body centre stage. It is informed by a theoretical synthesis of knowledge from the domains of Philosophy, Science and Technology, including practices in the clinical, technology design and performance arts arenas. The C-i-A is a descriptive, non-computational feature-based framework. Its development centred around two key questions that served as operational starting points: What can gestures reveal about children’s cognition-in-action? and Is there the potential to influence gestural capacity in children? These are supported by my research objectives. Three case studies are presented that focus on the annotation and interpretative analyses of corporeal exemplars from two adolescent males aged 16.9 and 17.9 years, and one female girl aged 10.7 years. These exemplars were contributed to the Child Gesture Corpus by these young people with SSMI-CP. The Gesture-Action-Entity (GAE) is proposed as a unit of interest for the analysis of procedural, semantic and episodic aspects of our corporeal knowledge. A body-based-action-annotation-system (G-ABAS) and Interpretative Phenomenological Analysis methodology is applied for the first time to gesture (G-IPA). These tools facilitate fine-grained corporeal dynamic and narrative gesture feature analyses. Phenomenal data reveal that these young people have latent resources, capacities and capabilities that they can express corporeally. Iteration of these interpretative findings with the Cognition-in-Action framework allows for the inference of processes that may underlie the strategies they use to achieve such social-motor-cognitive functions. In summary, their Cognition-in-Action is brought-forth, carried forward and has the potential to be culturally embodied. The utility of C-i-A framework lies in its explanatory power to contribute to a deeper understanding of child gesture. Furthermore, I discuss and illustrate its potential to influence practice in the domains of pedagogy, rehabilitation and the design of future intimate, assistive and perceptually sensitive technologies. Such technologies are increasingly mediating our social interactions. My work offers an ecologically valid alternative to tradition conceptualization of perception, cognition and action. My thesis contributes both new knowledge and carries implications across the domains of movement science, gesture studies and applied participatory performance arts and health practices