Visible movements of the orofacial area: evidence for gestural or multimodal theories of language evolution?

The age-old debate between the proponents of the gesture-first and speech-first positions has returned to occupy a central place in current language evolution theorizing. The gestural scenarios, suffering from the problem known as “modality transition” (why a gestural system would have changed into a predominantly spoken system), frequently appeal to the gestures of the orofacial area as a platform for this putative transition. Here, we review currently available evidence on the significance of the orofacial area in language evolution. While our review offers some support for orofacial movements as an evolutionary “bridge” between manual gesture and speech, we see the evidence as far more consistent with a multimodal approach. We also suggest that, more generally, the “gestural versus spoken” formulation is limiting and would be better expressed in terms of the relative input and interplay of the visual and vocal-auditory sensory modalities

A Dio: A sociosemiotic/phenomenological account of the formationof collective narrative identity in the context of a rock legend’s memorial

God is dead, but, contrary to Nietzsche’s diagnosis, ‘we’ didn’t kill him; he died of cancer. This perhaps crudely cold and off-putting opening does not refer to a naively metaphorically constituted transcendental abstraction, but to a spatio-temporally situated rock legend, Ronnie James Dio. This study aims at contributing to the burgeoning research field of memory and collective identity by providing a sociosemiotic account of the formation of collective narrative identity. By drawing on the three major categories whereby collective memory is formed, that is artifacts, processes, places, as well as on the three key sociosemiotic metafunctions which are responsible for shaping a cultural event as sign system, the pursued interpretive route seeks to effectively contextualize how collective memory is fleshed out situationally in the context of Dio’s memorial. At the same time, by expanding the interpretive canvass to incorporate phenomenological perspectives on the mode of formation of collective memory, the offered analytic is intent on tracing invisible structures that point to operative mechanisms beyond the formal constraints of a sociosemiotic reading. Both phenomenological and sociosemiotic approaches are reinscribed within an overarching narrativity paradigm, wherein their relative merits in addressing the scrutinized phenomenon are discussed in an attempt to formulate a hybrid sociosemiotic phenomenological perspective of memorial events

Gesture in Karnatak Music: Pedagogy and Musical Structure in South India

This thesis presents an examination of gesture in Karnatak music, the art music of South India. The topic is approached from two perspectives; the first considers Karnatak music structure from a gestural perspective, looking both at the music itself and at the gestures that create it, while the second enquires into the role played by physical gesture in vocal pedagogy. The broader aims of the thesis are to provide insight into the musical structure of the Karnatak style, and to contribute to wider discourses on connections between music and movement. An interdisciplinary approach to the research is taken, drawing on theories and methods from the fields of ethnomusicology, embodied music cognition, and gesture studies. The first part of the thesis opens with a discussion of differences between practical and theoretical conceptions of the Karnatak style. I argue for the significance in practice of svara-gamaka units and longer motifs formed of chains of such units, and also consider the gestural qualities of certain motifs and their contribution to bhāva (mood). Subsequently, I present a joint musical and motoric analysis of a section of Karnatak violin performance, seeking to elucidate the dynamic processes that form the style. The second part of the thesis enquires into the role played by hand gestures produced by teachers and students in vocal lessons, looking at what is indexed by the gestures and how such indexing contributes to the pedagogic process. This part of the thesis also considers how gestures contribute to the formation and maintenance of common ground between teacher and student. The final chapter brings the two strands of this thesis together to discuss the connections that exist between musical and physical gesture in Karnatak music

Origins of vocal-entangled gesture

Gestures during speaking are typically understood in a representational framework: they represent absent or distal states of affairs by means of pointing, resemblance, or symbolic replacement. However, humans also gesture along with the rhythm of speaking, which is amenable to a non-representational perspective. Such a perspective centers on the phenomenon of vocal-entangled gestures and builds on evidence showing that when an upper limb with a certain mass decelerates/accelerates sufficiently, it yields impulses on the body that cascade in various ways into the respiratory–vocal system. It entails a physical entanglement between body motions, respiration, and vocal activities. It is shown that vocal-entangled gestures are realized in infant vocal–motor babbling before any representational use of gesture develops. Similarly, an overview is given of vocal-entangled processes in non-human animals. They can frequently be found in rats, bats, birds, and a range of other species that developed even earlier in the phylogenetic tree. Thus, the origins of human gesture lie in biomechanics, emerging early in ontogeny and running deep in phylogeny

A review of temporal aspects of hand gesture analysis applied to discourse analysis and natural conversation

Lately, there has been a\ud n increasing\ud interest in hand gesture analysis systems. Recent works have employed\ud pat\ud tern recognition techniques and have focused on the development of systems with more natural user\ud interfaces. These systems may use gestures to control interfaces or recognize sign language gestures\ud , which\ud can provide systems with multimodal interaction; o\ud r consist in multimodal tools to help psycholinguists to\ud understand new aspects of discourse analysis and to automate laborious tasks.\ud Gestures are characterized\ud by several aspects, mainly by movements\ud and sequence of postures\ud . Since data referring to move\ud ments\ud or\ud sequences\ud carry temporal information\ud , t\ud his paper presents a\ud literature\ud review\ud about\ud temporal aspects of\ud hand gesture analysis, focusing on applications related to natural conversation and psycholinguistic\ud analysis, using Systematic Literature Revi\ud ew methodology. In our results, we organized works according to\ud type of analysis, methods, highlighting the use of Machine Learning techniques, and applications.FAPESP 2011/04608-

Comprehension in-situ: how multimodal information shapes language processing

The human brain supports communication in dynamic face-to-face environments where spoken words are embedded in linguistic discourse and accompanied by multimodal cues, such as prosody, gestures and mouth movements. However, we only have limited knowledge of how these multimodal cues jointly modulate language comprehension. In a series of behavioural and EEG studies, we investigated the joint impact of these cues when processing naturalistic-style materials. First, we built a mouth informativeness corpus of English words, to quantify mouth informativeness of a large number of words used in the following experiments. Then, across two EEG studies, we found and replicated that native English speakers use multimodal cues and that their interactions dynamically modulate N400 amplitude elicited by words that are less predictable in the discourse context (indexed by surprisal values per word). We then extended the findings to second language comprehenders, finding that multimodal cues modulate L2 comprehension, just like in L1, but to a lesser extent; although L2 comprehenders benefit more from meaningful gestures and mouth movements. Finally, in two behavioural experiments investigating whether multimodal cues jointly modulate the learning of new concepts, we found some evidence that presence of iconic gestures improves memory, and that the effect may be larger if information is presented also with prosodic accentuation. Overall, these findings suggest that real-world comprehension uses all cues present and weights cues differently in a dynamic manner. Therefore, multimodal cues should not be neglected for language studies. Investigating communication in naturalistic contexts containing more than one cue can provide new insight into our understanding of language comprehension in the real world