In this keynote, musical gestures will be discussed in relation to the basic concepts of the embodied music cognition paradigm. Video examples are given of stud- ies and applications that are based on these concepts.

Leman, Marc

English

Ghent University Academic Bibliography

   MUSICAL GESTURES AND EMBODIED COGNITION Marc Leman IPEM, Dept. of Musicology Ghent University, Belgium Marc.Leman@UGent.be ABSTRACT In this keynote, musical gestures will be discussed in relation to the basic concepts of the embodied music cognition paradigm. Video examples are given of stud-ies and applications that are based on these concepts.1 1. INTRODUCTION Interactive music systems have both a technological and experiential component. The technological compo-nent is concerned with the tools that translate human movements into manipulations of sounds. The experien-tial component is concerned with the intended control of the tools, which is accomplished through gestures. In this keynote, we mainly focus on the later component, as we believe that a good understanding of it is a pre-requisite for a successful tool development. Recently, the foundations of musical gestures have been reconsidered, and new viewpoints have been in-spired by the insights from research on embodied music cognition (Godøy and Leman, 2011). The goal of the keynote is, firstly, to clarify the concept of musical ges-tures in relation to the basic concepts of embodied mu-sic cognition, secondly, to consider a number of impli-cations for technology development (Leman, 2007).  To illustrate that point, a number of video excerpts will be shown and discussed. I should add here that in the paradigm of embodied music cognition, it is assumed that cognition is more than just mental processing and its corresponding brain activations. Indeed, music cognition is considered to be situated (i.e. embedded in an environment), and enacted (i.e. put into practice through action) (see Varela et al., 1991; Barsalou, 2008). The embodiment hypothesis refers to an action-oriented basis for making sense of our environment. In that context, we consider the former focus on music and mind within in a broader perspective that puts emphasis on embodiment, action, and conse-quently, gestures.                                                    1 This text is a modified version of a chapter that will appear in the book M. Lesaffre and M. Leman (eds.), “The power of music – researching musical experi-ences”, Leuven: ACCO, 2012. 2. BASIC CONCEPTS OF MUSICAL EMBODIMENT  The paradigm of embodied music cognition is based on a number of concepts, related to: (i) the body as me-diator, (ii) the gesture/action repertoire, (iii) the action-perception coupling, and (iv) the link with subjective experiences, such as intentions, expressions, empathy, and emotions. Let me introduce these concepts and dis-cuss how musical gestures are related to them. 2.1. Body as mediator The body of a person can be considered as the media-tor between the person’s environment and the person’s subjective experience of that environment. This view-point is widely known (e.g. Merleau-Ponty, 1945), but it always needs a careful explanation. Indeed, the role of the body is particular in that it causally connects with a physical environment as well as with an experience (of the subject who owns that body). The physical environ-ment can be described in an objective way (e.g. the waveform of the music on my sound recorder). The ex-perience can only be described in a subjective way (my subjective feeling in response to the music on my sound recorder). Similarly, musical gestures can be described in an objective way as movement of body parts, but they have an important experiential component that is related to intentions, goals, and expressions. 2.2. The action/gesture repertoire When the body acts as a mediator between our envi-ronment and the experience, it will build up a repertoire of gestures and gesture/action consequences. Such a repertoire is called a gesture/action-oriented ontology. It is the set of commands and their perceived outcomes that is somehow kept in memory and used for the next action and the interpretation of the next perceptions.   The repertoire is based on the idea that humans inter-act with the environment on the basis of actions, that is, movements that are made to achieve a particular goal, such as access to resources, or access to other persons. The repertoire can also be considered as the reservoir of experiences. This repertoire thus comprises connections between commands, sensations of the external world Actes des Journées d’Informatique Musicale (JIM 2012), Mons, Belgique, 9-11 mai 20125   through our senses (exterioception), but also sensations of our body movements (proprioception) and of our body state (interioception). Musical gestures form the core component of this repertoire. 2.2.1. Gestures Interestingly, the concept of musical gesture applies both to sounds as well as to body movements (Godøy & Leman, 2010). For example, in a study on guqin per-formance (Henbing and Leman, 2007), it was possible to show that sonic gestures (identified in music) reflect sound-producing gestures (hand movements), and that these gestures can be understood as concatenations of more elementary gestural components. The rules for combining elements of this alphabet are defined by natural constraints (i.e. physical, physiological limita-tions) as well as cultural constraints (preferred move-ments). More recently, Desmet et al. (2012) studied clarinet playing gestures in relation to musical inten-tionality and expressiveness. Leman and Naveda (2010) studied Samba dance from the viewpoint of spatiotem-poral representations.  2.2.2. Entrainment The research on entrainment considers how resonant systems adapt their synchronization with each other. Traditionally, this work has a more exclusively focus on timing aspects in relation to synchronization, such as the tempo and phase during synchronization tasks in music playing or dancing.  2.2.3. Gestures and entrainment In several more recent studies conducted at IPEM, we consider the idea that gestures may condition entrain-ment, and therefore, that the focus of entrainment should be broadened to include a spatiotemporal dimension that is rooted in bodily gestures. Such an approach may lead to a gestural topology of point clouds, a concept that has been introduced by Naveda and Leman (2010) and fur-ther developed in Maes et al. (submitted). Typically, with gross motor gestures that subsume a particular re-petitive choreography, there is a large spatial region where the body part can be at a particular point in time, which ensures tolerance for synchronization variability, and therefore some flexibility in entrainment.   In contrast with these gross motor gestures, fine mo-tor gestures require a more precise spatiotemporal de-ployment. This happens when people interact with a music instrument. In that case, there is less tolerance for synchronization variability, and less flexibility in en-trainment. Just consider the fact that rehearsals of music ensembles often aim at getting the synchronization right.  2.3. The coupling of action and perception From the previous paragraph it is clear that the ges-ture/action repertoire forms part of a more complex mechanism that controls the interaction between envi-ronment and subjective experience. This mechanism, also called, the action-perception coupling system, or the action-perception engine, is responsible for predic-tion and, as we will see, also for issues that involve mu-sical intentions (musical action-goals).  Understanding the components and the dynamics of the action-perception coupling system is a hot research topic in modern cognitive science (see Wolpert et al., 2011).  In general terms, it makes sense to consider at least two mechanisms for learning, which I call the sen-sorimotor loop and the action-perception loop. The sen-sorimotor loop is a low-level loop where the motor ac-tivity is basically driven by sensory input from the envi-ronment. In contrast, the action-perception loop is a high-level loop that involves the gesture/action reper-toire. The two loops are not exclusive and they can run in parallel. For example, in producing a sound on the clarinet, the sensorimotor loop can maintain the control of the mouth and breath in relation to the sound produc-tion, while the action-perception loop can use the reper-toire of learned fingering patterns (that have been prac-ticed earlier) as a global controller for the expressive production of the sound. As such, the assumption is that action and perception are controlled by an action-perception coupling system involving different loops.   Interestingly, the prediction principle also works the other way around. When a person perceives a certain sound, the person may rely on previously learned ac-tion-outcome knowledge in order to assume the action for the sound. The latter principle is indeed fundamental in our approach because it explains why interaction is embodied. This principle, which is comparable to a Bayesian inference in statistics, also offers a key to modeling our multimodal (sound-gesture) interaction with music. Typically, when an every-day sound is heard, the person will tend to act in relation to how that sound is produced. In contrast, when an abstract sound is heard, the person will tend to act in relation to particu-lar parameters of the sound that can be reproduced by movements (such as general contours). 2.4. Musical intentionality, expressiveness, and empa-thy  Our research at IPEM has contributed to the idea that music may establish an intentional layer of communica-tion between listener and player. This intentional layer provides access to the (assumed) goals of the music. It is based on an action-perception coupling system that tends to use actions as causes of perceived patterns, thus turning perceived sound patterns into action patterns that could have caused the perceived sounds. This mechanism forms the basis of a number of social phe-Actes des Journées d’Informatique Musicale (JIM 2012), Mons, Belgique, 9-11 mai 20126   nomena of music, ranging from empathy to social bond-ing.  3. THE ROLE OF TOOLS Finally a word should be said about tools. Tools are of outmost interest to us for two reasons. Firstly, with tools, we can study the characteristics of the action-perception coupling system. For example, during piano playing, the action-perception coupling can be disturbed by using a headphone that introduces an auditory delay. In order to be able to cope with the music playing, the agent may have to change the usual strategy for predict-ing the consequences of his actions. The change of strat-egy may reveal the mechanisms that underlay the cou-pling system. Secondly, with tools, we can build new applications that intervene with the action-perception coupling system. One example is DJogger (Moens et al., 2010) that extracts the tempo of your walking and pro-vides music at the same tempo of the walking tempo. Humans tend to entrain and thus walk in phase with the musical pulse. Typically, they minimize the prediction error of the walking pulse until it fits with the musical pulse. IPEM has developed tools for different domains such as music education, therapy or entertainment. Dur-ing this keynote, several examples will be given and further discussed 4. CONCLUSION The paradigm of embodied music cognition provides an interesting viewpoint on musical gestures. It assumes that music is based on a tight relationship between sounds and experiences that are mediated by the body. Apart from the research that aims at understanding this phenomenon, the goal is to be able to control the nature of this complex phenomenon, and apply this control in applications for music education, health, art and other domains.  The paradigm of embodied music cognition provides an approach of how to deal with the experiential com-ponent in interactive music systems. The research field is likely to become an important partner of technology-driven music research. 5. REFERENCES  Barsalou, L. W. (2008). Grounded cognition. Annu. Rev. Psychol., 59, 617-645. Desmet, F., Nijs, L., Demey, M., Lesaffre, M., Mar-tens, J.P., Leman, M. (2012 - in press). A statistical analysis framework for assessing a clarinet player’s per-former gestures in relation to locally intended musical targets. Journal of New Music Research. Godøy, R. I., & Leman, M. (Eds.) (2011). Musical gestures : sound, movement, and meaning. New York (N.Y.): Routledge. Henbing, Li., & Leman, M. (2007). A gesture-based typology of sliding-tones in guqin music. Journal of New Music Research, 36(2), 61-82. Leman, M. (2007). Embodied Music Cognition and Mediation Technology. Cambridge, MA: MIT Press. Leman, M., and Naveda, L. A. (2010). Basic gestures as spatiotemporal reference frames for repetitive dance/music patterns in Samba and Charleston. Music Perception 28, 71‐91. Maes, P.-J., Amelynck, D., Lesaffre, M., Arvind, D., Leman, M. (submitted). Conducting Master: An interac-tive, real-time gesture monitoring system based on spa-tiotemporal motion templates. Merleau-Ponty, M. (1945). Phénomé ́nologie de la perception. Paris: Gallimard. Moens, B., van Noorden, L., Leman, M. (2010). D-Jogger: Syncing Music with Walking. In Proceedings of the SMC conference 2010, pp 451-456. Naveda, L. A., and Leman, M. (2010). The spatio-temporal representation of dance and music gestures using topological gesture analysis (TGA). Music Per-ception 28, 93-111. Varela, F. J., Thompson, E., & Rosch, E. (1991). The Embodied Mind: Cognitive Science and Human Experi-ence. Cambridge, MA: MIT Press. Wolpert, D. M., Diedrichsen, J., & Flanagan, J. R. (2011). Principles of sensorimotor learning. Nature Re-views Neuroscience, 12(12), 739-751.  BIOGRAPHY OF THE AUTHOR Marc Leman is “Methusalem” research professor in systematic musicology and director of IPEM, Dept. of musicology, at Ghent University. He has a background in musicology, philosophy and computing. He has about 300 publications, including the books “Music and schema theory” (Springer, 1995), and “Embodied music cognition and mediation technology” (MIT Press, 2007). He did pioneering work in the epistemological and methodological foundations of computational modeling and embodied music cognition. In 2007 he became lau-reate of the Methusalem for his project on musical em-bodiment. During the free time (if any) he plays trumpet in jazzcombos. Actes des Journées d’Informatique Musicale (JIM 2012), Mons, Belgique, 9-11 mai 20127

Musical gestures and embodied cognition

https://biblio.ugent.be/publication/2999983/file/2999985

Musical gestures and embodied cognition

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography