The coupling of action and perception in musical meaning formation

The embodied perspective on music cognition has stressed the central role of the body and body move- ments in musical meaning formation processes. In the present study, we investigate by means of a behavioral experiment how free body movements in response to music (i.e., action) can be linked to specific linguistic, metaphorical descriptions people use to describe the expressive qualities they perceive in the music (i.e., per- ception). We introduce a dimensional model based on the Effort/Shape theory of Laban in order to target musical expressivity from an embodied perspective. Also, we investigate whether a coupling between action and perception is dependent on the musical background of the participants (i.e., trained versus untrained). The results show that the physical appearance of the free body movements that participants perform in response to music are reliably linked to the linguistic descriptions of musical expressiveness in terms of the underlying quality. Moreover, this result is found to be independent of the participants’ musical background

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance

The influence of body movements on children's perception of music with an ambiguous expressive character

The theory of embodied music cognition states that the perception and cognition of music is firmly, although not exclusively, linked to action patterns associated with that music. In this regard, the focus lies mostly on how music promotes certain action tendencies (i.e., dance, entrainment, etc.). Only recently, studies have started to devote attention to the reciprocal effects that people’s body movements may exert on how people perceive certain aspects of music and sound (e.g., pitch, meter, musical preference, etc.). The present study positions itself in this line of research. The central research question is whether expressive body movements, which are systematically paired with music, can modulate children’s perception of musical expressiveness. We present a behavioral experiment in which different groups of children (7–8 years, N = 46) either repetitively performed a happy or a sad choreography in response to expressively ambiguous music or merely listened to that music. The results of our study show indeed that children’s perception of musical expressiveness is modulated in accordance with the expressive character of the dance choreography performed to the music. This finding supports theories that claim a strong connection between action and perception, although further research is needed to uncover the details of this connection

Representation of Pitch in Horizontal Space and Its Dependence on Musical and Instrumental Experience

Representation of pitch in horizontal space and its relationship to musical and instrumental experience was examined in three behavioral experiments. Each experiment investigated the influence of a task-irrelevant dimension (pitch or location) on the perception of a task-relevant dimension (location or pitch, respectively). Sine tones with nine different pitches were presented from nine locations, and participants estimated the pitch or location of the stimuli. Experiment 1 showed an influence of the (task irrelevant) pitch of presented stimuli on the perceived location of the stimuli in musically experienced participants only. This influence increased with the degree of musical training of participants. No influence was found of presented location on the perception of pitch. Experiments 2 and 3 investigated the influence of instrumental expertise comparing the responses of a group of flutists with a group of pianists. An interaction with instrumental expertise was found only in Experiment 3, where participants played shortly on their respective instruments before doing the perceptual judgments. The experiments indicate that musical training in general influence the pitch-location association, and pianistic experience in particular

Interactive Sound Texture Synthesis through Semi-Automatic User Annotations

We present a way to make environmental recordings controllable again by the use of continuous annotations of the high-level semantic parameter one wishes to control, e.g. wind strength or crowd excitation level. A partial annotation can be propagated to cover the entire recording via cross-modal analysis between gesture and sound by canonical time warping (CTW). The annotations serve as a descriptor for lookup in corpus-based concatenative synthesis in order to invert the sound/annotation relationship. The workflow has been evaluated by a preliminary subject test and results on canonical correlation analysis (CCA) show high consistency between annotations and a small set of audio descriptors being well correlated with them. An experiment of the propagation of annotations shows the superior performance of CTW over CCA with as little as 20 s of annotated material

Machine Learning of Musical Gestures: Principles and Review

We present an overview of machine learning (ML) techniques and their application in interactive music and new digital instrument design. We first provide the non-specialist reader an introduction to two ML tasks, classification and regression, that are particularly relevant for gestural interaction. We then present a review of the literature in current NIME research that uses ML in musical gesture analysis and gestural sound control. We describe the ways in which machine learning is useful for creating expressive musical interaction, and in turn why live music performance presents a pertinent and challenging use case for machine learning

The Role of Sound Source Perception in Gestural Sound Description

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source, and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds where no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterwards, we conducted interviews asking participants to verbalize their experience, watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated to the sound, participants trace contours related to sound acoustic features. We also found that the inter-participants gesture variability is higher for causal sounds compared to non-causal sounds. Variability demonstrates that in the first case, participants have several ways of producing the same action whereas in the second case, the sound features tend to make the gesture responses consistent

A importância do corpo na praxis musical: execução, interpretação e percepção

A importância do corpo na praxis musical: execução, interpretação e percepção. A forma como o corpo está envolvido em toda a prática musical tem vindo a despertar cada vez mais curiosidade em diferentes áreas de investigação. Através destas abordagens científicas os músicos ganham consciência deste facto intimamente ligado a todo o processo que vai desde a preparação de uma obra até à performance musical. Este trabalho oferece uma visão geral de como o corpo e o movimento corporal contribuem para uma maior eficácia na comunicação musical e, consequentemente, melhor qualidade na performance, do ponto de vista do público; ### Abstract: The importance of the body in musical praxis: execution, interpretation and perception. The way how the body is involved in all musical practice has been increasing the curiosity in different fields of research. Through these scientific approaches musicians gain consciousness of this fact that is intimately linked to the entire process that starts with the preparation of a piece and ends with the musical performance. This work offers a general view on how the body and bodily movement contribute to a higher effectiveness in musical communication and, consequently, higher quality in performance, from the audience’s point of view