From expressive gesture to sound: the development of an embodied mapping trajectory inside a musical interface

This paper contributes to the development of a multimodal, musical tool that extends the natural action range of the human body to communicate expressiveness into the virtual music domain. The core of this musical tool consists of a low cost, highly functional computational model developed upon the Max/MSP platform that (1) captures real-time movement of the human body into a 3D coordinate system on the basis of the orientation output of any type of inertial sensor system that is OSC-compatible, (2) extract low-level movement features that specify the amount of contraction/expansion as a measure of how a subject uses the surrounding space, (3) recognizes these movement features as being expressive gestures, and (4) creates a mapping trajectory between these expressive gestures and the sound synthesis process of adding harmonic related voices on an in origin monophonic voice. The concern for a user-oriented and intuitive mapping strategy was thereby of central importance. This was achieved by conducting an empirical experiment based on theoretical concepts from the embodied music cognition paradigm. Based on empirical evidence, this paper proposes a mapping trajectory that facilitates the interaction between a musician and his instrument, the artistic collaboration between (multimedia) artists and the communication of expressiveness in a social, musical context

Four applications of embodied cognition

This article presents the views of four sets of authors, each taking concepts of embodied cognition into problem spaces where the new paradigm can be applied. The first considers consequences of embodied cognition on the legal system. The second explores how embodied cognition can change how we interpret and interact with art and literature. The third examines how we move through archi- tectural spaces from an embodied cognition perspective. And the fourth addresses how music cogni- tion is influenced by the approach. Each contribution is brief. They are meant to suggest the potential reach of embodied cognition, increase the visibility of applications, and inspire potential avenues for research

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance

The impact of induced emotions on free movement

The goal of this study was to examine the effect of two basic emotions, happiness and sadness, on free movement. A total of 32 adult participants took part in the study. Following an emotion induction procedure intended to induce emotional states of happiness or sadness by means of music and guided imagery, participants moved to an emotionally neutral piece of music that was composed for the experiment. Full body movement was captured using motion caption. In order to explore whether differences in corporeal articulations between the two conditions existed, several movement cues were examined. The criteria for selection of these cues was based on Effort-Shape. Results revealed that in the happy condition, participants showed faster and more accelerated body movement. Moreover, movements proved to be more expanded and more impulsive in the happy condition. These findings provide evidence of the effect of emotion induction as related to body movement

Inclusive improvisation: exploring the line between listening and playing music

The field of Accessible Digital Musical Instruments (ADMIs) is growing rapidly, with instrument designers recognising that adaptations to existing Digital Musical Instruments (DMIs) can foster inclusive music making. ADMIs offer opportunities to engage with a wider range of sounds than acoustic instruments. Furthermore, gestural ADMIs free the music maker from relying on screen, keyboard, and mouse-based interfaces for engaging with these sounds. This brings greater opportunities for exploration, improvisation, empowerment, and flow through music making for people with disability and the communities of practice they are part of. This article argues that developing ADMIs from existing DMIs can speed up the process and allow for more immediate access for those with diverse needs. It presents three case studies of a gestural DMI, originally designed by the first author for his own creative practice, played by people with disability in diverse contexts. The article shows that system-based considerations that enabled an expert percussionist to achieve virtuoso performances with the instrument required minimal hardware and software changes to facilitate greater inclusivity. Understanding the needs of players and customising the system-based movement to sound mappings was of far greater importance in making the instrument accessible

Musical control gestures in mobile handheld devices: Design guidelines informed by daily user experience

Mobile handheld devices, such as smartphones and tablets, have become some of the most prominent ubiquitous terminals within the information and communication technology landscape. Their transformative power within the digital music domain changed the music ecosystem from production to distribution and consumption. Of interest here is the ever-expanding number of mobile music applications. Despite their growing popularity, their design in terms of interaction perception and control is highly arbitrary. It remains poorly addressed in related literature and lacks a clear, systematized approach. In this context, our paper aims to provide the first steps towards defining guidelines for optimal sonic interaction design practices in mobile music applications. Our design approach is informed by user data in appropriating mobile handheld devices. We conducted an experiment to learn links between control gestures and musical parameters, such as pitch, duration, and amplitude. A twofold action—reflection protocol and tool-set for evaluating the aforementioned links—are also proposed. The results collected from the experiment show statistically significant trends in pitch and duration control gesture mappings. On the other hand, amplitude appears to elicit a more diverse mapping approach, showing no definitive trend in this experiment.info:eu-repo/semantics/publishedVersio