Interpersonal sensorimotor communication shapes intrapersonal coordination in a musical ensemble

Social behaviors rely on the coordination of multiple effectors within one's own body as well as between the interacting bodies. However, little is known about how coupling at the interpersonal level impacts coordination among body parts at the intrapersonal level, especially in ecological, complex, situations. Here, we perturbed interpersonal sensorimotor communication in violin players of an orchestra and investigated how this impacted musicians' intrapersonal movements coordination. More precisely, first section violinists were asked to turn their back to the conductor and to face the second section of violinists, who still faced the conductor. Motion capture of head and bow kinematics showed that altering the usual interpersonal coupling scheme increased intrapersonal coordination. Our perturbation also induced smaller yet more complex head movements, which spanned multiple, faster timescales that closely matched the metrical levels of the musical score. Importantly, perturbation differentially increased intrapersonal coordination across these timescales. We interpret this behavioral shift as a sensorimotor strategy that exploits periodical movements to effectively tune sensory processing in time and allows coping with the disruption in the interpersonal coupling scheme. As such, head movements, which are usually deemed to fulfill communicative functions, may possibly be adapted to help regulate own performance in time

Interpersonal sensorimotor communication shapes intrapersonal coordination in a musical ensemble

Social behaviors rely on the coordination of multiple effectors within one’s own body as well as between the interacting bodies. However, little is known about how coupling at the interpersonal level impacts coordination among body parts at the intrapersonal level, especially in ecological, complex, situations. Here, we perturbed interpersonal sensorimotor communication in violin players of an orchestra and investigated how this impacted musicians’ intrapersonal movements coordination. More precisely, first section violinists were asked to turn their back to the conductor and to face the second section of violinists, who still faced the conductor. Motion capture of head and bow kinematics showed that altering the usual interpersonal coupling scheme increased intrapersonal coordination. Our perturbation also induced smaller yet more complex head movements, which spanned multiple, faster timescales that closely matched the metrical levels of the musical score. Importantly, perturbation differentially increased intrapersonal coordination across these timescales. We interpret this behavioral shift as a sensorimotor strategy that exploits periodical movements to effectively tune sensory processing in time and allows coping with the disruption in the interpersonal coupling scheme. As such, head movements, which are usually deemed to fulfill communicative functions, may possibly be adapted to help regulate own performance in time

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Creativity, collaboration and development in Jeremy Thurlow's Ouija for Peter Sheppard Skaerved

This paper documents and analyses a creative collaboration between the Composer Jeremy Thurlow and the violinist Peter Sheppard Skærved in the production of Ouija, a work for solo violin and laptop computer. The paper situates the account of this creative process within recent literature on distributed and collaborative creativity, and focuses on three aspects of the project: verbal interaction between the two musicians, analysed in terms of ‘creative-talk’ and ‘face-talk’, and the relationship between immediate and more contextual concerns (‘inside/outside the room’); a quantitative analysis of changes in the musical material, focusing on timing; and a qualitative analysis of the role of the violinist’s embodied and instrumental engagement with the music. The paper discusses the findings in relation to forward-oriented (process) and backward-oriented (product) conceptions of creativity, the operation of different social components in creative collaboration, and the relationship between craft, history and embodiment

Gesture in Karnatak Music: Pedagogy and Musical Structure in South India

This thesis presents an examination of gesture in Karnatak music, the art music of South India. The topic is approached from two perspectives; the first considers Karnatak music structure from a gestural perspective, looking both at the music itself and at the gestures that create it, while the second enquires into the role played by physical gesture in vocal pedagogy. The broader aims of the thesis are to provide insight into the musical structure of the Karnatak style, and to contribute to wider discourses on connections between music and movement. An interdisciplinary approach to the research is taken, drawing on theories and methods from the fields of ethnomusicology, embodied music cognition, and gesture studies. The first part of the thesis opens with a discussion of differences between practical and theoretical conceptions of the Karnatak style. I argue for the significance in practice of svara-gamaka units and longer motifs formed of chains of such units, and also consider the gestural qualities of certain motifs and their contribution to bhāva (mood). Subsequently, I present a joint musical and motoric analysis of a section of Karnatak violin performance, seeking to elucidate the dynamic processes that form the style. The second part of the thesis enquires into the role played by hand gestures produced by teachers and students in vocal lessons, looking at what is indexed by the gestures and how such indexing contributes to the pedagogic process. This part of the thesis also considers how gestures contribute to the formation and maintenance of common ground between teacher and student. The final chapter brings the two strands of this thesis together to discuss the connections that exist between musical and physical gesture in Karnatak music

Moving with Music: Approaches to the Analysis of Movement-Music Interactions

In this study I investigate the variegated and complex ways in which music and movement can interact in works that involve both media, such as ballets, modern dance works, music videos, and dance films. My dissertation centers around analyses of pieces in diverse styles and genres; each analysis focuses on different aspects of human movement or movement analysis tools. Some of these concepts – Effort, Space, Body, and Shape – are sourced from Laban Movement Analysis, while others – synchronization, body language, kinesthetic empathy, and form – do not belong to a cohesive system. Taking an intersubjective approach, my analyses highlight instances in which watching co-occurring movement affects my musical perceptions, or vice versa. I also examine conscious interventions on perception, where deliberate changes in perspective, theoretical frameworks, or prioritization of my embodied responses affect the way I hear and see the works. I aim not only to account for structural complexities in movement-music interactions, but also to examine ways in which those interactions participate in articulating identities and politics or in suggesting narrative interpretations. I aim to provide a versatile toolkit that would facilitate the analysis of many different kinds of music-movement interactions. Each chapter outlines two analytical tools and then demonstrates how the tools can be used in an analytical example. In the first chapter, I investigate the role of body language and movement-music synchronization in a hip hop music video by the rapper Tyler, The Creator. I argue that Tyler’s movements fail to synchronize to the music in straightforward ways and fail to convey the cool confidence that his lyrics purport to. As a result, the movement-music relationship helps to articulate a version of masculinity that can be read as non-normative and politically charged. In the second chapter, I examine the role of kinesthetic empathy in the perception of choreographic and musical form in the “Rose Adagio” from Tchaikovsky’s and Petipa’s The Sleeping Beauty. While both character and performer inhabit a single onstage body, the observer’s empathetic embodied responses to the dancer may diverge depending on whether she is read as character or performer. This perceptual contrast depends in part on the ballet’s narrative world. The two possible empathetic alignments yield, in turn, divergent analytical observations about the relationship between music and movement. In the third chapter, I offer an analysis of “Duet” from Lar Lubovitch’s Concerto 622,which is set to the Adagio movement of Mozart’s Clarinet Concerto in A Major K.622. Examining Lubovitch’s choreography helps me to arrive at a more sensitive hearing of the music than I initially expected. Also, in comparing two phrases whose music is nearly identical but which feature different choreography, I find an especially compelling case in support of the proposition that dance affects musical perceptions. In the final chapter, I consider the role of Body and Shape in Nijinsky’s and Debussy’s Jeux. Movement-music analysis provides support for an interpretation of the ballet that acknowledges a pervasive, yet ultimately unfulfilled sexual desire. Movement-music analysis also sheds light on the ever-changing and moment-focused nature of Debussy’s musical form. Motives are not developed nor organized by a large-scale formal design, but instead give rise to ever new musical ideas, unprepared and unresolved. The ballet’s choreography often helps these rapid and abrupt transitions to cohere

Methods and Technologies for the Analysis and Interactive Use of Body Movements in Instrumental Music Performance

List of related publications: http://www.federicovisi.com/publications/A constantly growing corpus of interdisciplinary studies support the idea that music is a complex multimodal medium that is experienced not only by means of sounds but also through body movement. From this perspective, musical instruments can be seen as technological objects coupled with a repertoire of performance gestures. This repertoire is part of an ecological knowledge shared by musicians and listeners alike. It is part of the engine that guides musical experience and has a considerable expressive potential. This thesis explores technical and conceptual issues related to the analysis and creative use of music-related body movements in instrumental music performance. The complexity of this subject required an interdisciplinary approach, which includes the review of multiple theoretical accounts, quantitative and qualitative analysis of data collected in motion capture laboratories, the development and implementation of technologies for the interpretation and interactive use of motion data, and the creation of short musical pieces that actively employ the movement of the performers as an expressive musical feature. The theoretical framework is informed by embodied and enactive accounts of music cognition as well as by systematic studies of music-related movement and expressive music performance. The assumption that the movements of a musician are part of a shared knowledge is empirically explored through an experiment aimed at analysing the motion capture data of a violinist performing a selection of short musical excerpts. A group of subjects with no prior experience playing the violin is then asked to mime a performance following the audio excerpts recorded by the violinist. Motion data is recorded, analysed, and compared with the expert’s data. This is done both quantitatively through data analysis xii as well as qualitatively by relating the motion data to other high-level features and structures of the musical excerpts. Solutions to issues regarding capturing and storing movement data and its use in real-time scenarios are proposed. For the interactive use of motion-sensing technologies in music performance, various wearable sensors have been employed, along with different approaches for mapping control data to sound synthesis and signal processing parameters. In particular, novel approaches for the extraction of meaningful features from raw sensor data and the use of machine learning techniques for mapping movement to live electronics are described. To complete the framework, an essential element of this research project is the com- position and performance of études that explore the creative use of body movement in instrumental music from a Practice-as-Research perspective. This works as a test bed for the proposed concepts and techniques. Mapping concepts and technologies are challenged in a scenario constrained by the use of musical instruments, and different mapping ap- proaches are implemented and compared. In addition, techniques for notating movement in the score, and the impact of interactive motion sensor systems in instrumental music practice from the performer’s perspective are discussed. Finally, the chapter concluding the part of the thesis dedicated to practical implementations describes a novel method for mapping movement data to sound synthesis. This technique is based on the analysis of multimodal motion data collected from multiple subjects and its design draws from the theoretical, analytical, and practical works described throughout the dissertation. Overall, the parts and the diverse approaches that constitute this thesis work in synergy, contributing to the ongoing discourses on the study of musical gestures and the design of interactive music systems from multiple angles

Toward a Postmodern Avant-Garde: Labour, Virtuosity, and Aesthetics in an American New Music Ensemble

This dissertation examines the aesthetic beliefs and labour practices of the American new music ensemble eighth blackbird (lower-case intentional). Drawing on ethnographic research conducted with the ensemble for the past six years, I show how the ensemble responds to specific cultural pressures endemic to the classical music scene, its new music vanguard, and to the contemporary United States. eighth blackbird, I argue, has created an ensemble identity and performance style designed to satisfy numerous audience positions, from experts well-versed in the intricacies of musical techniques to lay-persons unacquainted with the values and practices of new or classical music. This attempt to satisfy such a range of perspectives has lead to an oscillation between modernist values traditionally associated with art music culture and postmodern values typically found within more profit-driven musical practices. Modernism—partially defined here as a belief in a linear and stable history, structural listening, musical virtuosity, and restrained performer movement—is foundational to eighth blackbird’s work. At the same time, ensemble members embrace postmodern values including a celebratory mix of musical styles, attempts to provide accessible concerts, and an adoption of post-Fordist branding wherein personality and the display of labour figures as part of a marketing strategy. By examining the specific rationalizations of and objections to eighth blackbird’s practices, I theorize the existence of a “postmodern avant-garde,” a subculture of musicians and arts workers who attempt to remake avant-garde music into a more accessible and profitable enterprise