Towards an interactive framework for robot dancing applications

Estágio realizado no INESC-Porto e orientado pelo Prof. Doutor Fabien GouyonTese de mestrado integrado. Engenharia Electrotécnica e de Computadores - Major Telecomunicações. Faculdade de Engenharia. Universidade do Porto. 200

Timbral Learning for Musical Robots

abstract: The tradition of building musical robots and automata is thousands of years old. Despite this rich history, even today musical robots do not play with as much nuance and subtlety as human musicians. In particular, most instruments allow the player to manipulate timbre while playing; if a violinist is told to sustain an E, they will select which string to play it on, how much bow pressure and velocity to use, whether to use the entire bow or only the portion near the tip or the frog, how close to the bridge or fingerboard to contact the string, whether or not to use a mute, and so forth. Each one of these choices affects the resulting timbre, and navigating this timbre space is part of the art of playing the instrument. Nonetheless, this type of timbral nuance has been largely ignored in the design of musical robots. Therefore, this dissertation introduces a suite of techniques that deal with timbral nuance in musical robots. Chapter 1 provides the motivating ideas and introduces Kiki, a robot designed by the author to explore timbral nuance. Chapter 2 provides a long history of musical robots, establishing the under-researched nature of timbral nuance. Chapter 3 is a comprehensive treatment of dynamic timbre production in percussion robots and, using Kiki as a case-study, provides a variety of techniques for designing striking mechanisms that produce a range of timbres similar to those produced by human players. Chapter 4 introduces a machine-learning algorithm for recognizing timbres, so that a robot can transcribe timbres played by a human during live performance. Chapter 5 introduces a technique that allows a robot to learn how to produce isolated instances of particular timbres by listening to a human play an examples of those timbres. The 6th and final chapter introduces a method that allows a robot to learn the musical context of different timbres; this is done in realtime during interactive improvisation between a human and robot, wherein the robot builds a statistical model of which timbres the human plays in which contexts, and uses this to inform its own playing.Dissertation/ThesisDoctoral Dissertation Media Arts and Sciences 201

Enhancing stroke generation and expressivity in robotic drummers - A generative physics model approach

The goal of this master's thesis research is to enhance the stroke generation capabilities and musical expressivity in robotic drummers. The approach adopted is to understand the physics of human fingers-drumstick-drumhead interaction and try to replicate the same behavior in a robotic drumming system with the minimum number of degrees of freedom. The model that is developed is agnostic to the exact specifications of the robotic drummer that will attempt to emulate human like drum strokes, and therefore can be used in any robotic drummer that uses actuators with complete control over the motor position angle. Initial approaches based on exploiting the instability of a PID control system to generate multiple bounces and the limitations of this approach are also discussed in depth. In order to assess the success of the model and the implementation in the robotic platform a subjective evaluation was conducted. The evaluation results showed that, the observed data was statistically equivalent to the subjects resorting to a blind guess in order to distinguish between a human playing a multiple bounce stroke and a robot playing a similar kind of stroke.M.S

Musicians (Don't) Play Algorithms. Or: What makes a musical performance

Our private perception of listening to an individualized playlist during a jog is very different from the interaction we might experience at a live concert. We do realize that music is not necessarily a performing art, such as dancing or theater, while our demands regarding musical performances are conflicting: We expect perfect sound quality and the thrill of the immediate. We want the artist to overwhelm us with her virtuosity and we want her to struggle, just like a human. We want to engage with the musical expression and rely on visual and physical cues. Considering that the ears of today’s listeners are used to technologically mediated music, in this paper I explore the unique qualities of musical live performances and examine if our conception allows for new mechatronic inventions, in particular robotic musicians, to participate in this art form. Some of Godlovitch’s main thoughts expounded in his work on “musical performance” [11] serve as a reference and starting point for this investigation. His concept of ‘personalism’, which deprives computer-/program-based musical performances from expressive potential and creative accomplishment is an issue that I want to challenge by pointing out new approaches arising from a reflective discourse on technology, embodiment and expression. The enquiry conducted illustrates, how in reasoning about machine performers and algorithmic realization of music, we also examine the perceptual, physical and social aspects of human musicianship, reconceptualizing our understanding of a musical live performance

Examining the Differences in Beat Perception and Production Between Musicians and Dancers

The ability to perceive and produce a beat is believed to be universal in humans, but there are factors that may give rise to individual differences. The research presented in this dissertation examined four factors that may influence beat processing and sensorimotor synchronization performance: 1) expertise: in music and dance, 2) training style: percussive and nonpercussive, 3) stimulus modality: auditory and visual, and 4) movement type: effector-specific or whole-body. Chapter 2 examined how percussive and nonpercussive music and dance training influence beat perception and production performance using an auditory beat perception task and a finger tapping beat production task. Chapter 3 also examined how percussive and nonpercussive music and dance training influence beat perception and production performance, but using an audiovisual variant of the beat perception task, and a knee bending beat production task recorded with motion capture to assess whole-body movements. Chapters 4 and 5 examined how music and dance training interact with the auditory and visual modalities to influence audiovisual integration measured using a just-noticeable-difference task, and audiovisual synchronization measured using a bimodal target-distractor synchronization task. In Chapter 4, sensorimotor synchronization was tested with finger tapping, whereas in Chapter 5 sensorimotor synchronization was tested with knee bending. Broadly, the data showed that 1) beat processing and sensorimotor synchronization performance differ among musicians, dancers, and their non-musician/non-dancer counterparts, 2) training style did not significantly influence beat perception and production, as performance did not significantly differ between percussionists and nonpercussionists, 3) musicians were biased toward the auditory modality, whereas dancers were biased toward the visual modality when synchronizing to bimodal sequences, and 4) musicians performed better with finger movements, while dancers performed better with whole-body movements. The research presented in this dissertation demonstrate how music and dance—similar, yet different types of training—may affect beat processing and sensorimotor synchronization abilities

Composing abstractions? General musical behaviours in five new works for humans and computer

This written thesis supports a portfolio of composition work. This portfolio was composed to study ‘music systems’ - things that I intentionally design in order to make music happen in future. As a composer of works for computers and humans, music technology is an inherent requirement of my practice. However, I have noticed a tendency to spend a lot of time working on the technological components of these systems, but little time making music. It is easy to conflate the task of composing music with the process of using technology. I argue that this is a matter of ‘analytical framing’. A purely technological framing of what music is, and how it can be worked on, is clearly reductive. To present experiment with different framings, I identify a set of systems that were used to compose my portfolio and ask what they are made of. I produce technical, temporal, communicative, social and ethical answers to this question; each of these new analytical framings respectively unpacked in a specific chapter of this thesis. This reveals that my compositional practice, as a total structure made manifest through the content of my portfolio, can be framed as a complex music system in itself. Crucially, this system is populated by general musical behaviours that are consistent across multiple different framings of what my practice is. My primary argument is that interesting things happen when these abstract behaviours are identified, sorted and instantiated into new works of music. As a contribution to contemporary music discourse, my portfolio documents a novel set of approaches to working with music and systems. From a theoretical standpoint, my multi-framed definition of ‘music system’ helps sidestep a commonly encountered problem in electronic music studies: an over-fixation on technicity which elides social and cultural context. Moreover, this project helps ingrain a practice-orientated understanding of music systems design, by concluding that music systems exist most significantly (i.e. most practically) as iterative, playful and emergent processes of modulating sound perception through time