Buttons, Handles, and Keys: Advances in Continuous-Control Keyboard Instruments

This is the peer reviewed version of the following article: Buttons, Handles, and Keys: Advances in Continuous-Control Keyboard Instruments, which has been published in final form at http://dx.doi.org/10.1162/COMJ_a_00297. This article may be used for non-commercial purposes in accordance with MIT Press Journal's Terms and Conditions for Self-Archiving. © 2015, MIT Press Journal

Causative role of left aIPS in coding shared goals during human-avatar complementary joint actions

Successful motor interactions require agents to anticipate what a partner is doing in order to predictively adjust their own movements. Although the neural underpinnings of the ability to predict others' action goals have been well explored during passive action observation, no study has yet clarified any critical neural substrate supporting interpersonal coordination during active, non-imitative (complementary) interactions. Here, we combine non-invasive inhibitory brain stimulation (continuous Theta Burst Stimulation) with a novel human-avatar interaction task to investigate a causal role for higher-order motor cortical regions in supporting the ability to predict and adapt to others' actions. We demonstrate that inhibition of left anterior intraparietal sulcus (aIPS), but not ventral premotor cortex, selectively impaired individuals' performance during complementary interactions. Thus, in addition to coding observed and executed action goals, aIPS is crucial in coding 'shared goals', that is, integrating predictions about one's and others' complementary actions

Machine Learning of Musical Gestures: Principles and Review

We present an overview of machine learning (ML) techniques and their application in interactive music and new digital instrument design. We first provide the non-specialist reader an introduction to two ML tasks, classification and regression, that are particularly relevant for gestural interaction. We then present a review of the literature in current NIME research that uses ML in musical gesture analysis and gestural sound control. We describe the ways in which machine learning is useful for creating expressive musical interaction, and in turn why live music performance presents a pertinent and challenging use case for machine learning

Modelling Instrumental Gestures and Techniques: A Case Study of Piano Pedalling

PhD ThesisIn this thesis we propose a bottom-up approach for modelling instrumental gestures and techniques, using piano pedalling as a case study. Pedalling gestures play a vital role in expressive piano performance. They can be categorised into di erent pedalling techniques. We propose several methods for the indirect acquisition of sustain-pedal techniques using audio signal analyses, complemented by the direct measurement of gestures with sensors. A novel measurement system is rst developed to synchronously collect pedalling gestures and piano sound. Recognition of pedalling techniques starts by using the gesture data. This yields high accuracy and facilitates the construction of a ground truth dataset for evaluating the audio-based pedalling detection algorithms. Studies in the audio domain rely on the knowledge of piano acoustics and physics. New audio features are designed through the analysis of isolated notes with di erent pedal e ects. The features associated with a measure of sympathetic resonance are used together with a machine learning classi er to detect the presence of legato-pedal onset in the recordings from a speci c piano. To generalise the detection, deep learning methods are proposed and investigated. Deep Neural Networks are trained using a large synthesised dataset obtained through a physical-modelling synthesiser for feature learning. Trained models serve as feature extractors for frame-wise sustain-pedal detection from acoustic piano recordings in a proposed transfer learning framework. Overall, this thesis demonstrates that recognising sustain-pedal techniques is possible to a high degree of accuracy using sensors and also from audio recordings alone. As the rst study that undertakes pedalling technique detection in real-world piano performance, it complements piano transcription methods. Moreover, the underlying relations between pedalling gestures, piano acoustics and audio features are identi ed. The varying e ectiveness of the presented features and models can also be explained by di erences in pedal use between composers and musical eras

The Observation and Execution of Actions Share Motor and Somatosensory Voxels in all Tested Subjects: Single-Subject Analyses of Unsmoothed fMRI Data

Many neuroimaging studies of the mirror neuron system (MNS) examine if certain voxels in the brain are shared between action observation and execution (shared voxels, sVx). Unfortunately, finding sVx in standard group analyses is not a guarantee that sVx exist in individual subjects. Using unsmoothed, single-subject analyses we show sVx can be reliably found in all 16 investigated participants. Beside the ventral premotor (BA6/44) and inferior parietal cortex (area PF) where mirror neurons (MNs) have been found in monkeys, sVx were reliably observed in dorsal premotor, supplementary motor, middle cingulate, somatosensory (BA3, BA2, and OP1), superior parietal, middle temporal cortex and cerebellum. For the premotor, somatosensory and parietal areas, sVx were more numerous in the left hemisphere. The hand representation of the primary motor cortex showed a reduced BOLD during hand action observation, possibly preventing undesired overt imitation. This study provides a more detailed description of the location and reliability of sVx and proposes a model that extends the original idea of the MNS to include forward and inverse internal models and motor and sensory simulation, distinguishing the MNS from a more general concept of sVx

Instrumenting the Interaction: Affective and Psychophysiological Features of Live Collaborative Musical Improvisation

NIME’14, June 30 – July 03, 2014, Goldsmiths, University of London, UK. Copyright remains with the author(s)

Sensorimotor cortex as a critical component of an 'extended' mirror neuron system: Does it solve the development, correspondence, and control problems in mirroring?

A core assumption of how humans understand and infer the intentions and beliefs of others is the existence of a functional self-other distinction. At least two neural systems have been proposed to manage such a critical distinction. One system, part of the classic motor system, is specialized for the preparation and execution of motor actions that are self realized and voluntary, while the other appears primarily involved in capturing and understanding the actions of non-self or others. The latter system, of which the mirror neuron system is part, is the canonical action 'resonance' system in the brain that has evolved to share many of the same circuits involved in motor control. Mirroring or 'shared circuit systems' are assumed to be involved in resonating, imitating, and/or simulating the actions of others. A number of researchers have proposed that shared representations of motor actions may form a foundational cornerstone for higher order social processes, such as motor learning, action understanding, imitation, perspective taking, understanding facial emotions, and empathy. However, mirroring systems that evolve from the classic motor system present at least three problems: a development, a correspondence, and a control problem. Developmentally, the question is how does a mirroring system arise? How do humans acquire the ability to simulate through mapping observed onto executed actions? Are mirror neurons innate and therefore genetically programmed? To what extent is learning necessary? In terms of the correspondence problem, the question is how does the observer agent know what the observed agent's resonance activation pattern is? How does the matching of motor activation patterns occur? Finally, in terms of the control problem, the issue is how to efficiently control a mirroring system when it is turned on automatically through observation? Or, as others have stated the problem more succinctly: "Why don't we imitate all the time?" In this review, we argue from an anatomical, physiological, modeling, and functional perspectives that a critical component of the human mirror neuron system is sensorimotor cortex. Not only are sensorimotor transformations necessary for computing the patterns of muscle activation and kinematics during action observation but they provide potential answers to the development, correspondence and control problems

Interactional leader-follower sensorimotor communication strategies during repetitive joint actions

Non-verbal communication is the basis of animal interactions. In dyadic leader – follower interactions, leaders master the ability to carve their motor behaviour in order to ‘signal’ their future actions and internal plans while these signals influence the behaviour of follower partners, who automatically tend to imitate the leader even in complementary interactions. Despite their usefulness, signalling and imitation have a biomechanical cost, and it is unclear how this cost – benefits trade-off is managed during repetitive dyadic interactions that present learnable regularities. We studied signalling and imitation dynamics (indexed by movement kinematics) in pairs of leaders and followers during a repetitive, rule-based, joint action. Trial-by-trial Bayesian model comparison was used to evaluate the relation between signalling, imitation and pair performance. The different models incorporate different hypotheses concerning the factors (past interactions versus online movements) influencing the leader’s signalling (or follower’s imitation) kinematics. This approach showed that (i) leaders’ signalling strategy improves future couple performance, (ii) leaders used the history of past interactions to shape their signalling, (iii) followers’ imitative behaviour is more strongly affected by the online movement of the leader. This study elucidates the ways online sensorimotor communication help individuals align their task representations and ultimately improves joint action performanc