Physical Modelling Concepts for a Collection of Multisensory Virtual Musical Instruments

International audienceThis paper discusses how haptic devices and physical modelling can be employed to design and simulate multisensory virtual musical instruments, providing the musician with joint audio, visual and haptic feedback. After briefly reviewing some of the main use-cases of haptics in Computer Music, we present GENESIS-RT, a software and hardware platform dedicated to the design and real-time haptic playing of virtual musical instruments using mass-interaction physical modelling. We discuss our approach and report on advancements in modelling various instrument categories, including physical models of percussion, plucked and bowed instruments. Finally, we comment on the constraints, challenges and new possibilities opened by modelling haptic virtual instruments with our platform, and discuss common points and differences in regards to classical Digital Musical Instruments

INTERACTIVE PHYSICAL DESIGN AND HAPTIC PLAYING OF VIRTUAL MUSICAL INSTRUMENTS

International audienceIn Computer Music, a practical approach of many Digital Musical Instruments is to separate the gestural input stage from the sound synthesis stage. While these instruments offer many creative possibilities, they present a strong rupture with traditional acoustic instruments, as the physical coupling between human and sound is broken. This coupling plays a crucial role for the expressive musical playing of acoustic instruments; we believe restoring it in a digital context is of equal importance for revealing the full expressive potential of digital instruments. This paper first presents haptic and physical modelling technologies for representing the mechano-acoustical instrumental situation in the context of DMIs. From these technologies, a prototype environment has been implemented for both designing virtual musical instruments and interacting with them via a force feedback device, able to preserve the energetic coherency of the musician-sound chain

A MODELLER-SIMULATOR FOR INSTRUMENTAL PLAYING OF VIRTUAL MUSICAL INSTRUMENTS

International audienceThis paper presents a musician-oriented modelling and simulation environment for designing physically modelled virtual instruments and interacting with them via a high performance haptic device. In particular, our system allows restoring the physical coupling between the user and the manipulated virtual instrument, a key factor for expressive playing of traditional acoustical instruments that is absent in the vast majority of computer-based musical systems. We first analyse the various uses of haptic devices in Computer Music, and introduce the various technologies involved in our system. We then present the modeller and simulation environments, and examples of musical virtual instruments created with this new environment

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

The composer as technologist : an investigation into compositional process

This work presents an investigation into compositional process. This is undertaken where a study of musical gesture, certain areas of cognitive musicology, computer vision technologies and object-orientated programming, provide the basis for a composer (author) to assume the role of a technologist and acquire knowledge and skills to that end. In particular, it focuses on the application and development of a video gesture recognition heuristic to the compositional problems posed. The result is the creation of an interactive musical work with score for violin and electronics that supports the research findings. In addition, the investigative approach into developing technology to solve musical problems that explores practical composition and aesthetic challenges is detailed

Physical modelling meets machine learning: performing music with a virtual string ensemble

This dissertation describes a new method of computer performance of bowed string instruments (violin, viola, cello) using physical simulations and intelligent feedback control. Computer synthesis of music performed by bowed string instruments is a challenging problem. Unlike instruments whose notes originate with a single discrete excitation (e.g., piano, guitar, drum), bowed string instruments are controlled with a continuous stream of excitations (i.e. the bow scraping against the string). Most existing synthesis methods utilize recorded audio samples, which perform quite well for single-excitation instruments but not continuous-excitation instruments. This work improves the realism of synthesis of violin, viola, and cello sound by generating audio through modelling the physical behaviour of the instruments. A string's wave equation is decomposed into 40 modes of vibration, which can be acted upon by three forms of external force: A bow scraping against the string, a left-hand finger pressing down, and/or a right-hand finger plucking. The vibration of each string exerts force against the instrument bridge; these forces are summed and convolved with the instrument body impulse response to create the final audio output. In addition, right-hand haptic output is created from the force of the bow against the string. Physical constants from ten real instruments (five violins, two violas, and three cellos) were measured and used in these simulations. The physical modelling was implemented in a high-performance library capable of simulating audio on a desktop computer one hundred times faster than real-time. The program also generates animated video of the instruments being performed. To perform music with the physical models, a virtual musician interprets the musical score and generates actions which are then fed into the physical model. The resulting audio and haptic signals are examined with a support vector machine, which adjusts the bow force in order to establish and maintain a good timbre. This intelligent feedback control is trained with human input, but after the initial training is completed the virtual musician performs autonomously. A PID controller is used to adjust the position of the left-hand finger to correct any flaws in the pitch. Some performance parameters (initial bow force, force correction, and lifting factors) require an initial value for each string and musical dynamic; these are calibrated automatically using the previously-trained support vector machines. The timbre judgements are retained after each performance and are used to pre-emptively adjust bowing parameters to avoid or mitigate problematic timbre for future performances of the same music. The system is capable of playing sheet music with approximately the same ability level as a human music student after two years of training. Due to the number of instruments measured and the generality of the machine learning, music can be performed with ensembles of up to ten stringed instruments, each with a distinct timbre. This provides a baseline for future work in computer control and expressive music performance of virtual bowed string instruments