‘Listening back’ : exploring the sonic interactions at the heart of historical sound effects performance

The cinematic sound design practice of Foley developed from a rich history of performative sound design through materials, objects and mechanical devices created for theatrical performance, magic lantern shows and silent cinema screenings in the late nineteenth and early twentieth century. Now, as virtual and digital methods become available in film sound design production, it is time to evaluate the contribution of these historical sound design methods through the discipline of Sonic Interaction Design (SID). Exploring the use of everyday objects and sound effects devices in the creation of a soundtrack allows us to ‘listen’ back and forward simultaneously. Our knowledge of historical sound practice can be updated and new practices can be generated, at the same time deepening our understanding of sound for screen that is performative in nature. This article investigates the interactivity inherent in historical sound effects performance through a case study of a mechanical and digital wind machine reconstruction, and explores its potential to inform new interactive digital methods for sound design

INTERACTIVE PHYSICAL DESIGN AND HAPTIC PLAYING OF VIRTUAL MUSICAL INSTRUMENTS

International audienceIn Computer Music, a practical approach of many Digital Musical Instruments is to separate the gestural input stage from the sound synthesis stage. While these instruments offer many creative possibilities, they present a strong rupture with traditional acoustic instruments, as the physical coupling between human and sound is broken. This coupling plays a crucial role for the expressive musical playing of acoustic instruments; we believe restoring it in a digital context is of equal importance for revealing the full expressive potential of digital instruments. This paper first presents haptic and physical modelling technologies for representing the mechano-acoustical instrumental situation in the context of DMIs. From these technologies, a prototype environment has been implemented for both designing virtual musical instruments and interacting with them via a force feedback device, able to preserve the energetic coherency of the musician-sound chain

Physical Modelling Concepts for a Collection of Multisensory Virtual Musical Instruments

International audienceThis paper discusses how haptic devices and physical modelling can be employed to design and simulate multisensory virtual musical instruments, providing the musician with joint audio, visual and haptic feedback. After briefly reviewing some of the main use-cases of haptics in Computer Music, we present GENESIS-RT, a software and hardware platform dedicated to the design and real-time haptic playing of virtual musical instruments using mass-interaction physical modelling. We discuss our approach and report on advancements in modelling various instrument categories, including physical models of percussion, plucked and bowed instruments. Finally, we comment on the constraints, challenges and new possibilities opened by modelling haptic virtual instruments with our platform, and discuss common points and differences in regards to classical Digital Musical Instruments

Artistic creation and computer interactive multisensory simulation force feedback gesture transducers

International audienceThe team built up from 1976 with fundamental aims motivated by the computer entering in several fields of artistic creation. The deep mutation represented by the computer, regarding the technology history, required a new and fundamental analysis of the role of the material tools in artistic creation as well as the role of the computer itself as tool. New concepts and theories were necessary that cannot be simply deduced from the previous. The computer was then envisaged to introduce explicitly a new mediation level in the creation process, through the paradigm of the interactive and multisensory simulation (IMS) of physical objects. This principle and its correlated techniques are the core of the computer creation tool envisaged. The higher level functionality being built from this base

A MODELLER-SIMULATOR FOR INSTRUMENTAL PLAYING OF VIRTUAL MUSICAL INSTRUMENTS

International audienceThis paper presents a musician-oriented modelling and simulation environment for designing physically modelled virtual instruments and interacting with them via a high performance haptic device. In particular, our system allows restoring the physical coupling between the user and the manipulated virtual instrument, a key factor for expressive playing of traditional acoustical instruments that is absent in the vast majority of computer-based musical systems. We first analyse the various uses of haptic devices in Computer Music, and introduce the various technologies involved in our system. We then present the modeller and simulation environments, and examples of musical virtual instruments created with this new environment

Design Strategies for Adaptive Social Composition: Collaborative Sound Environments

In order to develop successful collaborative music systems a variety of subtle interactions need to be identified and integrated. Gesture capture, motion tracking, real-time synthesis, environmental parameters and ubiquitous technologies can each be effectively used for developing innovative approaches to instrument design, sound installations, interactive music and generative systems. Current solutions tend to prioritise one or more of these approaches, refining a particular interface technology, software design or compositional approach developed for a specific composition, performer or installation environment. Within this diverse field a group of novel controllers, described as ‘Tangible Interfaces’ have been developed. These are intended for use by novices and in many cases follow a simple model of interaction controlling synthesis parameters through simple user actions. Other approaches offer sophisticated compositional frameworks, but many of these are idiosyncratic and highly personalised. As such they are difficult to engage with and ineffective for groups of novices. The objective of this research is to develop effective design strategies for implementing collaborative sound environments using key terms and vocabulary drawn from the available literature. This is articulated by combining an empathic design process with controlled sound perception and interaction experiments. The identified design strategies have been applied to the development of a new collaborative digital instrument. A range of technical and compositional approaches was considered to define this process, which can be described as Adaptive Social Composition. Dan Livingston

Haptic Control of Multistate Generative Music Systems

Force-feedback controllers have been considered as a solution to the lack of sonically coupled physical feedback in digital-music interfaces, with researchers focusing on instrument-like models of interaction. However, there has been little research applied to the use of force-feedback interfaces to the control of real-time generative-music systems. This paper proposes that haptic interfaces could enable performers to have a more fully embodied engagement with such systems, increasing expressive control and enabling new compositional and performance potentials. A proof-of-concept project is described, which entailed development of a core software toolkit and implementation of a series of test cases

bEADS:Extended Actuated Digital Shaker

While there are a great variety of digital musical interfaces available to the working musician, few o er the level of immediate, nuanced and instinctive control that one nds in an acoustic shaker. bEADS is a prototype of a digital musical instrument that utilises the gestural vocabulary associated with shaken idiophones and expands on the techniques and sonic possibilities associated with them. By using a bespoke physically informed synthesis engine, in conjunction with accelerometer and pressure sensor data, an actuated handheld instrument has been built that allows for quickly switching between widely di ering percussive sound textures. The prototype has been evaluated by three experts with di erent levels of involvement in professional music making

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

A Virtual Reality Platform for Musical Creation

International audienceVirtual reality aims at interacting with a computer in a similar form to interacting with an object of the real world. This research presents a VR platform allowing the user (1) to interactively create physically-based musical instruments and sounding objects, (2) play them in real time by using multisensory interaction by ways of haptics, 3D visualisation during playing, and real time physically-based sound synthesis. So doing, our system presents the two main properties expected in VR systems: the possibility of designing any type of objects and manipulating them in a multisensory real time fashion. By presenting our environment, we discuss the scientific underlying questions: (1) concerning the real time simulation, the way to manage simultaneous audio-haptic-visual cooperation during the real time multisensory simulations and (2) the Computer Aided Design functionalities for the creation of new physically-based musical instruments and sounding objects