Visually Indicated Sounds

Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions

Feeling for Sound:Mapping Sonic Data to Haptic Perceptions

This paper presents a system for exploring different dimensions of a soundthrough the use of haptic feedback. The Novint Falcon force feedback interfaceis used to scan through soundfiles as a subject moves their hand horizontallyfrom left to right, and to relay information about volume, frequency content,noisiness, or potentially any analysable parameter back to the subject throughforces acting on their hand. General practicalities of mapping sonic elements to physical forces areconsidered, such as the problem of representing detailed data through vaguephysical sensation, approaches to applying forces to the hand that do notinterfering with the smooth operation of the device, and the relative merits ofdiscreet and continuous mappings. Three approaches to generating the forcevector are discussed: 1) the use of simulated detents to identify areas of anaudio parameter over a certain threshold, 2) applying friction proportional tothe level of the audio parameter along the axis of movement, and 3) creatingforces perpendicular to the subject's hand movements.Presentation of audio information in this manner could be beneficial for`pre-feeling' as a method for selecting material to play during a liveperformance, assisting visually impaired audio engineers, and as a generalaugmentation of standard audio editing environments

DigiDrum:A Haptic-based Virtual Reality Musical Instrument and a Case Study

This paper presents DigiDrum – a novel virtual reality musical instrument (VRMI) which consists of a physical drum augmented by virtual reality (VR) to produce enhanced auditory and haptic feedback. The physical drum membrane is driven by a simulated membrane of which the parameters can be changed on the fly. The design and implementation of the instrument setup are detailed together with the preliminary results of a user study which investigates users’ haptic perception of the material stiffness of the drum membrane. The study tests whether the tension in the membrane simulation and the sound damping (how fast the sound dies out) changes users’ perception of drum membrane stiffness. Preliminary results show that higher values for both tension and damping give the illusion of higher material stiffness in the drum membrane, where the damping appears to be the more important factor. The goal and contribution of this work is twofold: on the one hand it introduces a musical instrument which allows for enhanced musical expression possibilities through VR. On the other hand, it presents an early investigation on how haptics influence users’ interaction in VRMIs by presenting a preliminary study

The impact of haptic guidance on musical motor learning

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2007.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 75-80).Skilled musical performance provides one of the best demonstrations of the upper limits of the human motor system's capabilities. It is therefore not surprising that learning to play an instrument is a long and difficult process. Teachers and education researchers alike have long since recognized that while learning rate is dependent on the quantity of practice, perhaps even more important is the quality of that practice. However, for non-trivial skills such as music performance, just gaining an understanding of what physical movements are required can be challenging since they are often difficult to describe verbally. Music teachers often communicate complex gesture by physically guiding their students' hands through the required motions. However, at best, this gives a rough approximation of the target movement and begs the question of whether technology might be leveraged to provide a more accurate form of physical guidance. The success of such a system could lead to significant advancements in music pedagogy by speeding and easing the learning process and providing a more effective means of home instruction. This thesis proposes a "learning-by-feel" approach to percussion instruction and presents two different systems to test the effect of guidance on motor learning. The first system, called the FielDrum, uses a combination of permanent and electromagnets to guide a player's drumstick tip through the motions involved in the performance of arbitrary rhythmic patterns. The second system, called the Haptic Guidance System, uses a servo motor and optical encoder pairing to provide precise measurement and playback of motions approximating those involved in snaredrum performance. This device was used in a pilot study of the effects of physical guidance on percussion learning.(cont.) Results indicate that physical guidance can significantly benefit recall of both note timing and velocity. When subject performance was compared in terms of note velocity recall, the addition of haptic guidance to audio-based training produced a 17% reduction in final error when compared to audio training alone. When performance was evaluated in terms of timing recall, the combination of audio and haptic guidance led to an 18% reduction in early-stage error.by Graham C. Grindlay.S.M

Multisensory plucked instrumentmodeling in unity3D: FromKeytar to accurate string prototyping

Keytar is a plucked guitar simulationmockup developed with Unity3D that provides auditory, visual, and haptic feedback to the player through a PhantomOmni robotic arm. Starting froma description of the implementation of the virtual instrument, we discuss our ongoing work. The ultimate goal is the creation of a set of software tools available for developing plucked instruments in Unity3D. Using such tools, sonic interaction designers can efficiently simulate plucked string prototypes and realize multisensory interactions with virtual instruments for unprecedented purposes, such as testing innovative plucked string interfaces or training machine learning algorithms with data about the dynamics of the performance, which are immediately accessible from the machine

Enhancing stroke generation and expressivity in robotic drummers - A generative physics model approach

The goal of this master's thesis research is to enhance the stroke generation capabilities and musical expressivity in robotic drummers. The approach adopted is to understand the physics of human fingers-drumstick-drumhead interaction and try to replicate the same behavior in a robotic drumming system with the minimum number of degrees of freedom. The model that is developed is agnostic to the exact specifications of the robotic drummer that will attempt to emulate human like drum strokes, and therefore can be used in any robotic drummer that uses actuators with complete control over the motor position angle. Initial approaches based on exploiting the instability of a PID control system to generate multiple bounces and the limitations of this approach are also discussed in depth. In order to assess the success of the model and the implementation in the robotic platform a subjective evaluation was conducted. The evaluation results showed that, the observed data was statistically equivalent to the subjects resorting to a blind guess in order to distinguish between a human playing a multiple bounce stroke and a robot playing a similar kind of stroke.M.S

Three-dimensional user interfaces for scientific visualization

The focus of this grant was to experiment with novel user interfaces for scientific visualization applications using both desktop and virtual reality (VR) systems, and thus to advance the state of the art of user interface technology for this domain. This technology has been transferred to NASA via periodic status reports and papers relating to this grant that have been published in conference proceedings. This final report summarizes the research completed over the past three years, and subsumes all prior reports

Audio-haptic relationships as compositional and performance strategies

As a performer of firstly acoustic and latterly electronic and electro-instrumental music, I constantly seek to improve my mode of interaction with the digital realm: that is, to achieve a high level of sensitivity and expression. This thesis illustrates reasons why making use of haptic interfaces—which offer physical feedback and resistance to the performer—may be viewed as an important approach in addressing the shortcomings of some the standard systems used to mediate the performer’s engagement with various sorts of digital musical information. By examining the links between sound and touch, and the performer-instrument relationship, various new compositional and performance strategies start to emerge. I explore these through a portfolio of original musical works, which span the continuum of composition and improvisation, largely based around performance paradigms for piano and live electronics. I implement new haptic technologies, using vibrotactile feedback and resistant interfaces, as well as exploring more metaphorical connections between sound and touch. I demonstrate the impact that the research brings to the creative musical outcomes, along with the implications that these techniques have on the wider field of live electronic musical performance