PiaF: A Tool for Augmented Piano Performance Using Gesture Variation Following

When performing a piece, a pianist's interpretation is communicated both through the sound produced and through body gestures. We present PiaF (Piano Follower), a prototype for augmenting piano performance by measuring gesture variations. We survey other augmented piano projects, several of which focus on gestural recognition, and present our prototype which uses machine learning techniques for gesture classification and estimation of gesture variations in real-time. Our implementation uses the Kinect depth sensor to track body motion in space, which is used as input data. During an initial learning phase, the system is taught a set of reference gestures, or templates. During performance, the live gesture is classi�ed in real-time, and variations with respect to the recognized template are computed. These values can then be mapped to audio processing parameters, to control digital effects which are applied to the acoustic output of the piano in real-time. We discuss initial tests using PiaF with a pianist, as well as potential applications beyond live performance, including pedagogy and embodiment of recorded performance

Canlı Ses İşleme Ve Geribesleme Kullanılarak Sempatik Titreşen Teller İle Sound Mass Tarzı Sesler Yaratmak

Tez (Doktora) -- İstanbul Teknik Üniversitesi, Sosyal Bilimler Enstitüsü, 2015Thesis (Ph.D.) -- İstanbul Technical University, Institute of Social Sciences, 2015DoktoraPh.D

Travelling Wave Control of Stringed Musical Instruments

PhDDespite the increasing sophistication of digital musical instruments, many performers, composer and listeners remain captivated by traditional acoustic instruments. Interest has grown in the past 2 decades in augmenting acoustic instruments with sensor and actuator technology and integrated digital signal processing, expanding the instrument’s capabilities while retaining its essential acoustic character. In this thesis we present a technique, travelling wave control, which allows active control of the vibrations of musical strings and yet has been little explored in the musical instrument literature to date. The thesis seeks to demonstrate that travelling wave control is capable of active damping and of modifying the timbre of a musical string in ways that go beyond those available through the more conventional modal control paradigm. However, we show that travelling wave control is highly sensitive to nonlinearity, which in practical settings can lead to harmonic distortion and even instability in the string response. To avoid these problems, we design and build a highly linear optical string displacement sensor, and investigate the use of piezoelectric stacks to actuate the termination point of a string. With these components we design and build a functioning travelling wave control system which is capable of damping the vibrations of a plucked string without adversely affecting its timbre. We go on to show that by deliberately adding nonlinearity into the control system, we are able to modify the timbre of the string in a natural way by affecting the evolution of the modal amplitudes. The results demonstrate the feasibility of the concept and lay the groundwork for future integration of travelling wave control into future actuated musical instruments

Expressive Musical Robots: Building, Evaluating, and Interfacing with an Ensemble of Mechatronic Instruments

An increase in the number of parameters of expression on musical robots can result in an increase in their expressivity as musical instruments. This thesis focuses on the design, construction, and implementation of four new robotic instruments, each designed to add more parametric control than is typical for the current state of the art of musical robotics. The principles followed in the building of the four new instruments are scalable and can be applied to musical robotics in general: the techniques exhibited in this thesis for the construction and use of musical robotics can be used by composers, musicians, and installation artists to add expressive depth to their own works with robotic instruments. Accompanying the increase in parametric depth applied to the musical robotics is an increase in difficulty in interfacing with them: robots with a greater number of actuators require more time to program. This document aims to address this problem in two ways: the use of closed-loop control for low-level adjustments of the robots and the use of a parametric encoding-equipped musical robot network to provide composers with intuitive musical commands for the robots. The musical robots introduced, described, and applied in this thesis were conceived of as musical instruments for performance and installation use by artists. This thesis closes with an exhibition of the performance and installation uses of these new robots and with a discussion of future research directions

Tangibility and Richness in Digital Musical Instrument Design

PhDThe sense of touch plays a fundamental role in musical performance: alongside hearing, it is the primary sensory modality used when interacting with musical instruments. Learning to play a musical instrument is one of the most developed haptic cultural practices, and within acoustic musical practice at large, the importance of touch and its close relationship to virtuosity and expression is well recognised. With digital musical instruments (DMIs) – instruments involving a combination of sensors and a digital sound engine – touch-mediated interaction remains the foremost means of control, but the interfaces of such instruments do not yet engage with the full spectrum of sensorimotor capabilities of a performer. This poses compelling questions for digital instrument design: how does the nuance and richness of physical interaction with an instrument manifest itself in the digital domain? Which design parameters are most important for haptic experience, and how do these parameters affect musical performance? Built around three practical studies which utilise DMIs as technology probes, this thesis addresses these questions from the point of view of design, of empirical musicology, and of tangible computing. In the first study musicians played a DMI with continuous pitch control and vibrotactile feedback in order to understand how dynamic tactile feedback can be implemented and how it influences musician experience and performance. The results suggest that certain vibrotactile feedback conditions can increase musicians’ tuning accuracy, but also disrupt temporal performance. The second study examines the influence of asynchronies between audio and haptic feedback. Two groups of musicians, amateurs and professional percussionists, were tasked with performing on a percussive DMI with variable action-sound latency. Differences between the two groups in terms of temporal accuracy and quality judgements illustrate the complex effects of asynchronous multimodal feedback. In the third study guitar-derivative DMIs with variable levels of control richness were observed with non-musicians and guitarists. The results from this study help clarify the relationship between tangible design factors, sensorimotor expertise and instrument behaviour. This thesis introduces a descriptive model of performer-instrument interaction, the projection model, which unites the design investigations from each study and provides a series of reflections and suggestions on the role of touch in DMI design.Doctoral Training Centre for Media and Arts Technolog

Not All Gestures Are Created Equal: Gesture and Visual Feedback in Interaction Spaces.

As multi-touch mobile computing devices and open-air gesture sensing technology become increasingly commoditized and affordable, they are also becoming more widely adopted. It became necessary to create new interaction design specifically for gesture-based interfaces to meet the growing needs of users. However, a deeper understanding of the interplay between gesture, and visual and sonic output is needed to make meaningful advances in design. This thesis addresses this crucial step in development by investigating the interrelation between gesture-based input, and visual representation and feedback, in gesture-driven creative computing. This thesis underscores the importance that not all gestures are created equal, and there are multiple factors that affect their performance. For example, a drag gesture in visual programming scenario performs differently than in a target acquisition task. The work presented here (i) examines the role of visual representation and mapping in gesture input, (ii) quantifies user performance differences in gesture input to examine the effect of multiple factors on gesture interactions, and (iii) develops tools and platforms for exploring visual representations of gestures. A range of gesture spaces and scenarios, from continuous sound control with open-air gestures to mobile visual programming with discrete gesture-driven commands, was assessed. Findings from this thesis reveals a rich space of complex interrelations between gesture input and visual feedback and representations. The contributions of this thesis also includes the development of an augmented musical keyboard with 3-D continuous gesture input and projected visualization, as well as a touch-driven visual programming environment for interactively constructing dynamic interfaces. These designs were evaluated by a series of user studies in which gesture-to-sound mapping was found to have a significant affect on user performance, along with other factors such as the selection of visual representation and device size. A number of counter-intuitive findings point to the potentially complex interactions between factors such as device size, task and scenarios, which exposes the need for further research. For example, the size of the device was found to have contradictory effects in two different scenarios. Furthermore, this work presents a multi-touch gestural environment to support the prototyping of gesture interactions.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113456/1/yangqi_1.pd

Live electronics in live performance : a performance practice emerging from the piano+ used in free improvisation

This thesis explores a performance practice within free improvisation. This is not a theory based improvisation – performances do not require specific preparation and the music refrains from repetition of musical structures. It engages in investigative and experimental approaches emerging from holistic considerations of acoustics, interaction and instrument, and also philosophy, psychology, sociopolitics and technology. The performance practice explores modes and approaches to working with the given potentiality of an electronically augmented acoustic instrument and involves the development of a suitably flexible computerised performance system, the piano+, combining extended techniques and real-time electroacoustic processes, which has the acoustic piano at its core. Contingencies of acoustic events and performance gestures – captured by audio analysis and sensors and combined to control the parameter space of computer processes – manipulate the fundamental properties of sound, timbre and time. Spherical abstractions, developed under consideration of Agamben’s potentiality and Sloterdijk’s philosophical theory of spheres, allow a shared metaphor for technical, instrumental, personal, and interpersonal concerns. This facilitates a theoretical approach for heuristic and investigative improvisation where performance is considered ‘Ereignis’ (an event) for sociopolitically aware activities that draw on the situational potentiality and present themselves in fragile and context dependent forms. Ever new relationships can be found and developed, but can equally be lost. Sloterdijk supplied the concept of knowledge resulting from equipping our ‘inner space’, an image suiting non-linearity of thought that transpires from Kuhl’s psychological PSI-theory to explain human motivation and behaviour. The role of technology – diversion and subversion of sound and activity – creates a space between performer and instrument that retains a fundamental pianism but defies expectation and anticipation. Responsibility for one’s actions is required to deal with the unexpected without resorting to preliminary strategies restricting potential discourses, particularly within ensemble situations. This type of performance embraces the ‘Ereignis’.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Live Electronics in Live Performance: A Performance Practice Emerging from the piano+ used in Free Improvisation.

This thesis explores a performance practice within free improvisation. This is not a theory based improvisation – performances do not require specific preparation and the music refrains from repetition of musical structures. It engages in investigative and experimental approaches emerging from holistic considerations of acoustics, interaction and instrument, and also philosophy, psychology, sociopolitics and technology. The performance practice explores modes and approaches to working with the given potentiality of an electronically augmented acoustic instrument and involves the development of a suitably flexible computerised performance system, the piano+, combining extended techniques and real-time electroacoustic processes, which has the acoustic piano at its core. Contingencies of acoustic events and performance gestures – captured by audio analysis and sensors and combined to control the parameter space of computer processes – manipulate the fundamental properties of sound, timbre and time. Spherical abstractions, developed under consideration of Agamben’s potentiality and Sloterdijk’s philosophical theory of spheres, allow a shared metaphor for technical, instrumental, personal, and interpersonal concerns. This facilitates a theoretical approach for heuristic and investigative improvisation where performance is considered ‘Ereignis’ (an event) for sociopolitically aware activities that draw on the situational potentiality and present themselves in fragile and context dependent forms. Ever new relationships can be found and developed, but can equally be lost. Sloterdijk supplied the concept of knowledge resulting from equipping our ‘inner space’, an image suiting non-linearity of thought that transpires from Kuhl’s psychological PSI-theory to explain human motivation and behaviour. The role of technology – diversion and subversion of sound and activity – creates a space between performer and instrument that retains a fundamental pianism but defies expectation and anticipation. Responsibility for one’s actions is required to deal with the unexpected without resorting to preliminary strategies restricting potential discourses, particularly within ensemble situations. This type of performance embraces the ‘Ereignis’

Extending physical instruments using sampled acoustics

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2007.Includes bibliographical references (p. 133-138).This thesis presents a system architecture for creating hybrid digital-acoustic percussion instruments by combining extensions of existing signal processing techniques with specially-designed semi-acoustic physical controllers. This work aims to provide greater realism to digital percussion, gaining much of the richness and understandability of acoustic instruments while preserving the flexibility of digital systems. For this thesis, I have collaborated with percussionists to develop a range of instruments, to refine and extend the algorithmic and physical designs, and to determine successful models of interaction. Conventional percussion controllers measure and discretize the intensity of strikes into discrete trigger messages, but they also ignore the timbre of the hits and fail to track more ambiguous input. In this work, the continuous acoustic output of a struck physical object is processed to add the resonance of a sampled instrument. This is achieved by employing existing low-latency convolution algorithms which have been extended to give the player control over features such as damping, spectral flattening, nonlinear effects, and pitch.(cont.) One of the advantages of this approach is that light taps, scrapes, rubs, or stirring with brushes all take on a hybrid timbre of the real and sampled sound that is surprisingly realistic and controllable. Since part of its behavior is inherently acoustic, a player's intuition about interacting with physical objects can be applied to controlling it. The ability to transform the apparent acoustic properties of objects also suggests applications to HCI and product design contexts.by Roberto Mario Aimi.Ph.D

Violin Augmentation Techniques for Learning Assistance

PhDLearning violin is a challenging task requiring execution of pitch tasks with the left hand using a strong aural feedback loop for correctly adjusting pitch, concurrent with the right hand moving a bow precisely with correct pressure across strings. Real-time technological assistance can help a student gain feedback and understanding helpful for learning and maintaining motivation. This thesis presents real-time low-cost low-latency violin augmentations that can be used to assist learning the violin along with other real-time performance tasks. To capture bow performance, we demonstrate a new means of bow tracking by measuring bow hair de ection from the bow hair being pressed against the string. Using near- eld optical sensors placed along the bow we are able to estimate bow position and pressure through linear regression from training samples. For left hand pitch tracking, we introduce low cost means for tracking nger position and illustrate the combination of sensed results with audio processing to achieve high accuracy low-latency pitch tracking. We subsequently verify our new tracking methods' e ectiveness and usefulness demonstrating low-latency note onset detection and control of real-time performance visuals. To help tackle the challenge of intonation, we used our pitch estimation to develop low latency pitch correction. Using expert performers, we veri ed that fully correcting pitch is not only disconcerting but breaks a violinist's learned pitch feedback loop resulting in worse asplayed performance. However, partial pitch correction, though also linked to worse as-played performance, did not lead to a signi cantly negative experience con rming its potential for use to temporarily reduce barriers to success. Subsequently, in a study with beginners, we veri ed that when the pitch feedback loop is underdeveloped, automatic pitch correction did not signi cantly hinder performance, but o ered an enjoyable low-pitch error experience and that providing an automatic target guide pitch was helpful in correcting performed pitch error