QuBits, an Interactive Virtual Reality Project and Compositional Space for Sound and Image

AbstractQuBits, an Interactive Virtual Reality Project and Compositional Spacefor Sound and ImagebyJonathan KulpaDoctor of Philosophy in Music CompositionUniversity of California, BerkeleyProfessor Edmund Campion, ChairThis paper describes the QuBits project, a virtual reality (VR) environment created by the composer, offering an expanded medium for musical experience with integrated space and visuals. The QuBits VR environment is the essential supporting material for the dissertation and this paper provides a full description of the project.The environment was designed to explore a musical aesthetic valuing sound mass, spatial sound, evolution, and algorithmically generated sonic structures. Additionally, the user of the VR system plays a key role in shaping these musical elements. The user first must discover what behaviors are possible by exploring and through chance encounters. They can then shape each discovered behavior with nuance if they choose. In the VR environment, each sound has a corresponding visual component. To achieve this, a system was built with two software platforms, one for digital sound processing and another for 3D graphics and algorithmic event generation. These platforms communicate via Open Sound Control (OSC). The sounds are a mix of real world sampled sound, granular synthesis, and real-time generated synthetic sound. The QuBits VR environment represents the results of this methodology. Pros and cons of the methodology are discussed, as well as implications for future projects

Real-time sound synthesis on a multi-processor platform

Real-time sound synthesis means that the calculation and output of each sound sample for a channel of audio information must be completed within a sample period. At a broadcasting standard, a sampling rate of 32,000 Hz, the maximum period available is 31.25 μsec. Such requirements demand a large amount of data processing power. An effective solution for this problem is a multi-processor platform; a parallel and distributed processing system. The suitability of the MIDI [Music Instrument Digital Interface] standard, published in 1983, as a controller for real-time applications is examined. Many musicians have expressed doubts on the decade old standard's ability for real-time performance. These have been investigated by measuring timing in various musical gestures, and by comparing these with the subjective characteristics of human perception. An implementation and its optimisation of real-time additive synthesis programs on a multi-transputer network are described. A prototype 81-polyphonic-note- organ configuration was implemented. By devising and deploying monitoring processes, the network's performance was measured and enhanced, leading to an efficient usage; the 88-note configuration. Since 88 simultaneous notes are rarely necessary in most performances, a scheduling program for dynamic note allocation was then introduced to achieve further efficiency gains. Considering calculation redundancies still further, a multi-sampling rate approach was applied as a further step to achieve an optimal performance. The theories underlining sound granulation, as a means of constructing complex sounds from grains, and the real-time implementation of this technique are outlined. The idea of sound granulation is quite similar to the quantum-wave theory, "acoustic quanta". Despite the conceptual simplicity, the signal processing requirements set tough demands, providing a challenge for this audio synthesis engine. Three issues arising from the results of the implementations above are discussed; the efficiency of the applications implemented, provisions for new processors and an optimal network architecture for sound synthesis

Hybridism: a practice-led investigation

Keele University PhD Thesi

Sample-level sound synthesis with recurrent neural networks and conceptors

Conceptors are a recent development in the field of reservoir computing; they can be used to influence the dynamics of recurrent neural networks (RNNs), enabling generation of arbitrary patterns based on training data. Conceptors allow interpolation and extrapolation between patterns, and also provide a system of boolean logic for combining patterns together. Generation and manipulation of arbitrary patterns using conceptors has significant potential as a sound synthesis method for applications in computer music but has yet to be explored. Conceptors are untested with the generation of multi-timbre audio patterns, and little testing has been done on scalability to longer patterns required for audio. A novel method of sound synthesis based on conceptors is introduced. Conceptular Synthesis is based on granular synthesis; sets of conceptors are trained to recall varying patterns from a single RNN, then a runtime mechanism switches between them, generating short patterns which are recombined into a longer sound. The quality of sound resynthesis using this technique is experimentally evaluated. Conceptor models are shown to resynthesise audio with a comparable quality to a close equivalent technique using echo state networks with stored patterns and output feedback. Conceptor models are also shown to excel in their malleability and potential for creative sound manipulation, in comparison to echo state network models which tend to fail when the same manipulations are applied. Examples are given demonstrating creative sonic possibilities, by exploiting conceptor pattern morphing, boolean conceptor logic and manipulation of RNN dynamics. Limitations of conceptor models are revealed with regards to reproduction quality, and pragmatic limitations are also shown, where rises in computation and memory requirements preclude the use of these models for training with longer sound samples. The techniques presented here represent an initial exploration of the sound synthesis potential of conceptors, demonstrating possible creative applications in sound design; future possibilities and research questions are outlined

Designing virtual spaces: redefining radio art through digital control

Radio Art is a composition practice that is constantly evolving. Artists share a commonality to redefine, reinvent, and repurpose analogue radio. It is an art that often bends to the will of antiqued technology, celebrating a wide pallet of found sounds. This research extends the boundaries of the art form by exploring Radio Art through sonic-centric lens and establishing a consistent and reproducible compositional framework. By shifting radio from a found object to an instrument, I have deconstructed its sonic aesthetics into two parallel materials for composition, gestural noise and broadcast signal. When tuning an analogue radio to a signal, relationships between these materials unfold. Contrast is a term found throughout my research. Contrast is embodied throughout radio and its history; radio is used as both a scientific communication device and for artistic expression. it is a symbol of democracy and oppression. Radio produces broadcast noise and signal, creating poetic reception, such as control and chaos, anxiety and ecstasy, distance and closeness. This research explores the characteristics of these forces and materials as a symbiotic relationship of unfolding radiophonic behaviours. A major focus of this research is the control of analogue radio through deconstruction and composition. I embarked on a twenty-four-month development period to build a Digital Audio Workstation called Radiophonic Environmental Designer, (RED). RED enables composers to create virtual radiophonic environments that are navigated by rotating the dial. Material is positioned along a horizon, and tuning behaviours sculpted. There is also a physical interface embedded into an analogue radio shell to control the virtual tuning, namely, Broadcast Link-up Environment, (BLUE). BLUE is an ad-on program offering an online digital platform for the diffusion of Radio Art. Using an internet connection and gyroscope technology that is built into most smart phones, a radiophonic environment is interacted through a purpose-built website. In my creative practice, analogue radio has been redesigned by adopting digital technological practices to control, edit and model it’s unique sound. In doing so, I reflect upon relationships between analogue and digital design principles through an extensive study on virtual analogue software and interfaces

Gesture and listening: towards a social and eco-systemic hyperinstrument composition

The research implements interactive music processes involving sound synthesis and symbolic treatments within a single environment. The algorithms are driven by classical instrumental performance through hybrid systems called hyperinstruments, in which the sensing of the performance gestures leads to open and goal-oriented generative music forms. The interactions are composed with MAX/Msp, designing contexts and relationships between real-time instrumental timbre analysis (sometimes with added inertial motion tracking) with a gesture-based idea of form shaping. Physical classical instruments are treated as interfaces, giving rise to the need to develop unconventional mapping strategies on account of the multi-dimensional and interconnecting quality of timbre. Performance and sound gestures are viewed as salient energies, phrasings and articulations carrying information about human intentions, in this way becoming able to change the musical behaviour of a composition inside a coded dramaturgy. The interactive networks are designed in order to integrate traditional music practices and “languages” with computational systems designed to be self-regulating, through the mediation of timbre space and performance gestural descriptions. Following its classic definition, technology aims to be mainly related not to mechanical practices but rather to rhetorical approaches: for this reason the software often foresees interactive scores, and must be performed in accordance with a set of external verbal (and video) explanations, whose technical detail should nevertheless not impair the most intuitive approach to music making

Homogeneity and Heterogeneity: two approaches for designing spatial sound

My research explores the process for designing and reproducing spatial sound through two personal sound-design projects: Pilate (2006) and Metamorphosis42 (2009). Located within the context of design, it explores two modes for spatial sound composition, which at its heart contemplates the potential for the phenomenon of the sound experience to guide the design and reproduction of spatial sound using electroacoustic techniques. The intent is to convey an approach to sound design, by combining the concepts with the tools and techniques used in sound composition, to create an impression of space that is perceived by the listener. This process has enabled me to draw a distinction in approach between my projects, based on the concepts, tools and techniques I apply in the act of making, to guide my work. In emphasising the sound space relation, the notions of the ‘homogeneous’ and the ‘heterogeneous’ emerge. I apply these terms in my research, to embody the essence of a compositional methodology, explored through an analysis of my projects and by engaging with a process of reflection and listening. On this basis, my research suggests that the notion of the ‘homogeneous’ builds on the physical relations of sound and space through an experience of sound within space. If our experience of sound in a real-world scenario can be considered coherent, then, in the context of a sound design process, my research proposes that a ‘homogeneous’ approach to sound composition seeks to faithfully reproduce the perceptually coherent sounding qualities observed in the real-world. This approach is predicated on the fact that the sensations of sound are the result of vibrating materials. The displacement, which occurs at the material surface, creates a pressure wave that moves through the atmosphere interacting with other materials within the environment. As a consequence, the pressure waves that reach our ears come from all directions, not just one and the physical qualities of the materials the pressure wave interacts with – their size, shape and density, for example, directly influence how sound is perceived. The notion of the ‘heterogeneous’ builds on the internal relation of sound and space - the space within sound, and how the physical and imagined qualities of space are interchangeable which conveys a dynamic and complex sense of space. A quality of this approach is that sound is suspended in its own time and space through a poetic re-imagination - an interior space in which sound is decoupled from its physical source. As François Bayle (2007, p.241) writes, ‘a sound from a transducer is like no other’. I interpret this as a way to think about the reproduced sound, which sits beyond the physical. I use the term ‘re-imagination’ to emphasise that the listener is not passive, but instead an active body engaged in this contemplation of sound. On this basis, the heterogeneous is not one singular homogeneous image bound entirely by the physical nature of the source. Instead, it is an affective experience that is bound to our senses and our emotions, reaching our ears through a patchwork of sounds that collide in space and time creating a transformation that occurs at the boundaries of perception

Snow Spell: An Interactive Composition for Erhu, Flute, Piano, Cello, and Max/MSP

Snow Spell is an interactive composition for erhu, flute, cello, piano, and Max/MSP interactive computer music system. This one-movement piece, Snow Spell, is intended to depict the beauty of a snow scene by presenting four different impressions of snow envisioned by the composer through music. The definition, history, and significance of interactive music are explored. Various modes of interactivity to control signal processing modules, and technical considerations for signal routing and level control in the interactive computer music system are also explored. Chinese music elements in Snow Spell including pentatonic scales, glissandi, and quotations from the Chinese folk tune River of Sorrow are investigated

Live electronics in live performance : a performance practice emerging from the piano+ used in free improvisation

This thesis explores a performance practice within free improvisation. This is not a theory based improvisation – performances do not require specific preparation and the music refrains from repetition of musical structures. It engages in investigative and experimental approaches emerging from holistic considerations of acoustics, interaction and instrument, and also philosophy, psychology, sociopolitics and technology. The performance practice explores modes and approaches to working with the given potentiality of an electronically augmented acoustic instrument and involves the development of a suitably flexible computerised performance system, the piano+, combining extended techniques and real-time electroacoustic processes, which has the acoustic piano at its core. Contingencies of acoustic events and performance gestures – captured by audio analysis and sensors and combined to control the parameter space of computer processes – manipulate the fundamental properties of sound, timbre and time. Spherical abstractions, developed under consideration of Agamben’s potentiality and Sloterdijk’s philosophical theory of spheres, allow a shared metaphor for technical, instrumental, personal, and interpersonal concerns. This facilitates a theoretical approach for heuristic and investigative improvisation where performance is considered ‘Ereignis’ (an event) for sociopolitically aware activities that draw on the situational potentiality and present themselves in fragile and context dependent forms. Ever new relationships can be found and developed, but can equally be lost. Sloterdijk supplied the concept of knowledge resulting from equipping our ‘inner space’, an image suiting non-linearity of thought that transpires from Kuhl’s psychological PSI-theory to explain human motivation and behaviour. The role of technology – diversion and subversion of sound and activity – creates a space between performer and instrument that retains a fundamental pianism but defies expectation and anticipation. Responsibility for one’s actions is required to deal with the unexpected without resorting to preliminary strategies restricting potential discourses, particularly within ensemble situations. This type of performance embraces the ‘Ereignis’.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Elaborazione di segmenti temporali

The text deals with sound processing in the discrete time domain, whereas a phase of segmentation, a subsequent analysis, and a temporal signal of the recomposition strategy in order to obtain transformations on audio resynthesized. Various algorithms are presented illustrating the historical analogue techniques as well as developments and implementations using the digital audio techniques. Anthology musical compositions are also cited where composers and musicians use these calculations, contributing to the evolution of techniques of electronic music and sound