Designing and Composing for Interdependent Collaborative Performance with Physics-Based Virtual Instruments

Interdependent collaboration is a system of live musical performance in which performers can directly manipulate each other’s musical outcomes. While most collaborative musical systems implement electronic communication channels between players that allow for parameter mappings, remote transmissions of actions and intentions, or exchanges of musical fragments, they interrupt the energy continuum between gesture and sound, breaking our cognitive representation of gesture to sound dynamics. Physics-based virtual instruments allow for acoustically and physically plausible behaviors that are related to (and can be extended beyond) our experience of the physical world. They inherently maintain and respect a representation of the gesture to sound energy continuum. This research explores the design and implementation of custom physics-based virtual instruments for realtime interdependent collaborative performance. It leverages the inherently physically plausible behaviors of physics-based models to create dynamic, nuanced, and expressive interconnections between performers. Design considerations, criteria, and frameworks are distilled from the literature in order to develop three new physics-based virtual instruments and associated compositions intended for dissemination and live performance by the electronic music and instrumental music communities. Conceptual, technical, and artistic details and challenges are described, and reflections and evaluations by the composer-designer and performers are documented

Hardward and algorithm architectures for real-time additive synthesis

Additive synthesis is a fundamental computer music synthesis paradigm tracing its origins to the work of Fourier and Helmholtz. Rudimentary implementation linearly combines harmonic sinusoids (or partials) to generate tones whose perceived timbral characteristics are a strong function of the partial amplitude spectrum. Having evolved over time, additive synthesis describes a collection of algorithms each characterised by the time-varying linear combination of basis components to generate temporal evolution of timbre. Basis components include exactly harmonic partials, inharmonic partials with time-varying frequency or non-sinusoidal waveforms each with distinct spectral characteristics. Additive synthesis of polyphonic musical instrument tones requires a large number of independently controlled partials incurring a large computational overhead whose investigation and reduction is a key motivator for this work. The thesis begins with a review of prevalent synthesis techniques setting additive synthesis in context and introducing the spectrum modelling paradigm which provides baseline spectral data to the additive synthesis process obtained from the analysis of natural sounds. We proceed to investigate recursive and phase accumulating digital sinusoidal oscillator algorithms, defining specific metrics to quantify relative performance. The concepts of phase accumulation, table lookup phase-amplitude mapping and interpolated fractional addressing are introduced and developed and shown to underpin an additive synthesis subclass - wavetable lookup synthesis (WLS). WLS performance is simulated against specific metrics and parameter conditions peculiar to computer music requirements. We conclude by presenting processing architectures which accelerate computational throughput of specific WLS operations and the sinusoidal additive synthesis model. In particular, we introduce and investigate the concept of phase domain processing and present several “pipeline friendly” arithmetic architectures using this technique which implement the additive synthesis of sinusoidal partials

Proceedings of the Linux Audio Conference 2018

These proceedings contain all papers presented at the Linux Audio Conference 2018. The conference took place at c-base, Berlin, from June 7th - 10th, 2018 and was organized in cooperation with the Electronic Music Studio at TU Berlin

Automatic annotation of musical audio for interactive applications

PhDAs machines become more and more portable, and part of our everyday life, it becomes apparent that developing interactive and ubiquitous systems is an important aspect of new music applications created by the research community. We are interested in developing a robust layer for the automatic annotation of audio signals, to be used in various applications, from music search engines to interactive installations, and in various contexts, from embedded devices to audio content servers. We propose adaptations of existing signal processing techniques to a real time context. Amongst these annotation techniques, we concentrate on low and mid-level tasks such as onset detection, pitch tracking, tempo extraction and note modelling. We present a framework to extract these annotations and evaluate the performances of different algorithms. The first task is to detect onsets and offsets in audio streams within short latencies. The segmentation of audio streams into temporal objects enables various manipulation and analysis of metrical structure. Evaluation of different algorithms and their adaptation to real time are described. We then tackle the problem of fundamental frequency estimation, again trying to reduce both the delay and the computational cost. Different algorithms are implemented for real time and experimented on monophonic recordings and complex signals. Spectral analysis can be used to label the temporal segments; the estimation of higher level descriptions is approached. Techniques for modelling of note objects and localisation of beats are implemented and discussed. Applications of our framework include live and interactive music installations, and more generally tools for the composers and sound engineers. Speed optimisations may bring a significant improvement to various automated tasks, such as automatic classification and recommendation systems. We describe the design of our software solution, for our research purposes and in view of its integration within other systems.EU-FP6-IST-507142 project SIMAC (Semantic Interaction with Music Audio Contents); EPSRC grants GR/R54620; GR/S75802/01

Automatic sound synthesizer programming: techniques and applications

The aim of this thesis is to investigate techniques for, and applications of automatic sound synthesizer programming. An automatic sound synthesizer programmer is a system which removes the requirement to explicitly specify parameter settings for a sound synthesis algorithm from the user. Two forms of these systems are discussed in this thesis: tone matching programmers and synthesis space explorers. A tone matching programmer takes at its input a sound synthesis algorithm and a desired target sound. At its output it produces a configuration for the sound synthesis algorithm which causes it to emit a similar sound to the target. The techniques for achieving this that are investigated are genetic algorithms, neural networks, hill climbers and data driven approaches. A synthesis space explorer provides a user with a representation of the space of possible sounds that a synthesizer can produce and allows them to interactively explore this space. The applications of automatic sound synthesizer programming that are investigated include studio tools, an autonomous musical agent and a self-reprogramming drum machine. The research employs several methodologies: the development of novel software frameworks and tools, the examination of existing software at the source code and performance levels and user trials of the tools and software. The main contributions made are: a method for visualisation of sound synthesis space and low dimensional control of sound synthesizers; a general purpose framework for the deployment and testing of sound synthesis and optimisation algorithms in the SuperCollider language sclang; a comparison of a variety of optimisation techniques for sound synthesizer programming; an analysis of sound synthesizer error surfaces; a general purpose sound synthesizer programmer compatible with industry standard tools; an automatic improviser which passes a loose equivalent of the Turing test for Jazz musicians, i.e. being half of a man-machine duet which was rated as one of the best sessions of 2009 on the BBC's 'Jazz on 3' programme

Retrieving Ambiguous Sounds Using Perceptual Timbral Attributes in Audio Production Environments

For over an decade, one of the well identified problem within audio production environments is the effective retrieval and management of sound libraries. Most of the self-recorded and commercially produced sound libraries are usually well structured in terms of meta-data and textual descriptions and thus allowing traditional text-based retrieval approaches to obtain satisfiable results. However, traditional information retrieval techniques pose limitations in retrieving ambiguous sound collections (ie. sounds with no identifiable origin, foley sounds, synthesized sound effects, abstract sounds) due to the difficulties in textual descriptions and the complex psychoacoustic nature of the sound. Early psychoacoustical studies propose perceptual acoustical qualities as an effective way of describing these category of sounds [1]. In Music Information Retrieval (MIR) studies, this problem were mostly studied and explored in context of content-based audio retrieval. However, we observed that most of the commercial available systems in the market neither integrated advanced content-based sound descriptions nor the visualization and interface design approaches evolved in the last years. Our research was mainly aimed to investigate two things; 1. Development of audio retrieval system incorporating high level timbral features as search parameters. 2. Investigate user-centered approach in integrating these features into audio production pipelines using expert-user studies. In this project, We present an prototype which is similar to traditional sound browsers (list-based browsing) with an added functionality of filtering and ranking sounds by perceptual timbral features such as brightness, depth, roughness and hardness. Our main focus was on the retrieval process by timbral features. Inspiring from the recent focus on user-centered systems ([2], [3]) in the MIR community, in-depth interviews and qualitative evaluation of the system were conducted with expert-user in order to identify the underlying problems. Our studies observed the potential applications of high-level perceptual timbral features in audio production pipelines using a probe system and expert-user studies. We also outlined future guidelines and possible improvements to the system from the outcomes of this research

Using simple controls to manipulate complex objects : application to the Drum-Boy interactive percussion system

Thesis (M.S.)--Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1993.Includes bibliographical references (leaves 90-93).by Fumiaki Matsumoto.M.S

Sound Synthesis Using Programmable System-On-Chip Devices

The last 20 years has witnessed a resurgence of interest in analogue synthesisers 1 . Manufacturers, such as Moog and Sequential Circuits, that had disappeared from the commercial marketplace by the end of the 1980’s, have reappeared with an impressive line of products. Other established companies such as Korg and Roland, as well as entrants that had made their name with digital technology, such as Novation and Arturia, have released analogue instruments. Although the feature set of digital synthesisers is extensive and with a falling comparative cost, the analogue market has continued to grow with more and more devices coming available. They are perceived to be of superior sound quality by users, but their primary drawback is price, as numerous discrete components or specialist integrated circuits are required. This thesis introduces two novel low-cost approaches to building analogue-type synthesisers. Such a low-cost instrument could have applications in an educational laboratory environment for synthesisers. The first approach is to exploit a new mixed-signal technology called the Programmable System-on-Chip (PSoC), which includes a CPU core and mixed-signal arrays of configurable integrated analogue and digital peripherals. The second exploits a System on Chip (SoC) comprising an ARM-based (Acorn RISC Machine) processor and a Field-Programmable Gate Array (FPGA). Two synthesisers were built and were evaluated for difficulty of implementation and assessed for their sound quality. The design and testing process was recorded and documented in detail. The mixed-signal approach was found to be cheaper than the FPGA-approach both in terms of component costs and development time compared to the FPGA-based approach. Actually, the FPGA-approach was determined to be prohibitively expensive in terms of the development time incurred. The sound quality analysis demonstrated that both instruments were perceived by users to be of high quality, achieving a noticeable analogue sound. Future work would be to repackage the PSoC system and modules into rack-mounted form for use in an educational synthesiser laboratory environment

Sound Synthesis Using Programmable System-On-Chip Devices

The last 20 years has witnessed a resurgence of interest in analogue synthesisers 1 . Manufacturers, such as Moog and Sequential Circuits, that had disappeared from the commercial marketplace by the end of the 1980’s, have reappeared with an impressive line of products. Other established companies such as Korg and Roland, as well as entrants that had made their name with digital technology, such as Novation and Arturia, have released analogue instruments. Although the feature set of digital synthesisers is extensive and with a falling comparative cost, the analogue market has continued to grow with more and more devices coming available. They are perceived to be of superior sound quality by users, but their primary drawback is price, as numerous discrete components or specialist integrated circuits are required. This thesis introduces two novel low-cost approaches to building analogue-type synthesisers. Such a low-cost instrument could have applications in an educational laboratory environment for synthesisers. The first approach is to exploit a new mixed-signal technology called the Programmable System-on-Chip (PSoC), which includes a CPU core and mixed-signal arrays of configurable integrated analogue and digital peripherals. The second exploits a System on Chip (SoC) comprising an ARM-based (Acorn RISC Machine) processor and a Field-Programmable Gate Array (FPGA). Two synthesisers were built and were evaluated for difficulty of implementation and assessed for their sound quality. The design and testing process was recorded and documented in detail. The mixed-signal approach was found to be cheaper than the FPGA-approach both in terms of component costs and development time compared to the FPGA-based approach. Actually, the FPGA-approach was determined to be prohibitively expensive in terms of the development time incurred. The sound quality analysis demonstrated that both instruments were perceived by users to be of high quality, achieving a noticeable analogue sound. Future work would be to repackage the PSoC system and modules into rack-mounted form for use in an educational synthesiser laboratory environment