Automatic annotation of musical audio for interactive applications

PhDAs machines become more and more portable, and part of our everyday life, it becomes apparent that developing interactive and ubiquitous systems is an important aspect of new music applications created by the research community. We are interested in developing a robust layer for the automatic annotation of audio signals, to be used in various applications, from music search engines to interactive installations, and in various contexts, from embedded devices to audio content servers. We propose adaptations of existing signal processing techniques to a real time context. Amongst these annotation techniques, we concentrate on low and mid-level tasks such as onset detection, pitch tracking, tempo extraction and note modelling. We present a framework to extract these annotations and evaluate the performances of different algorithms. The first task is to detect onsets and offsets in audio streams within short latencies. The segmentation of audio streams into temporal objects enables various manipulation and analysis of metrical structure. Evaluation of different algorithms and their adaptation to real time are described. We then tackle the problem of fundamental frequency estimation, again trying to reduce both the delay and the computational cost. Different algorithms are implemented for real time and experimented on monophonic recordings and complex signals. Spectral analysis can be used to label the temporal segments; the estimation of higher level descriptions is approached. Techniques for modelling of note objects and localisation of beats are implemented and discussed. Applications of our framework include live and interactive music installations, and more generally tools for the composers and sound engineers. Speed optimisations may bring a significant improvement to various automated tasks, such as automatic classification and recommendation systems. We describe the design of our software solution, for our research purposes and in view of its integration within other systems.EU-FP6-IST-507142 project SIMAC (Semantic Interaction with Music Audio Contents); EPSRC grants GR/R54620; GR/S75802/01

Design Strategies for Adaptive Social Composition: Collaborative Sound Environments

In order to develop successful collaborative music systems a variety of subtle interactions need to be identified and integrated. Gesture capture, motion tracking, real-time synthesis, environmental parameters and ubiquitous technologies can each be effectively used for developing innovative approaches to instrument design, sound installations, interactive music and generative systems. Current solutions tend to prioritise one or more of these approaches, refining a particular interface technology, software design or compositional approach developed for a specific composition, performer or installation environment. Within this diverse field a group of novel controllers, described as ‘Tangible Interfaces’ have been developed. These are intended for use by novices and in many cases follow a simple model of interaction controlling synthesis parameters through simple user actions. Other approaches offer sophisticated compositional frameworks, but many of these are idiosyncratic and highly personalised. As such they are difficult to engage with and ineffective for groups of novices. The objective of this research is to develop effective design strategies for implementing collaborative sound environments using key terms and vocabulary drawn from the available literature. This is articulated by combining an empathic design process with controlled sound perception and interaction experiments. The identified design strategies have been applied to the development of a new collaborative digital instrument. A range of technical and compositional approaches was considered to define this process, which can be described as Adaptive Social Composition. Dan Livingston

Pitchclass2vec: Symbolic Music Structure Segmentation with Chord Embeddings

Structure perception is a fundamental aspect of music cognition in humans. Historically, the hierarchical organization of music into structures served as a narrative device for conveying meaning, creating expectancy, and evoking emotions in the listener. Thereby, musical structures play an essential role in music composition, as they shape the musical discourse through which the composer organises his ideas. In this paper, we present a novel music segmentation method, pitchclass2vec, based on symbolic chord annotations, which are embedded into continuous vector representations using both natural language processing techniques and custom-made encodings. Our algorithm is based on long-short term memory (LSTM) neural network and outperforms the state-of-the-art techniques based on symbolic chord annotations in the field

DEEP-RHYTHM FOR TEMPO ESTIMATION AND RHYTHM PATTERN RECOGNITION

International audienceIt has been shown that the harmonic series at the tempo frequency of the onset-strength-function of an audio signal accurately describes its rhythm pattern and can be used to perform tempo or rhythm pattern estimation. Recently, in the case of multi-pitch estimation, the depth of the input layer of a convolutional network has been used to represent the harmonic series of pitch candidates. We use a similar idea here to represent the harmonic series of tempo candidates. We propose the Harmonic-Constant-Q-Modulation which represents, using a 4D-tensors, the harmonic series of modulation frequencies (considered as tempo frequencies) in several acoustic frequency bands over time. This representation is used as input to a convolutional network which is trained to estimate tempo or rhythm pattern classes. Using a large number of datasets, we evaluate the performance of our approach and compare it with previous approaches. We show that it slightly increases Accuracy-1 for tempo estimation but not the average-mean-Recall for rhythm pattern recognition

Defragmenting Beethoven: Sound appropriation as bridge between classical tradition and electroacoustic music

This thesis serves as a written companion for two artistic-based research products built upon the concept of appropriation as connecting bridge between music technology and the classical tradition of music composition. The first artistic work is a set of 9 pieces called "Collages Vol 2", a continuation of a first release meant to be an exploratory work for the present research. The second work is the sound interaction design, and the creation of two compositions for the Network of Intelligent Sound Agents, or "NOISA", built at the Sound and Physical Interaction Research Group from the Department of Media, Aalto University. After providing context and a short survey of influences on music appropriation, I made a comprehensive documentation of each of the pieces created for this thesis, describing form, content, compositional approach and sound processing in a systematic way. I investigated on the diverse forms of appropriation as a technique for electroacoustic music composition. The most influential references for my work are documented in this written work: From the historical approach of appropriation to borrowing in music of the XX century and recent times; including a description of the first volume of my original Collages. Later on, I described my second collection of Collages and the utilisation of appropriation theories in the context of NOISA, a music interface for live performance. Finally, there is a section dedicated to a discussion featuring a commentary of a number of reviews of "Collages” preceding a closing segment with conclusions and further plans to expand this research in the future

Este Futuro es Otro Futuro: The role of social discourse on the [under]development of contemporary academic electronic music in Perú

This dissertation explores the history of Peruvian electroacoustic and electronic musical experimentation since its arrival to the country, while confronting the particular issues that have kept their practice from being academically embraced or implemented into the official channels for musical learning. It reviews the failures and successes of the Peruvian technologically based musical arts in front of the social structures and ideologies that have historically permeated all the cultural activities of the country. By moving away from a traditional technical and formalist approach that confronts academically based electroacoustic and electronic music from the perspective of the musical object, this work unveils the highly politicized history of responses towards these arts in Perú and the direct role of the particular nationalistic postcolonial models on their development or lack of it. It is within a plural and ambiguous set of nationalists, nativist, socialist, anti-foreign and indigenist ideologies, analyzed in this work, that we can identify a constant: the suspicion towards, and ultimately, the rejection of technologically based musical traditions as an element in the construction of a history of Peruvian music. Most actors in this work, whether composers, performers or music researchers, that have attempted to dedicate their efforts towards the implementation of technology beyond the logic of the recording studio and tech-support, have had to negotiate their participation in musical academic and popular cultures according to the way they were perceived by the political and cultural surroundings of their time. This situation would force many of them to construct the cultural identities that could allow them to participate in national institutionalized discourses and practices, moving away from technology or maintaining it as a peripheral activity. In the majority of cases, those musicians interested in technology would either migrate in search of training and professional opportunities, or shift direction towards other musical practices. By analyzing, in this work, the national discourses of/about our actors and the silence of the institutions (public and private) we reveal an agent-structure relational behavioral pattern toward musical technology and related sound arts that is at the heart of its historical undermining

Measuring Expressive Music Performances: a Performance Science Model using Symbolic Approximation

Music Performance Science (MPS), sometimes termed systematic musicology in Northern Europe, is concerned with designing, testing and applying quantitative measurements to music performances. It has applications in art musics, jazz and other genres. It is least concerned with aesthetic judgements or with ontological considerations of artworks that stand alone from their instantiations in performances. Musicians deliver expressive performances by manipulating multiple, simultaneous variables including, but not limited to: tempo, acceleration and deceleration, dynamics, rates of change of dynamic levels, intonation and articulation. There are significant complexities when handling multivariate music datasets of significant scale. A critical issue in analyzing any types of large datasets is the likelihood of detecting meaningless relationships the more dimensions are included. One possible choice is to create algorithms that address both volume and complexity. Another, and the approach chosen here, is to apply techniques that reduce both the dimensionality and numerosity of the music datasets while assuring the statistical significance of results. This dissertation describes a flexible computational model, based on symbolic approximation of timeseries, that can extract time-related characteristics of music performances to generate performance fingerprints (dissimilarities from an ‘average performance’) to be used for comparative purposes. The model is applied to recordings of Arnold Schoenberg’s Phantasy for Violin with Piano Accompaniment, Opus 47 (1949), having initially been validated on Chopin Mazurkas.1 The results are subsequently used to test hypotheses about evolution in performance styles of the Phantasy since its composition. It is hoped that further research will examine other works and types of music in order to improve this model and make it useful to other music researchers. In addition to its benefits for performance analysis, it is suggested that the model has clear applications at least in music fraud detection, Music Information Retrieval (MIR) and in pedagogical applications for music education

X Reality Networked Performance: Message Based Distributed Systems For Controlling And Presenting Multiple Realities

X reality networked performances connect physical, fictional and computer generated realities in a new world of performance, one that is without geographical bounds and that can include many physical locations—with their own performers and audience members— within a single event. They explore a unique medium while drawing on historical and contemporary performing arts practices that normally occur within the confines of a single physical location. Such performances present a special set of requirements on the system that supports them. They need to access and integrate all the systems that are typically found in the physical place of the performance (such as theatre lighting) with those that are unique to the medium, such as network technologies and environments for the delivery of computer generated realities. Yet, no suitable systems or frameworks have been developed to support them. Technologies are available (such as LoLA and UltraGrid) that support individual aspects—like audio/video streaming—but which do not address the wider requirements of controlling and synchronising, of integrating all these technologies into a system of systems for X reality networked performance. Therefore, this research investigates the creation of a systems framework whereby existing hardware and software components can be continuously integrated with bespoke components to provide a platform for the delivery of X reality networked performances. The methodological approach to this investigation is through the lens of the author’s previous experience in other fields of complex systems integration, including, approaches employed in the design and integration of avionics systems. Specifically, it tests if a systems integration approach to providing a technical platform for X reality networked performances, one that employs strongly‐defined interfaces and communication protocols, and that is based on open and industry standards, delivers an elegant platform that can be characterised as: deterministic, reliable, extendable, scalable, reconfigurable, testable and cost effective. iii The platform for X reality networked performance has been developed iteratively—using the results of a framework investigation—and tested in four different performance projects over a period of 24‐months, in ten different venues, across five countries. The research concludes that the enabling framework is well suited to the delivery of X reality networked performances. Also, that the approaches employed could equally be applied to the needs of other arts practitioners who rely on complex technical systems for the creation and delivery of their work

An Abstraction Framework for Tangible Interactive Surfaces

This cumulative dissertation discusses - by the example of four subsequent publications - the various layers of a tangible interaction framework, which has been developed in conjunction with an electronic musical instrument with a tabletop tangible user interface. Based on the experiences that have been collected during the design and implementation of that particular musical application, this research mainly concentrates on the definition of a general-purpose abstraction model for the encapsulation of physical interface components that are commonly employed in the context of an interactive surface environment. Along with a detailed description of the underlying abstraction model, this dissertation also describes an actual implementation in the form of a detailed protocol syntax, which constitutes the common element of a distributed architecture for the construction of surface-based tangible user interfaces. The initial implementation of the presented abstraction model within an actual application toolkit is comprised of the TUIO protocol and the related computer-vision based object and multi-touch tracking software reacTIVision, along with its principal application within the Reactable synthesizer. The dissertation concludes with an evaluation and extension of the initial TUIO model, by presenting TUIO2 - a next generation abstraction model designed for a more comprehensive range of tangible interaction platforms and related application scenarios