Musical timbre: bridging perception with semantics

Musical timbre is a complex and multidimensional entity which provides information regarding the properties of a sound source (size, material, etc.). When it comes to music, however, timbre does not merely carry environmental information, but it also conveys aesthetic meaning. In this sense, semantic description of musical tones is used to express perceptual concepts related to artistic intention. Recent advances in sound processing and synthesis technology have enabled the production of unique timbral qualities which cannot be easily associated with a familiar musical instrument. Therefore, verbal description of these qualities facilitates communication between musicians, composers, producers, audio engineers etc. The development of a common semantic framework for musical timbre description could be exploited by intuitive sound synthesis and processing systems and could even influence the way in which music is being consumed. This work investigates the relationship between musical timbre perception and its semantics. A set of listening experiments in which participants from two different language groups (Greek and English) rated isolated musical tones on semantic scales has tested semantic universality of musical timbre. The results suggested that the salient semantic dimensions of timbre, namely: luminance, texture and mass, are indeed largely common between these two languages. The relationship between semantics and perception was further examined by comparing the previously identified semantic space with a perceptual timbre space (resulting from pairwise dissimilarity rating of the same stimuli). The two spaces featured a substantial amount of common variance suggesting that semantic description can largely capture timbre perception. Additionally, the acoustic correlates of the semantic and perceptual dimensions were investigated. This work concludes by introducing the concept of partial timbre through a listening experiment that demonstrates the influence of background white noise on the perception of musical tones. The results show that timbre is a relative percept which is influenced by the auditory environment

A perceptually motivated approach to timbre representation and visualisation.

Musical timbre is a complex phenomenon and is often understood in relation to the separation and comparison of different sound categories. The representation of musical timbre has traditionally consisted of instrumentation category (e.g. violin, piano) and articulation technique (e.g. pizzicato, staccato). Electroacoustic music places more emphasis on timbre variation as musical structure, and has highlighted the need for better, more in-depth forms of representation of musical timbre. Similarly, research from experimental psychology and audio signal analysis has deepened our understanding of the perception, description, and measurement of musical timbre, suggesting the possibility of more exact forms of representation that directly reference low-level descriptors of the audio signal (rather than high-level categories of sound or instrumentation). Research into the perception of timbre has shown that ratings of similarity between sounds can be used to arrange sounds in an N-dimensional perceptual timbre space, where each dimension relates to a particular axis of differentiation between sounds. Similarly, research into the description of timbre has shown that verbal descriptors can often be clustered into a number of categories, resulting in an N-dimensional semantic timbre space. Importantly, these semantic descriptors are often physical, material, and textural in nature. Audio signal processing techniques can be used to extract numeric descriptors of the spectral and dynamic content of an audio signal. Research has suggested correlations between these audio descriptors and different semantic descriptors and perceptual dimensions in perceptual timbre spaces. This thesis aims to develop a perceptually motivated approach to timbre representation by making use of correlations between semantic and acoustic descriptors of timbre. User studies are discussed that explored participant preferences for different visual mappings of acoustic timbre features. The results of these studies, together with results from existing research, have been used in the design and development of novel systems for timbre representation. These systems were developed both in the context of digital interfaces for sound design and music production, and in the context of real-time performance and generative audio-reactive visualisation. A generalised approach to perceptual timbre representation is presented and discussed with reference to the experimentation and resulting systems. The use of semantic visual mappings for low-level audio descriptors in the representation of timbre suggests that timbre would be better defined with reference to individual audio features and their variation over time. The experimental user studies and research-led development have highlighted specific techniques and audio-visual mappings that would be very useful to practitioners and researchers in the area of audio analysis and representation

From Line to Shape to Space: Composition as Representation of The Visual Elements

It is widely understood that the experience of music exists beyond that of just listening. The idea that audience members draw pictures in their mind or trace shapes with their hands while listening to music has been well studied. This thesis draws from this research and examines it from the mirror image: what if instead of studying the images created while experiencing music, I pre-empted this and translated pre-constructed images into music. In order to create a framework around this translation, I draw from visual art theory and pedagogy, namely the concept of The Visual Elements from Otto G. Ocvirk, et. al’s Art Fundamentals: Theory and Practice, Twelfth Edition (2012). Through the duration of this project, I created seven original compositions, five of which explore the translation of visual elements. In this thesis, I first present relevant literature, including quantitative research around cross-modal perception and later around the visual elements and their musical translations. Then, I discuss my compositional portfolio, methodological and analytical approaches. Finally, I will present my analysis, demonstrating how I translated the different visual elements in my body of work. This analysis is segmented based on the different elements, line, shape, form, space, colour, value, and texture. I used the idea of translating visual elements as a concept to broaden my composition technique and content inspiration

Computational Modeling and Analysis of Multi-timbral Musical Instrument Mixtures

In the audio domain, the disciplines of signal processing, machine learning, psychoacoustics, information theory and library science have merged into the field of Music Information Retrieval (Music-IR). Music-IR researchers attempt to extract high level information from music like pitch, meter, genre, rhythm and timbre directly from audio signals as well as semantic meta-data over a wide variety of sources. This information is then used to organize and process data for large scale retrieval and novel interfaces. For creating musical content, access to hardware and software tools for producing music has become commonplace in the digital landscape. While the means to produce music have become widely available, significant time must be invested to attain professional results. Mixing multi-channel audio requires techniques and training far beyond the knowledge of the average music software user. As a result, there is significant growth and development in intelligent signal processing for audio, an emergent field combining audio signal processing and machine learning for producing music. This work focuses on methods for modeling and analyzing multi-timbral musical instrument mixtures and performing automated processing techniques to improve audio quality based on quantitative and qualitative measures. The main contributions of the work involve training models to predict mixing parameters for multi-channel audio sources and developing new methods to model the component interactions of individual timbres to an overall mixture. Linear dynamical systems (LDS) are shown to be capable of learning the relative contributions of individual instruments to re-create a commercial recording based on acoustic features extracted directly from audio. Variations in the model topology are explored to make it applicable to a more diverse range of input sources and improve performance. An exploration of relevant features for modeling timbre and identifying instruments is performed. Using various basis decomposition techniques, audio examples are reconstructed and analyzed in a perceptual listening test to evaluate their ability to capture salient aspects of timbre. These tests show that a 2-D decomposition is able to capture much more perceptually relevant information with regard to the temporal evolution of the frequency spectrum of a set of audio examples. The results indicate that joint modeling of frequencies and their evolution is essential for capturing higher level concepts in audio that we desire to leverage in automated systems.Ph.D., Electrical Engineering -- Drexel University, 201

Explorations in Timbre

Timbre is the quality of sound that enables us to recognise different instruments. The most successful methods for automatic instrument recognition are performed using information from sound that does not mean anything to the musician and is not directly related to the perception of timbre. Instead we start with an attribute of sound that is known to affect the way we perceive timbre. Two such attributes are the harmonic content of the sound and the “envelope” which describes the way volume rises and falls. This thesis explored several methods for detecting envelopes for the purpose of timbre description. A tool was developed that allows a user to match a synthetic wave to recorded audio by adding and adjusting amplitude and frequency control points. The resulting synthetic wave sounds subjectively close to the real audio. This provides avenues for future psychoacoustic research

Proceedings of the 19th Sound and Music Computing Conference

Proceedings of the 19th Sound and Music Computing Conference - June 5-12, 2022 - Saint-Étienne (France). https://smc22.grame.f