The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

Sensory Substitution is a promising technique for mitigating the loss of a sensory modality. Sensory Substitution Devices (SSDs) work by converting information from the impaired sense (e.g. vision) into another, intact sense (e.g. audition). However, there are a potentially infinite number of ways of converting images into sounds and it is important that the conversion takes into account the limits of human perception and other user-related factors (e.g. whether the sounds are pleasant to listen to). The device explored here is termed “polyglot” because it generates a very large set of solutions. Specifically, we adapt a procedure that has been in widespread use in the design of technology but has rarely been used as a tool to explore perception – namely Interactive Genetic Algorithms. In this procedure, a very large range of potential sensory substitution devices can be explored by creating a set of ‘genes’ with different allelic variants (e.g. different ways of translating luminance into loudness). The most successful devices are then ‘bred’ together and we statistically explore the characteristics of the selected-for traits after multiple generations. The aim of the present study is to produce design guidelines for a better SSD. In three experiments we vary the way that the fitness of the device is computed: by asking the user to rate the auditory aesthetics of different devices (Experiment 1), by measuring the ability of participants to match sounds to images (Experiment 2) and the ability to perceptually discriminate between two sounds derived from similar images (Experiment 3). In each case the traits selected for by the genetic algorithm represent the ideal SSD for that task. Taken together, these traits can guide the design of a better SSD

Sound for Fantasy and Freedom

Sound is an integral part of our everyday lives. Sound tells us about physical events in the environ- ment, and we use our voices to share ideas and emotions through sound. When navigating the world on a day-to-day basis, most of us use a balanced mix of stimuli from our eyes, ears and other senses to get along. We do this totally naturally and without effort. In the design of computer game experiences, traditionally, most attention has been given to vision rather than the balanced mix of stimuli from our eyes, ears and other senses most of us use to navigate the world on a day to day basis. The risk is that this emphasis neglects types of interaction with the game needed to create an immersive experience. This chapter summarizes the relationship between sound properties, GameFlow and immersive experience and discusses two projects in which Interactive Institute, Sonic Studio has balanced perceptual stimuli and game mechanics to inspire and create new game concepts that liberate users and their imagination

Toward a synthetic acoustic ecology: sonically situated, evolutionary agent based models of the acoustic niche hypothesis

We introduce the idea of Synthetic Acoustic Ecology (SAC) as a vehicle for transdisciplinary investigation to develop methods and address open theoretical, applied and aesthetic questions in scientific and artistic disciplines of acoustic ecology. Ecoacoustics is an emerging science that investigates and interprets the ecological role of sound. It draws conceptually from, and is reinvigorating the related arts-humanities disciplines historically associated with acoustic ecology, which are concerned with sonically-mediated relationships between human beings and their environments. Both study the acoustic environment, or soundscape, as the literal and conceptual site of interaction of human and non-human organisms. However, no coherent theories exist to frame the ecological role of the soundscape, or to elucidate the evolutionary processes through which it is structured. Similarly there is a lack of appropriate computational methods to analyse the macro soundscape which hampers application in conservation. We propose that a sonically situated flavour of Alife evolutionary agent-based model could build a productive bridge between the art, science and technologies of acoustic ecological investigations to the benefit of all. As a first step, two simple models of the acoustic niche hypothesis are presented which are shown to exhibit emergence of complex spectro-temporal soundscape structures and adaptation to and recovery from noise pollution events. We discuss the potential of SAC as a lingua franca between empirical and theoretical ecoacoustics, and wider transdisciplinary research in ecoacoustic ecology

Soundscape Generation Using Web Audio Archives

Os grandes e crescentes acervos de áudio na web têm transformado a prática do design de som. Neste contexto, sampling -- uma ferramenta essencial do design de som -- mudou de gravações mecânicas para os domínios da cópia e reprodução no computador. A navegação eficaz nos grandes acervos e a recuperação de conteúdo tornaram-se um problema bem identificado em Music Information Retrieval, nomeadamente através da adoção de metodologias baseadas no conteúdo do áudio.Apesar da sua robustez e eficácia, as soluções tecnológicas atuais assentam principalmente em métodos (estatísticos) de processamento de sinal, cuja terminologia atinge um nível de adequação centrada no utilizador.Esta dissertação avança uma nova estratégia orientada semanticamente para navegação e recuperação de conteúdo de áudio, em particular, sons ambientais, a partir de grandes acervos de áudio na web. Por fim, pretendemos simplificar a extração de pedidos definidos pelo utilizador para promover uma geração fluida de paisagens sonoras. No nosso trabalho, os pedidos aos acervos de áudio na web são feitos por dimensões afetivas que se relacionam com estados emocionais (exemplo: baixa ativação e baixa valência) e descrições semânticas das fontes de áudio (exemplo: chuva). Para tal, mapeamos as anotações humanas das dimensões afetivas para descrições espectrais de áudio extraídas do conteúdo do sinal. A extração de novos sons dos acervos da web é feita estipulando um pedido que combina um ponto num plano afetivo bidimensional e tags semânticas. A aplicação protótipo, MScaper, implementa o método no ambiente Ableton Live. A avaliação da nossa pesquisa avaliou a confiabilidade perceptual dos descritores espectrais de áudio na captura de dimensões afetivas e a usabilidade da MScaper. Os resultados mostram que as características espectrais do áudio capturam significativamente as dimensões afetivas e que o MScaper foi entendido pelos os utilizadores experientes como tendo excelente usabilidade.The large and growing archives of audio content on the web have been transforming the sound design practice. In this context, sampling -- a fundamental sound design tool -- has shifted from mechanical recording to the realms of the copying and cutting on the computer. To effectively browse these large archives and retrieve content became a well-identified problem in Music Information Retrieval, namely through the adoption of audio content-based methodologies. Despite its robustness and effectiveness, current technological solutions rely mostly on (statistical) signal processing methods, whose terminology do attain a level of user-centered explanatory adequacy.This dissertation advances a novel semantically-oriented strategy for browsing and retrieving audio content, in particular, environmental sounds, from large web audio archives. Ultimately, we aim to streamline the retrieval of user-defined queries to foster a fluid generation of soundscapes. In our work, querying web audio archives is done by affective dimensions that relate to emotional states (e.g., low arousal and low valence) and semantic audio source descriptions (e.g., rain). To this end, we map human annotations of affective dimensions to spectral audio-content descriptions extracted from the signal content. Retrieving new sounds from web archives is then made by specifying a query which combines a point in a 2-dimensional affective plane and semantic tags. A prototype application, MScaper, implements the method in the Ableton Live environment. An evaluation of our research assesses the perceptual soundness of the spectral audio-content descriptors in capturing affective dimensions and the usability of MScaper. The results show that spectral audio features significantly capture affective dimensions and that MScaper has been perceived by expert-users as having excellent usability

Malay sound arts::Reimagining biophony and geophony materials. Commentary of original composition portfolio 2019-2023

This PhD takes the research theme of Nada Bumi or Voice of the Earth: exploring andaccentuating hidden Malaysia biophonic and geophonic materials, for expressing self-cultural identity and narrative through sound arts practice. The portfolio and accompanying commentary present eleven sound-art works ranging from instrumental electroacoustic music to Web-Audio API based sound installation. The main idea for this portfolio research is to explore the association of folklore, tales, myths, legends and art cultural narrative of the Malay race and the ancestors of Malay (proto-Malay), with the selected hidden and unheard Malaysia natural soundscape, in producing new sound art works. Therefore, I proposed two major compositional themes each comprising several works; Miroirs of Malay Rebab (MiMaR), and Seed of Life (SoL). The works in Miroirs of Malay Rebab reimagine selected unheard biophonic and geophonic materials as mirrors of several Malay performing art-cultural narrative and their stories, such as Makyung theater dance, Malay Gamelan music dance, Ulek Mayang dance and their stories that I have been exposed to during my undergraduate music studies in Malaysia. The works in Seed of Life (SoL) take a similar approach but focused more on local Malay and proto-Malay folklores, tales, legends and myths associated with my childhood experience. Furthermore, as I delved into the conceptual and compositional aspects of creating the Miroirs of Malay Rebab (MiMaR) set and Seed of Life I (SoL) set, I had the privilege to engage in an enriching journey of (self-) exploration through the creation of sound art within the vibrant Bristol soundscape with support from the local sound art community. This experience was part of my involvement in the Hidden Bristol Soundwalks project, which provided a unique platform for my creative endeavors. I have decided to include this project in this portfolio, which has similar compositional approach with Seed of Life (SoL). Both the major cycles, Miroirs of Malay Rebab (MiMaR) and Seed of Life (SoL), include Western classical music instrumentations with electronics; fixed media; and interactive media. This portfolio was composed and developed at the Studio One, Department of Music in the Faculty of Arts, University of Bristol; the Bristol Interaction Group (B.I.G.) Lab in the Faculty of Engineering, University of Bristol; and my home studios in Clevedon, UK, during the period of October 2019 until September 2022. The portfolio consists of scores, studio-audio production recordings, and several live performance recordings. The commentary comprises a set of philosophical considerations about my compositions and intent for creation based on the Nada Bumi theme and sub themes. Further chapters are dedicated to compositional techniques, related traditions and piece-specific documentations. The portfolio is supplied as a set of digital media, containing pdf files of musical scores in notation, associated software or media components of the works, recordings of the studio-based music, and recordings of several live public performances made in mid-2022 after the period of covid-19 lock-downs

AI Methods in Algorithmic Composition: A Comprehensive Survey

Algorithmic composition is the partial or total automation of the process of music composition by using computers. Since the 1950s, different computational techniques related to Artificial Intelligence have been used for algorithmic composition, including grammatical representations, probabilistic methods, neural networks, symbolic rule-based systems, constraint programming and evolutionary algorithms. This survey aims to be a comprehensive account of research on algorithmic composition, presenting a thorough view of the field for researchers in Artificial Intelligence.This study was partially supported by a grant for the MELOMICS project (IPT-300000-2010-010) from the Spanish Ministerio de Ciencia e Innovación, and a grant for the CAUCE project (TSI-090302-2011-8) from the Spanish Ministerio de Industria, Turismo y Comercio. The first author was supported by a grant for the GENEX project (P09-TIC- 5123) from the Consejería de Innovación y Ciencia de Andalucía