Deep Learning Techniques for Music Generation -- A Survey

This paper is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. We propose a methodology based on five dimensions for our analysis: Objective - What musical content is to be generated? Examples are: melody, polyphony, accompaniment or counterpoint. - For what destination and for what use? To be performed by a human(s) (in the case of a musical score), or by a machine (in the case of an audio file). Representation - What are the concepts to be manipulated? Examples are: waveform, spectrogram, note, chord, meter and beat. - What format is to be used? Examples are: MIDI, piano roll or text. - How will the representation be encoded? Examples are: scalar, one-hot or many-hot. Architecture - What type(s) of deep neural network is (are) to be used? Examples are: feedforward network, recurrent network, autoencoder or generative adversarial networks. Challenge - What are the limitations and open challenges? Examples are: variability, interactivity and creativity. Strategy - How do we model and control the process of generation? Examples are: single-step feedforward, iterative feedforward, sampling or input manipulation. For each dimension, we conduct a comparative analysis of various models and techniques and we propose some tentative multidimensional typology. This typology is bottom-up, based on the analysis of many existing deep-learning based systems for music generation selected from the relevant literature. These systems are described and are used to exemplify the various choices of objective, representation, architecture, challenge and strategy. The last section includes some discussion and some prospects.Comment: 209 pages. This paper is a simplified version of the book: J.-P. Briot, G. Hadjeres and F.-D. Pachet, Deep Learning Techniques for Music Generation, Computational Synthesis and Creative Systems, Springer, 201

Analysis and resynthesis of polyphonic music

This thesis examines applications of Digital Signal Processing to the analysis, transformation, and resynthesis of musical audio. First I give an overview of the human perception of music. I then examine in detail the requirements for a system that can analyse, transcribe, process, and resynthesise monaural polyphonic music. I then describe and compare the possible hardware and software platforms. After this I describe a prototype hybrid system that attempts to carry out these tasks using a method based on additive synthesis. Next I present results from its application to a variety of musical examples, and critically assess its performance and limitations. I then address these issues in the design of a second system based on Gabor wavelets. I conclude by summarising the research and outlining suggestions for future developments

Composer-Centered Computer-Aided Soundtrack Composition

For as long as computers have been around, people have looked for ways to involve them in music. Research in computer music progresses in many varied areas: algorithmic composition, music representation, music synthesis, and performance analysis to name a few. However, computer music research, especially relating to music composition, does very little toward making the computer useful for artists in practical situations. This lack of consideration for the user has led to the containment of computer music, with a few exceptions, to academia. In this thesis, I propose a system that enables a computer to aide users composing music in a specific setting: soundtracks. In the process of composing a soundtrack, a composer is faced with solving non-musical problems that are beyond the experience of composers of standalone music. The system I propose utilizes the processing power of computers to address the non-musical problems thus preventing users from having to deal with them. Therefore, users can focus on the creative aspect of composing soundtrack music. The guiding principal of the system is to help the composer while not assuming any creative power and while leaving the user in full control of the music. This principal is a major step toward helping users solve problems while not introducing new ones. I present some carefully chosen tasks that a computer can perform with guidance from the user that follow this principal. For example, the system performs calculations to help users compose music that matches the visual presentation and allows users to specify music, using the idea of timed regular expressions, so that a computer can fill arbitrary amounts of time with music in a controlled manner. A prototype application, called EMuse, was designed and implemented to illustrate the use and benefits of the proposed system. To demonstrate that the system is capable of serving as a tool to create music, two soundtracks were created for two sample animations. It is beyond the scope of the work presented here to evaluate if the system achieves the goal of being a practical tool for composers. However, the innovations herein discussed are analyzed and found to be useful for soundtrack composition and for future user-centered computer-music research

Proceedings of the 7th Sound and Music Computing Conference

Proceedings of the SMC2010 - 7th Sound and Music Computing Conference, July 21st - July 24th 2010

Data Musicalization

Data musicalization is the process of automatically composing music based on given data, as an approach to perceptualizing information artistically. The aim of data musicalization is to evoke subjective experiences in relation to the information, rather than merely to convey unemotional information objectively. This paper is written as a tutorial for readers interested in data musicalization. We start by providing a systematic characterization of musicalization approaches, based on their inputs, methods and outputs. We then illustrate data musicalization techniques with examples from several applications: one that perceptualizes physical sleep data as music, several that artistically compose music inspired by the sleep data, one that musicalizes on-line chat conversations to provide a perceptualization of liveliness of a discussion, and one that uses musicalization in a game-like mobile application that allows its users to produce music. We additionally provide a number of electronic samples of music produced by the different musicalization applications.Peer reviewe

Music Encoding Conference Proceedings 2021, 19–22 July, 2021 University of Alicante (Spain): Onsite & Online

Este documento incluye los artículos y pósters presentados en el Music Encoding Conference 2021 realizado en Alicante entre el 19 y el 22 de julio de 2022.Funded by project Multiscore, MCIN/AEI/10.13039/50110001103

XR, music and neurodiversity: design and application of new mixed reality technologies that facilitate musical intervention for children with autism spectrum conditions

This thesis, accompanied by the practice outputs,investigates sensory integration, social interaction and creativity through a newly developed VR-musical interface designed exclusively for children with a high-functioning autism spectrum condition (ASC).The results aim to contribute to the limited expanse of literature and research surrounding Virtual Reality (VR) musical interventions and Immersive Virtual Environments (IVEs) designed to support individuals with neurodevelopmental conditions. The author has developed bespoke hardware, software and a new methodology to conduct field investigations. These outputs include a Virtual Immersive Musical Reality Intervention (ViMRI) protocol, a Supplemental Personalised, immersive Musical Experience(SPiME) programme, the Assisted Real-time Three-dimensional Immersive Musical Intervention System’ (ARTIMIS) and a bespoke (and fully configurable) ‘Creative immersive interactive Musical Software’ application (CiiMS). The outputs are each implemented within a series of institutional investigations of 18 autistic child participants. Four groups are evaluated using newly developed virtual assessment and scoring mechanisms devised exclusively from long-established rating scales. Key quantitative indicators from the datasets demonstrate consistent findings and significant improvements for individual preferences (likes), fear reduction efficacy, and social interaction. Six individual case studies present positive qualitative results demonstrating improved decision-making and sensorimotor processing. The preliminary research trials further indicate that using this virtual-reality music technology system and newly developed protocols produces notable improvements for participants with an ASC. More significantly, there is evidence that the supplemental technology facilitates a reduction in psychological anxiety and improvements in dexterity. The virtual music composition and improvisation system presented here require further extensive testing in different spheres for proof of concept

Music information retrieval: conceptuel framework, annotation and user behaviour

Understanding music is a process both based on and influenced by the knowledge and experience of the listener. Although content-based music retrieval has been given increasing attention in recent years, much of the research still focuses on bottom-up retrieval techniques. In order to make a music information retrieval system appealing and useful to the user, more effort should be spent on constructing systems that both operate directly on the encoding of the physical energy of music and are flexible with respect to users’ experiences. This thesis is based on a user-centred approach, taking into account the mutual relationship between music as an acoustic phenomenon and as an expressive phenomenon. The issues it addresses are: the lack of a conceptual framework, the shortage of annotated musical audio databases, the lack of understanding of the behaviour of system users and shortage of user-dependent knowledge with respect to high-level features of music. In the theoretical part of this thesis, a conceptual framework for content-based music information retrieval is defined. The proposed conceptual framework - the first of its kind - is conceived as a coordinating structure between the automatic description of low-level music content, and the description of high-level content by the system users. A general framework for the manual annotation of musical audio is outlined as well. A new methodology for the manual annotation of musical audio is introduced and tested in case studies. The results from these studies show that manually annotated music files can be of great help in the development of accurate analysis tools for music information retrieval. Empirical investigation is the foundation on which the aforementioned theoretical framework is built. Two elaborate studies involving different experimental issues are presented. In the first study, elements of signification related to spontaneous user behaviour are clarified. In the second study, a global profile of music information retrieval system users is given and their description of high-level content is discussed. This study has uncovered relationships between the users’ demographical background and their perception of expressive and structural features of music. Such a multi-level approach is exceptional as it included a large sample of the population of real users of interactive music systems. Tests have shown that the findings of this study are representative of the targeted population. Finally, the multi-purpose material provided by the theoretical background and the results from empirical investigations are put into practice in three music information retrieval applications: a prototype of a user interface based on a taxonomy, an annotated database of experimental findings and a prototype semantic user recommender system. Results are presented and discussed for all methods used. They show that, if reliably generated, the use of knowledge on users can significantly improve the quality of music content analysis. This thesis demonstrates that an informed knowledge of human approaches to music information retrieval provides valuable insights, which may be of particular assistance in the development of user-friendly, content-based access to digital music collections

The Seaboard: discreteness and continuity in musical interface design

The production of acoustic music bridges two senses—touch and hearing—by connecting physical movements, gestures, and tactile interactions with the creation of sound. Mastery of acoustic music depends on the development and refinement of muscle memory and ear training in concert. This process leads to a capacity for great depth of expression even though the actual timbral palette of each given acoustic instrument is relatively limited. By contrast, modern modes of music creation involving recorded music and digital sound manipulation sacrifice this immediate bridge and substitute more abstract processes that enable sonic possibilities extending far beyond the acoustic palette. Mastery in abstract approaches to music making doesn’t necessarily rely on muscle memory or ear training, as many key processes do not need to happen in realtime. This freedom from the limits of time and practiced physical manipulation radically increases the range of achievable sounds, rhythms and effects, but sometimes results in a loss of subtlety of expressiveness. This practice-based PhD asks whether it is possible, and if so how, to achieve an integration of relevant sensor technologies, design concepts, and formation techniques to create a new kind of musical instrument and sound creation tool that bridges this gap with a satisfying result for musicians and composers. In other words, can one create new, multi-dimensional interfaces which provide more effective ways to control the expressive capabilities of digital music creation in real-time? In particular, can one build on the intuitive, logical, and well-known layout of the piano keyboard to create a new instrument that more fully enables both continuous and discrete approaches to music making? My research practice proposes a new musical instrument called the Seaboard, documents its invention, development, design, and refinement, and evaluates the extent to which it positively answers the above question. The Seaboard is a reinterpretation of the piano keyboard as a soft, continuous wavelike surface that places polyphonic pitch bend, vibrato and continuous touch right at the musician’s fingertips. The addition of new realtime parameters to a familiar layout means it combines the intuitiveness of the traditional instrument with some of the versatility of digital technology. Designing and prototyping the Seaboard to the point of successfully proving that a new synthesis between acoustic techniques and digital technologies is possible is shown to require significant coordination and integration of a range of technical disciplines. The research approach has been to build and refine a series of prototypes that successively grapple with the integration of these elements, whilst rigorously documenting the design issues, engineering challenges, and ultimate decisions that determine whether an intervention in the field of musical instrumentation is fruitful