MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative adversarial networks (GANs). The three models, which differ in the underlying assumptions and accordingly the network architectures, are referred to as the jamming model, the composer model and the hybrid model. We trained the proposed models on a dataset of over one hundred thousand bars of rock music and applied them to generate piano-rolls of five tracks: bass, drums, guitar, piano and strings. A few intra-track and inter-track objective metrics are also proposed to evaluate the generative results, in addition to a subjective user study. We show that our models can generate coherent music of four bars right from scratch (i.e. without human inputs). We also extend our models to human-AI cooperative music generation: given a specific track composed by human, we can generate four additional tracks to accompany it. All code, the dataset and the rendered audio samples are available at https://salu133445.github.io/musegan/ .Comment: to appear at AAAI 201

Deep Learning Techniques for Music Generation -- A Survey

This paper is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. We propose a methodology based on five dimensions for our analysis: Objective - What musical content is to be generated? Examples are: melody, polyphony, accompaniment or counterpoint. - For what destination and for what use? To be performed by a human(s) (in the case of a musical score), or by a machine (in the case of an audio file). Representation - What are the concepts to be manipulated? Examples are: waveform, spectrogram, note, chord, meter and beat. - What format is to be used? Examples are: MIDI, piano roll or text. - How will the representation be encoded? Examples are: scalar, one-hot or many-hot. Architecture - What type(s) of deep neural network is (are) to be used? Examples are: feedforward network, recurrent network, autoencoder or generative adversarial networks. Challenge - What are the limitations and open challenges? Examples are: variability, interactivity and creativity. Strategy - How do we model and control the process of generation? Examples are: single-step feedforward, iterative feedforward, sampling or input manipulation. For each dimension, we conduct a comparative analysis of various models and techniques and we propose some tentative multidimensional typology. This typology is bottom-up, based on the analysis of many existing deep-learning based systems for music generation selected from the relevant literature. These systems are described and are used to exemplify the various choices of objective, representation, architecture, challenge and strategy. The last section includes some discussion and some prospects.Comment: 209 pages. This paper is a simplified version of the book: J.-P. Briot, G. Hadjeres and F.-D. Pachet, Deep Learning Techniques for Music Generation, Computational Synthesis and Creative Systems, Springer, 201

Teaching Construction in the Virtual University: the WINDS project

This paper introduces some of the Information Technology solutions adopted in Web based INtelligent Design Support (WINDS) to support education in A/E/C design. The WINDS project WINDS is an EC-funded project in the 5th Framework, Information Society Technologies programme, Flexible University key action. WINDS is divided into two actions: ·The research technology action is going to implement a learning environment integrating an intelligent tutoring system, a computer instruction management system and a set of co-operative supporting tools. ·The development action is going to build a large knowledge base supporting Architecture and Civil Engineering Design Courses and to experiment a comprehensive Virtual School of Architecture and Engineering Design. During the third year of the project, more than 400 students all over Europe will attend the Virtual School. During the next three years the WINDS project will span a total effort of about 150 man-years from 28 partners of 10 European countries. The missions of the WINDS project are: Advanced Methodologies in Design Education. WINDS drives a breakdown with conventional models in design education, i.e. classroom or distance education. WINDS implements a problem oriented knowledge transfer methodology following Roger Schank's Goal Based Scenario (GBS) pedagogical methodology. GBS encourages the learning of both skills and cases, and fosters creative problem solving. Multidisciplinary Design Education. Design requires creative synthesis and open-end problem definition at the intersection of several disciplines. WINDS experiments a valuable integration of multidisciplinary design knowledge and expertise to produce a high level standard of education. Innovative Representation, Delivery and Access to Construction Education. WINDS delivers individual education customisation by allowing the learner access through the Internet to a wide range of on-line courses and structured learning objects by means of personally tailored learning strategies. WINDS promotes the 3W paradigm: learn What you need, Where you want, When you require. Construction Practice. Construction industry is a repository of ""best practices"" and knowledge that the WINDS will profit. WINDS system benefits the ISO10303 and IFC standards to acquire knowledge of the construction process directly in digital format. On the other hand, WINDS reengineers the knowledge in up-to-date courses, educational services, which the industries can use to provide just-in-time rather than in-advance learning. WINDS IT Solutions The missions of the WINDS project state many challenging requirements both in knowledge and system architecture. Many of the solutions adopted in these fields are innovative; others are evolution of existing technologies. This paper focuses on the integration of this set of state-of-the-art technologies in an advanced and functionally sound Computer Aided Instruction system for A/E/C Design. In particular the paper deals with the following aspects: Standard Learning Technology Architecture The WINDS system relies on the in progress IEEE 1484.1 Learning Technology Standard Architecture. According to this standard the system consists of two data stores, the Knowledge Library and the Record Database, and four process: System Coach, Delivery, Evaluation and the Learner. WINDS implements the Knowledge Library into a three-tier architecture: 1.Learning Objects: ·Learning Units are collections of text and multimedia data. ·Models are represented in either IFC or STEP formats. ·Cases are sets of Learning Units and Models. Cases are noteworthy stories, which describes solutions, integrate technical detail, contain relevant design failures etc. 2.Indexes refer to the process in which the identification of relevant topics in design cases and learning units takes place. Indexing process creates structures of Learning Objects for course management, profile planning procedures and reasoning processes. 3.Courses are taxonomies of either Learning Units or a design task and Course Units. Knowledge Representation WINDS demonstrates that it is possible and valuable to integrate a widespread design expertise so that it can be effectively used to produce a high level standard of education. To this aim WINDS gathers area knowledge, design skills and expertise under the umbrellas of common knowledge representation structures and unambiguous semantics. Cases are one of the most valuable means for the representation of design expertise. A Case is a set of Learning Units and Product Models. Cases are noteworthy stories, which describe solutions, integrate technical details, contain relevant design failures, etc. Knowledge Integration Indexes are a medium among different kind of knowledge: they implement networks for navigation and access to disparate documents: HTML, video, images, CAD and product models (STEP or IFC). Concept indexes link learning topics to learning objects and group them into competencies. Index relationships are the base of the WINDS reasoning processes, and provide the foundation for system coaching functions, which proactively suggest strategies, solutions, examples and avoids students' design deadlock. Knowledge Distribution To support the data stores and the process among the partners in 10 countries efficiently, WINDS implements an object oriented client/server as COM objects. Behind the DCOM components there is the Dynamic Kernel, which dynamically embodies and maintains data stores and process. Components of the Knowledge Library can reside on several servers across the Internet. This provides for distributed transactions, e.g. a change in one Learning Object affects the Knowledge Library spread across several servers in different countries. Learning objects implemented as COM objects can wrap ownership data. Clear and univocal definition of ownerships rights enables Universities, in collaboration with telecommunication and publisher companies, to act as "education brokers". Brokerage in education and training is an innovative paradigm to provide just-in-time and personally customised value added learning knowledg

MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer

We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce medleys and create mixtures of entire songs. The interpolations smoothly change pitches, dynamics and instrumentation to create a harmonic bridge between two music pieces. To the best of our knowledge, this work represents the first successful attempt at applying neural style transfer to complete musical compositions.Comment: Paper accepted at the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, Paris, Franc