PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the piano rolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between piano rolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two commercial sound libraries. We open our source code at https://github.com/bwang514/PerformanceNetComment: 8 pages, 6 figures, AAAI 2019 camera-ready versio

On the Modeling of Musical Solos as Complex Networks

Notes in a musical piece are building blocks employed in non-random ways to create melodies. It is the "interaction" among a limited amount of notes that allows constructing the variety of musical compositions that have been written in centuries and within different cultures. Networks are a modeling tool that is commonly employed to represent a set of entities interacting in some way. Thus, notes composing a melody can be seen as nodes of a network that are connected whenever these are played in sequence. The outcome of such a process results in a directed graph. By using complex network theory, some main metrics of musical graphs can be measured, which characterize the related musical pieces. In this paper, we define a framework to represent melodies as networks. Then, we provide an analysis on a set of guitar solos performed by main musicians. Results of this study indicate that the presented model can have an impact on audio and multimedia applications such as music classification, identification, e-learning, automatic music generation, multimedia entertainment.Comment: to appear in Information Science, Elsevier. Please cite the paper including such information. arXiv admin note: text overlap with arXiv:1603.0497

Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Illustrations and associative logic. Short circuits between images of science, art and architecture / Illustrazioni e logiche associative. Cortocircuiti tra immagini della scienza, dell'arte e dell'architettura

Le illustrazioni di carattere scientifico non servono solo per veicolare informazioni tra addetti che operano nello stesso ambito all’interno del quale sono concepite e generate. Spesso modelli concettuali espressi attraverso illustrazioni possono migrare tra discipline diverse aprendo possibilità di inaspettati sviluppi. Le immagini, infatti, hanno il potere di comunicare suggestioni diverse, andando dunque oltre al loro compito di raffigurare i contenuti per i quali sono state elaborate. Forme, colori, artifici comunicativi, schemi, notazioni, ideogrammi, figure geometriche, forme diverse di rappresentazione possono “rivelare” e restituire il visibile ma anche “costruire” un’idea del possibile. Le immagini hanno dunque il potere di poter parlare a tutti a prescindere dalla condivisione dei codici figurativi che usano. Anche qualora non si parli lo stesso “linguaggio iconico” e si incorra in fraintendimenti, Hans Georg Gadamer ci ha insegnato che questi non sempre generano conseguenze negative, anzi, possono aprire nuove strade interpretative. Le immagini organizzano, d'altronde, la nostra memoria e il nostro pensiero e sono in grado di far scaturire una molteplicità di associazioni – come ha dimostrato Aby Warburg con la pratica ermeneutica esercitata sulle immagini nel suo Bilderatlas. La proprietà associativa è d’altra parte peculiare dell’immagine e ci consente di cogliere, di ricordare e di rivelare tanto elementi della realtà quanto interi sistemi di ordinamento della nostra memoria e dei nostri modi di interagire con la realtà stessa. Quando operiamo per alterare la realtà percepibile producendo nuove immagini, come avviene con l’arte o l’architettura, è inevitabile che la creatività si alimenti con altre immagini e suggestioni. Il gioco dei rimandi può essere più o meno esplicitato nell’opera compiuta, ma sarà comunque colto in virtù delle proprietà associative che sarà in grado di sollecitare dando vita a sua volta a nuove possibilità di sviluppi figurativi e/o concettuali.Although the main purpose of scientific illustrations is to facilitate the sharing of information between operators working in environments in which they are widely understood, drawings of conceptual models may also be used in other fields, often with the consequence of generating new, unexpected developments. The power of images to communicate ideas other than the contents for which they were originally designed means that shapes, colours, communicational expedients, diagrams, notes, ideographs, geometric figures and different forms of representation not only “show the visible” but also propose a way of “constructing the possible”. Indeed, images have the power to speak to everyone, regardless of their knowledge of the figurative codes employed. Even when those who ‘read’ the images are familiar with different “iconic languages” and there is a strong risk of misunderstanding, Hans Georg Gadamer teaches us that this does not always lead to negative results but, on the contrary, often paves the way for new interpretations. Images organize our memory and thoughts and enable us to generate multitudes of associations, as demonstrated by Aby Warburg’s exploitation of hermeneutics in the composition of Der Bilderatlas: Mnemosyne. However, the ability to stimulate the power of association is a feature peculiar to the image as images enable us to capture, remember and exploit not only elements of reality but also entire systems of ordering our memories and interacting with reality. Inevitably, when artists or architects endeavour to change perceived reality by producing new images, their creativity is fuelled by other images and ideas. Yet, regardless of the degree of explicitness of the references used in the finished work, the references are understood as a result of the power of association. This is the key to generating new figurative and/or conceptual developments