Network bending [1] aims to elicit interesting creative output from generative neural networks by applying various transformations to the activations of groups of network nodes. This paper describes the investigation of how this emerging technique of ‘network bending’ can be used to provide novel creative control over sound synthesis networks based on the Magenta DDSP API [2] and how best to provide access to the resulting sound synthesis neural networks to creative practitioners

McCallum, Louis

Yee-King, Matthew

UAL Research Online

Network Bending Neural VocodersLouis McCallumCreative Computing InstituteUniversity of the Arts, LondonLondon, Peckhaml.mccallum@arts.ac.ukMatthew Yee-KingComputing Dept.Goldsmiths, University of LondonLondon, New Crossm.yee-king@gold.ac.ukAbstractNetwork bending [1] aims to elicit interesting creative output from generativeneural networks by applying various transformations to the activations of groupsof network nodes. This paper describes the investigation of how this emergingtechnique of ‘network bending’ can be used to provide novel creative controlover sound synthesis networks based on the Magenta DDSP API [2] and howbest to provide access to the resulting sound synthesis neural networks to creativepractitioners.1 IntroductionGoogle Magenta’s DDSP API allows for rich audio synthesis to be controlled by deep learningmodels [2]. One of the primary use cases stated by its developers is timbre transfer, where an inputaudio signal is used as a control signal via audio analysis techniques. We are interested in how wemight further control the output of the network beyond those parameters provided at the input, i.e.pitch and amplitude. Broad et al. [1] have taken an approach of eliciting interesting creative outputfrom generative neural networks by applying various transformations to the activations of groups ofnetwork nodes. In this paper, we describe initial explorations with the use of this "network bending"in context of DDSP models, including which transforms to apply, where in the network to apply thesetranformations and the parameterisation of tranformations over time.2 ImplementationCompared to Broad et al.’s work [1], both the output domain (music, as opposed to visual art) and thestructure of the models (Autoencoder with recurrent layers as opposed to GAN with convolutionallayers) are different. This means it is unlikely that the techniques used by Broad et al. will be ableto be transferred verbatim. Whilst the activations of a group of units in StyleGAN2 can be seen asrepresentations of visual features in the training set, the output matrices from layers in the DDSPmodel represent the activations of units in one dimension, and a sequence of activations over time inanother dimension. This means transformations such as rotations and reflections applied to the wholeactivation matrix (such as those used by Broad et. al) are in effect smearing time information acrossthe 4 second block that will be crucial to reconstruction, especially with the recurrent layers. As suchthey are unlikely to result in any satisfactory or controllable sonic outcomes.We instead look to apply transforms that preserve some information in the time domain whilst stimu-lating interesting timbral outcomes. We have developed a toolkit to apply any stacked combinationof ablation, invertion, translation in the unit axis (wrap around) and oscillation layers after the firstfully connected layer, after the GRU layer or after the second fully connected layer (see Fig. 1).The composer is able to pick a proportion of units to apply these to (selected randomly), with thetranslation and oscillation layers having parameters that can themselves be altered over time witheither linear ramps or low frequency oscillators (LFOs).4th Workshop on Machine Learning for Creativity and Design at NeurIPS 2020, Vancouver, Canada.Figure 1: a) How the transforms are applied b) Table of transforms and parameters c) Top left,clockwise (1) Clean (2) FC1 Oscillation (ramp on freq and depth, 100% units) (3) FC2 oscillation(LFO on freq and depth, 70% units) (4) FC1 Ablation (82% units)3 ResultsFig. 1 demonstrates the spectral output (see here for audio examples 1) of several uses of networkbending using a model trained on approximately 12 minutes of Whitney Houston acapella tracks.A short excerpt from a different vocal piece was used as input. Our early explorations have shownthat altering activations along the time axis, either through oscillating the activations themselves oraltering parameters of the transformation layers over time, results in audible changes to the modeloutput in a recognisable manner. We also see that adjusting the number of units the transforms areapplied to alters the magnitude of the effects.Primarily, we demonstrate how adding an oscillating signal with a linear ramp on the frequency tothe activations of the first dense layer produces clear alterations to the audio output in the time andfrequency domains in line with the increasing frequency of the control signal. This is also the case inthe second dense layer, although we use a significantly lower depth of LFO ( 0.2 as opposed to 2.5)to avoid distorted, noisy whiteouts in the output audio. Finally, we show how ablating (zeroing) aproportion of units in the first dense layer creates a vibrato effect on the audio, with the influence ofthis effect proportional to the number of units transformed.By exposing controllable parameters that are already in the electronic musician’s toolkit, for example,LFO frequency and depth and parameter envelopes, we have been able to provide a workflow andinterface that a computer musician will recognise from experimenting with and chaining togetherexisting digital audio effect units. However, with potentially more exciting, non-standard timbraloutcomes.4 Future WorkWhilst at an early stage, our exploratory work opens up avenues in audio performance with DDSPmodels which may also be applicable to other models. This being said, even within the audio domain,different architechtures will generate different responses and it would be enlightening to combine ourapproaches with, for example, work by Collins et al. [3] that augments activations in spectral LSTMmodels. Moreover, we are currently using a fairly blunt, non-deterministic method of selecting setsof units and clearly, improved results will come from a more developed selection process. There aretechniques for network interpretability that will prove useful in clustering units in this regard andwork in the space of music generation has already been attempted by Brink [4].1http://louismccallum.com/network-bending-audio-examples25 Ethical ConsiderationsIn our research, we take the position that AI and ML technology should be used to enhance andenrich human creativity and experiences. Our aim is not to replace the activity of human creativeswith AI/ML systems. We would rather work towards AI tools that might be seen as supportivecollaborators. Therefore, a lot of our work revolves around taking existing AI/ML systems andadapting them so that they can be accessed and manipulated by people who are not ML experts, butwho do have creative domain expertise.With specific regards to network bending, beyond displeasing machine learning devotees who maybaulk at the subversion of a carefully optimised model, Collins et al. [3] evoke the spectre ofbrainwashing when rewiring the brains of AIs. They do however note this will not be a considerationuntil much more sophisticiated models exist, meaning our current interventions for creative goals areprobably acceptable.Acknowledgments and Disclosure of FundingThis work was supported by the Arts and Humanities Research Council funded project MusicallyIntelligent Machines Interacting Creatively (MIMIC) (mimicproject.com) and UKRI Human DataInteraction Network in collaboration with composer Max de Wardener.References[1] Broad, T., Leymarie, F.F. & Grierson, M. (2020) Network Bending: Manipulating The Inner Representationsof Deep Generative Models, arXiv:2005.12420 [cs.Cv].[2] Engel, J., Lamtharn, H., Gu, C. & Roberts, A. (2020) DDSP: Differentiable Digital Signal Processing, inProc. International Conference on Learning Representations 2020[3] Collins, N., Ruzicka, V., & Grierson, M. (2020) Remixing AIs: mind swaps, hybrainity, and splicing musicalmodels, in Proc. 2020 Joint Conference on AI Music Creativity[4] Brink, P. (2019) Dissection of a Generative Network for Music Composition, Masters Thesis, School ofElectrical Engineering and Computer Science3

Network Bending Neural Vocoders

https://ualresearchonline.arts.ac.uk/id/eprint/20634/2/Network_Bending_DDSP_models-3.pdf

Network Bending Neural Vocoders

Abstract

Similar works

Full text

Available Versions

UAL Research Online