We introduce a system that allows users of Ableton Live to create MIDI-clips by naming them with musical descriptions. Users can compose by typing the desired musical content directly in Ableton's clip view, which is then inserted by our integrated system. This allows users to stay in the flow of their creative process while quickly generating musical ideas. The system works by prompting ChatGPT to reply using one of several text-based musical formats, such as ABC notation, chord symbols, or drum tablature. This is an important step in integrating generative AI tools into pre-existing musical workflows, and could be valuable for content makers who prefer to express their creative vision through descriptive language. Code is available at https://github.com/supersational/JAMMIN-GPT

Hollowell, Sven

Marshall, Paul

Namgyal, Tashi

English

Explore Bristol Research

                          Hollowell, S., Namgyal, T., & Marshall, P. (2023). Jammin-gpt: Text-based Improvisation using LLMs in Ableton Live. Paper presented at23rd International Society for Music Information Retrieval Conference,Milan, Italy.Peer reviewed versionLicense (if available):CC BYLink to publication record in Explore Bristol ResearchPDF-documentThis is the accepted author manuscript (AAM) of the article which has been made Open Access under theUniversity of Bristol's Scholarly Works Policy. The final published version (Version of Record) can be found onthe publisher's website. The copyright of any third-party content, such as images, remains with the copyrightholder.University of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only thepublished version using the reference above. Full terms of use are available:http://www.bristol.ac.uk/red/research-policy/pure/user-guides/ebr-terms/JAMMIN-GPT: TEXT-BASED IMPROVISATION USING LLMS INABLETON LIVESven Hollowell Tashi Namgyal Paul MarshallDepartment of Computer Science, University of Bristol, Bristol, UKsven.hollowell@bristol.ac.ukABSTRACTWe introduce a system that allows users of Ableton Liveto create MIDI-clips by naming them with musical descrip-tions. Users can compose by typing the desired musicalcontent directly in Ableton’s clip view, which is then in-serted by our integrated system. This allows users to stay inthe flow of their creative process while quickly generatingmusical ideas. The system works by prompting ChatGPTto reply using one of several text-based musical formats,such as ABC notation, chord symbols, or drum tablature.This is an important step in integrating generative AI toolsinto pre-existing musical workflows, and could be valuablefor content makers who prefer to express their creative vi-sion through descriptive language. Code is available at 1 .1. INTRODUCTIONDigital Audio Workstations (DAWs) are the primary toolfor music production. Ableton Live is a popular DAWthat uses a clip-based workflow, where users create andedit musical ideas in short clips that can be launched in agrid (rows are "scenes", columns are "tracks"). Typically,MIDI-clips would be created by drawing notes in a pianoroll editor, or by playing notes on a MIDI keyboard.Music creation can be done in a variety of ways but iswidely seen to be an iterative process [1, 2]. For exam-ple, with a "flare and focus" approach [3] where musicianssuccessively expand upon and refine ideas. This oftenincludes communicating with other musicians who havedifferent roles, interaction styles and levels of grounding[4].According to Sawyer [5] “In group creativity, the per-formance must be constantly negotiated and constructedfrom moment to moment”. However traditional DAWs aretailored for solo creators, lacking the inherent spark of cre-ativity that collaboration with a creative partner allows for.Using a virtual collaborator moves the user into the roleof a producer who might give more general directions, e.g.“play that again but with more energy”. This process is1 https://github.com/supersational/JAMMIN-GPT© S. Hollowell, T. Namgyal, and P. Marshall. Licensedunder a Creative Commons Attribution 4.0 International License (CCBY 4.0). Attribution: S. Hollowell, T. Namgyal, and P. Mar-shall, “JAMMIN-GPT: Text-based Improvisation using LLMs in AbletonLive”, in Extended Abstracts for the Late-Breaking Demo Session of the22nd Int. Society for Music Information Retrieval Conf., Online, 2023.often done in natural language, and so LLMs are a naturalgo-to for emulating this part of the co-creation process.Another important aspect of composition is ’flow’ [6],where musicians enter a state of continuous inspiration.However, it is easier to break this state than to enter itand so AI tools that take users away from creative to ad-ministrative tasks should be avoided. For example, hav-ing to load special environments or wait for long trainingand/or inferences times. One solution is to embed inter-faces within existing workflows, such as DAWs. We there-fore propose JAMMIN-GPT, a natural language interfacefor music generation embedded within a DAW.2. RELATED WORKThere are many examples of AI-based composition toolsbeing imported into DAWs as plugins, such as DrumNet[7], MMM4Live [8] and Magenta Studio [9]. These allowcreation of MIDI clips by using a generative model suchas MusicVAE [10], which can generate novel MIDI clipsor variations of existing clips. The central disadvantage ofthis approach is that the user cannot specify the musicalcontent of the clip, since the clip is generated from a latentspace that is not always interpretable to humans. Anotherdisadvantage is that the interface makes it awkward to se-lect input and output clips, which must be selected fromdropdown menus. Our approach improves on this as it builtdirectly into the clip-view of Ableton Live.Text-conditioned generative models for audio have re-cently enabled users to generate clips of music from a textdescription, such as AudioLM [11], AudioLDM [12] andMusicGen [13]. However, generating directly in the audiodomain makes it difficult for users to tweak model outputscompared to symbolic approaches.Language models have been used to generate symbolicmusic in various ways. Models can be trained solely onmusic data, solely on natural language, or trained on natu-ral language and then fine-tuned on music data. For exam-ple Music Transformer [14] uses an LLM-like architecturetrained specifically for music generation, but is smaller inscale than natural language models so is less expressive.There are many kinds of symbolic music representa-tion, such as MIDI, that are not text-based and so are notpresent in the data used to train LLMs. However, these canbe converted to a text format and used to fine-tune LLMs,for example fine-tuning GPT-2 to piano music [15]. Chat-GPT and GPT-3 models have been used by Tomoki [16] toarXiv:2312.03479v1  [cs.SD]  6 Dec 2023generate code for the TidalCycles live coding environment,by fine-tuning it on example pairs of text-descriptions andcode. GPT-3 has also been used by Zhang et. al. [17] togenerate drum pattern continuations from a starting snip-pet. Zhang et. al. used a similar approach to Tomoki, butused a larger dataset of drum patterns. They demonstratethat the generated patterns are both musically plausible anddiverse, being different from any of the training examples.3. DESIGNAt a high level, our main system operates by reacting tomessages from Ableton Live via the OSC protocol. Whena user creates and names an empty MIDI clip, we use thename of the clip as part of a prompt for ChatGPT (GPT-4-turbo [18]). We get ChatGPT to create MIDI data byprompting it to respond in one of several text-based musi-cal formats, such as ABC notation, chord symbols, or drumtablature. Our system processes ChatGPT’s response, con-verting it back into MIDI, and inserting it into the clip.3.1 ChatGPT AbilitiesChatGPT is able to produce music in a variety of text-basedmusic formats, which were discovered by asking it for a listof formats it knows. This is an important choice for gen-erating high quality output, since ChatGPT will have seenexamples of different styles of music in different formats.For example, ABC music notation typically represents folkmusic, so the generated music will reflect this bias.3.2 Ableton to Python InterfaceAbleton Live supports the use of MIDI remote scripts tocontrol the DAW from an external device. We provide aremote script for the user to install, which provides an OSCinterface from Ableton Live.Using Python, we can control many features of AbletonLive, such as reading or launching clips, or changing theBPM. Our Python script polls Ableton Live for changesin clip names, and sends this information along with rele-vant musical context to the ChatGPT model. We use Able-ton Live’s clip colour feature to indicate that a clip is be-ing generated. When the generation is complete we parsethe MIDI from ChatGPTs response and insert it into theclip, changing color to indicate it is completed. We alsoallow for editing existing clips. If a clip already containingMIDI is renamed, we prompt ChatGPT with the content ofthe original clip and ask it to alter the MIDI based on theprompt. Instrument choice is left to the user. The systemuses the name of the clip’s track as part of its prompt, so ithas information on which instrument is being used.3.3 MIDI ParsingWe determine the output format based on the prompt.One mode uses simple keyword extraction i.e. it useschord-symbols if the prompt contains the word "chord"or "chords". The other method is to prompt ChatGPT tochoose the format by prompting it to first "choose" a for-mat and then generate the output. This is not optimal sinceFigure 1. System overview, from user input to MIDI out-put. The user interaction consists of selecting clips andtyping a description. JAMMIN-GPT automates the re-maining workflow.the model often favours chord-symbol notation, even whenit’s not suitable. For instance, when asked for a "funkybassline" it will choose chord-symbol notation which lacksthe required syntax for rhythm.4. DISCUSSIONOur system has been tested by a small number of Abletonusers. We observed that users were able to quickly be-gin using and prompting the system, since the interface isfamiliar to them. Some users prompted ChatGPT with au-dio based descriptions that are not representable with MIDIsuch as "reverb-y" or "grainy synth sound". These kinds offeatures can not currently be controlled by the model.Musicality of the generated MIDI varies by formatand style. Generally, ChatGPT generates meaningful andprompt-sensitive chord progressions which are of highquality. However, its lack of exposure to styles other thanfolk in the ABC format biases the model towards folkierstyles regardless of the prompt. In future we hope to im-prove its ability to generate ABC notation by convertinga dataset of various styles of music into ABC and fine-tuning the model on it. Since the backend is not fixed touse ChatGPT only, we could also fine-tune a model such asLLaMA [19] or incorporate other models such as Music-VAE [10] to refine the output of ChatGPT. The LLM canalso be swapped or upgraded as future models are released.We note that existing methods for using AI within aDAW are limited in both their UI and the fact that natu-ral language description cannot be used. To this end wedesigned a system that is both intuitive to use, and usesChatGPT so that MIDI clips can be generated from naturallanguage musical descriptions. We see the clip-based text-to-MIDI interface as a modality worth exploring further,since it allows for quick experimentation and iteration.5. ACKNOWLEDGMENTSSven Hollowell and Tashi Namgyal are supported by theUKRI Centre for Doctoral Training in Interactive ArtificialIntelligence (EP/S022937/1).6. REFERENCES[1] C.-Z. A. Huang, H. V. Koops, E. Newton-Rex, M. Din-culescu, and C. J. Cai, “Ai song contest: Human-ai co-creation in songwriting,” in International Society Mu-sic Information Retreival Conference (ISMIR), 2020.[2] J. Garcia, T. Tsandilas, C. Agon, and W. E. Mackay,“Structured observation with polyphony: a multi-faceted tool for studying music composition,” in ACMSIGCHI Conference on Designing Interactive Systems,2014.[3] B. Buxton, Sketching user experiences: getting the de-sign right and the right design. Morgan Kaufmann,2010.[4] N. Bryan-Kinns, B. Banar, C. Ford, C. N. Reed,Y. Zhang, S. Colton, and J. Armitage, “Explor-ing xai for the arts: Explaining latent spacein generative music,” in Workshop on eXplain-able AI Approaches for Debugging and Diagnosis(XAI4Debugging@NeurIPS2021), 2021.[5] R. K. Sawyer, Group creativity: Music, theater, collab-oration. Psychology Press, 2014.[6] M. Csikszentmihalyi, Flow: The Psychology of Opti-mal Experience. Ingram International Inc, 2008.[7] S. Lattner and M. Grachten, “High-level control ofdrum track generation using learned patterns of rhyth-mic interaction,” in IEEE Workshop on Applicationsof Signal Processing to Audio and Acoustics (WASPAA2019), 2019.[8] J. Ens and P. Pasquier, “Mmm : Exploring conditionalmulti-track music generation with the transformer,”arXiv preprint arXiv:2008.06048, 2020.[9] A. Roberts, J. Engel, Y. Mann, J. Gillick, C. Kay-acik, S. Nørly, M. Dinculescu, C. Radebaugh,C. Hawthorne, and D. Eck, “Magenta studio: Aug-menting creativity with deep learning in ableton live,”2019.[10] A. Roberts, J. Engel, C. Raffel, C. Hawthorne, andD. Eck, “A hierarchical latent vector model for learninglong-term structure in music,” in International confer-ence on machine learning. PMLR, 2018, pp. 4364–4373.[11] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov,O. Pietquin, M. Sharifi, D. Roblek, O. Teboul,D. Grangier, M. Tagliasacchi, and N. Zeghidour, “Au-diolm: a language modeling approach to audio genera-tion,” arXiv preprint arXiv:2209.03143, 2023.[12] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic,W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXivpreprint arXiv:2301.12503, 2023.[13] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant,G. Synnaeve, Y. Adi, and A. Défossez, “Simpleand controllable music generation,” arXiv preprintarXiv:2306.05284, 2023.[14] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer,I. Simon, C. Hawthorne, A. M. Dai, M. D. Hoff-man, M. Dinculescu, and D. Eck, “Music transformer:Generating music with long-term structure,” in In-ternational Conference on Learning Representations(ICLR), 2019.[15] B. Banar and S. Colton, “A systematic evaluationof gpt-2-based music generation,” in InternationalConference on Computational Intelligence in Music,Sound, Art and Design (Part of EvoStar). Springer,2022, pp. 19–35.[16] T. Okuda, “Investigation of Live Coding Us-ing a Combination of ChatGPT and Fine-Tuned GPT-3,” AIMC 2023, aug 29 2023,https://aimc2023.pubpub.org/pub/kba1r63j.[17] L. Zhang and C. Callison-Burch, “Language modelsare drummers: Drum composition with natural lan-guage pre-training,” arXiv preprint arXiv:2301.01162,2023.[18] [Online]. Available: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo[19] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A.Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Ham-bro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, andG. Lample, “Llama: Open and efficient foundationlanguage models,” arXiv preprint arXiv:2302.13971,2023.

Jammin-gpt:Text-based Improvisation using LLMs in Ableton Live

https://research-information.bris.ac.uk/files/393139227/2312.03479v1.pdf

Jammin-gpt:Text-based Improvisation using LLMs in Ableton Live

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research