We demonstrate two generative models created by training
a recurrent neural network (RNN) with three hidden
layers of long short-term memory (LSTM) units. This extends
past work in numerous directions, including training
deeper models with nearly 24,000 high-level transcriptions
of folk tunes. We discuss our on-going work

Korshunova, Iryna

Santos, João Felipe

Sturm, Bob

Ghent University Academic Bibliography

FOLKMUSIC STYLE MODELLING BY RECURRENT NEURALNETWORKS WITH LONG SHORT TERMMEMORY UNITSBob L. SturmCentre for Digital Music, QMULb.sturm@qmul.ac.ukJoa˜o Felipe SantosINRS-EMT, Montreal Canadajfsantos@emt.inrs.caIryna KorshunovaGhent University, Belgiumiryna.korshunova@ugent.beABSTRACTWe demonstrate two generative models created by train-ing a recurrent neural network (RNN) with three hiddenlayers of long short-term memory (LSTM) units. This ex-tends past work in numerous directions, including trainingdeeper models with nearly 24,000 high-level transcriptionsof folk tunes. We discuss our on-going work.1. INTRODUCTIONThe application of artificial neural networks to modellingand generating music is well-studied, e.g., [1–7]. Todd etal. [7] train an RNN with one hidden layer of 8-15 units toreproduce melodies of length 34 notes with quantised timesteps. Mozer [6] builds a similar model for melody, butuses an encoding of pitch and time that is pyschoacousti-cally motivated. Eck and Schmidhuber [3] use LSTM tomodel larger structures than possible with an RNN, andtrain them on chord progressions and melodies exhibiting12-bar blues conventions. Eck and Lapamle [2] expandsupon this work to model folk tunes, and uses one-hot en-coded input vectors to represent pitches at uniform timedelays. Franklin [4] uses a coding scheme similar to [6]to build an RNN with LSTM for pitch and duration, butmodelling “jazz-related tasks.”All of the work above entails network architectures thatare shallow (e.g., one hidden layer of a number of units ofthe order 10), and training with a number of music tran-scriptions that are of the orders 10-100, most with fixedlength. We new look at deeper recurrent architectures (3hidden layers of 512 LSTM units each), and training ontens of thousands of textual transcriptions of music.2. DATAWe use data retrieved from The Session, 1 an online com-munity of folk music enthusiasts discussing relevant top-ics, and contributing transcriptions of tunes in ABC for-mat. An example transcription is below:1 https://thesession.org/c© Bob L. Sturm, Joa˜o Felipe Santos, Iryna Korshunova.Licensed under a Creative Commons Attribution 4.0 International Li-cense (CC BY 4.0). Attribution: Bob L. Sturm, Joa˜o Felipe Santos,Iryna Korshunova. “Folk Music Style Modelling by Recurrent NeuralNetworks with Long Short Term Memory Units”, Extended abstracts forthe Late-Breaking Demo Session of the 16th International Society forMusic Information Retrieval Conference, 2015.X: 1T: Scottish Horse, TheZ: WeejieS: https://thesession.org/tunes/12696#setting21449R: jigM: 6/8L: 1/8K: AmixA|c3 ecA|d>ef a2 f|edc cAe|B3 B<BA|c3 ecA|d>ef afd|ecA B2e|A3 A<A:||:e|a2 e A<Ae|d2 f a2 f|e>dc cAe|B3 B<Be/d/|[1 ceA eaA|faA dfA|ceA B2 e|A3 A<A:|[2 c3 ecA|d>ef afd|ecA B2 e|A3 A<A||We retrieve 23,958 tunes from this resource from theirweekly compilations. 2 Eck and Lapamle [2] also use tran-scriptions from this resource in 2008, but only 56 reels con-verted to MIDI format. We create two different datasets:A) removes from the original data all the following ABCfields: X:, Z:, S:, and R:; B) as A, but further removesthe ABC fields T: and L:, removes tunes with differentmeters and/or keys and multiple voices, removes ornamentsand gracenotes, transposes all tunes to have the tonic C,(thus giving four modes: major, mixolydian, dorian, andminor), expresses each transcription as a sequence of to-kens (separating pitches, duration, measure bars), and re-moves entries that are not complete transcriptions but in-stead are comments and suggestions, e.g., alternative end-ings. The number of transcriptions in B is 23,636.3. MODELLING AND GENERATIONWe use two approaches to modelling and generating ABC.The architecture of our models involves 3 hidden layerswith 512 LSTM units each, and a softmax output layergiven the distribution over the vocabulary conditioned onthe one-hot encoded input. We train each model by back-propogation with one-hot encoded vectors, a mini-batchapproach (batch size 50), and with drop out of 0.5. Wetrain our first model using dataset A, with sequences of 50characters. 3 In the above ABC, examples of characters are“M”, “>” and “:”. The size of the vocabulary is 134. Train-ing makes 100 full passes through the dataset. As an exam-ple, a good model should always predict “:” given an in-put of “M” or “K”, but never given “>”. We train our secondmodel using dataset B with a vocabulary of tokens. Exam-ples of tokens are, “d>”, “K:Amix”, and “|[1”, but not“/” or “:”. Unlike the model above, we train this modelusing complete tunes: variable length sequences ranging2 https://github.com/adactio/TheSession-data3 https://github.com/karpathy/char-rnnT: Bornity HorseM: 4/4L: 1/8K: Dmaj|:FG A2 BGBd|AFGE DEdB|AF A2 BGBd|(3cBA cd eAFG|FA B2 ABde|fedB AF F2|BG G2 ABdf|1 afea fdd2:|2 afea ˜d3z|||:d3e fd d2|Bd d2 Adfd|effe dff2|afeg fd d2 |defd ed B2|ABde faaf|a2 fd Bd d2|AFde fdeg||Bornity Horse3 1 244Figure 1. Verbatim output of the model trained on dataset A, and its typeset notation. The model has also titled the work.M:4/4K:Cmaj|: G 2 E > C G > C E > C | G 2 E > G c > G E > C | D 2 D > F (3 D E D [ B, C ] > E | C 2 (3 E C C B > F D > A |G 2 E > C E > C E > C | G 2 E 2 G > C E > C | D 2 d 2 G > B d > B |1 c 4 c 2 (3 G A B :| |2 c 2 B > A C 3 G /2 A /2|: B > c d > e f > d B > c | d > e d > e c > A (3 G A B | c 2 e > c c 4 | e > a (3 e e e c 2 (3 G A B |c 2 c > e f > e d > c | _B 2 B 2 A < B d < c | B > G F > D D > _B B < d |1 c 2 B 2 c > A G < B :| |2 c 2 B 2 c 2 c 2 |3 331 244333 1 2Figure 2. Verbatim output of the model trained on dataset B, and its typeset notation. Tokens are visible.from about 50 to 2000 tokens depending on the tune. 4 Wedo not split the tunes into unrelated parts, and do not mixparts from different tunes. This preserves the structure ofthe data. Training makes 100 passes through the dataset.The size of the vocabulary is 137. (with <s> and </s> tomark the start and the end of each tune).To generate ABC from the first model we merely primeit with input text, e.g., “M: 4/4”, and sample any numberof times from the distribution at the softmax output. Eachgenerated character produces the next input to the model.For the second model, we prime it with a start symbol <s>and sequentially sample tokens from the softmax outputsuntil we encounter the end symbol </s>.4. RESULTS AND FUTUREWORKFigures 1 and 2 show example output of the two models.Both models have learned some of the structural rules ofthe transcriptions in the dataset, as well as stylistic con-ventions: 1) each measure in these examples is correctlycounted (but this is not always the case); 2) a transcriptioncan involve two repeated 8-measure sections; 3) a sectioncan have a second ending that varies the first; 4) a sectioncan consist of 4-measure subsections delimited by a var-4 https://github.com/IraKorshunova/folk-rnnied introduction figure (see m. 1 and 4, 9 and 13 in Fig.1); 5) a section ends on the tonic; 6) the dominant appearsa measure before a section ends. Regardless, these tran-scriptions are not immediately “session-ready” tunes. Ourcurrent work explores refining such models by incorporat-ing corrections made by a domain expert to the output.5. REFERENCES[1] D. Correa, J. Saito, and S. Abib. Composing music with BPTT andLSTM networks: Comparing learning and generalization aspects. InProc. Int. Conf. Comp. Sci. Eng. Workshops, pages 95–100, 2008.[2] D. Eck and J. Lapamle. Learning musical structure directly from se-quences of music. Technical report, University of Montreal, 2008.[3] D. Eck and J. Schmidhuber. Learning the long-term structure of theblues. In Proc. Int. Conf. on Artificial Neural Networks, 2002.[4] Judy A. Franklin. Recurrent neural networks for music computation.ORSA journal on computing, 18(3):321–338, 2006.[5] Niall Griffith and Peter M Todd. Musical networks: Parallel dis-tributed perception and performance. MIT Press, 1999. In Proc. Ar-tificial Intell. Interactive Digital Entertainment Conf., 2014.[6] M. C. Mozer. Neural network composition by prediction: Exploringthe benefits of psychophysical constraints and multiscale processing.Cog. Science, 6:(247-280), 1994.[7] Peter M. Todd. A connectionist approach to algorithmic composition.Computer Music Journal, 13(4):pp. 27–43, 1989.

Folk music style modelling by recurrent neural networks with long short term memory units

https://biblio.ugent.be/publication/7053070/file/7053255

Folk music style modelling by recurrent neural networks with long short term memory units

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography