Emotion is an essential aspect of music, and its recognition is a prevalent research topic in the field of computer audition. Machine learning-based Music Emotion Recognition ( MER) systems could boost the accessibility of music collections by providing standardised methodologies of music categorisation. In this paper, we introduce our (team name: AugsBurger) machine learning architecture sequentially composed of a convolutional feature extractor with block attention modules and a recurrent stack with self-attention for automatic MER. We train 5 models and conduct various late fusion experiments. Utilising a Convolutional Recurrent Neural Network ( CRNN ) with convolutional block attention applied throughout a 18-layer ResNet and a single recurrent layer with a Gated Recurrent Unit cell, a ROC-AUC of 73.9 % can be achieved on the test partition of the MediaEval 2020 Emotion & Themes in Music task. Applying late fusion on the individual model predictions and another challenge submission, this result is further increased to 75.3 % ROC-AUC

Amiriparian, Shahin

Gerczuk, Maurice

Ottl, Sandra

Rajamani, Srividya Tirunellai

Schuller, Björn W.

English

OPUS Augsburg

Emotion and Themes Recognition in Music with Convolutionaland Recurrent Attention-BlocksMaurice Gerczuk1, Shahin Amiriparian1, Sandra Ottl1, Srividya Tirunellai Rajamani1,Björn Schuller1,21Chair of Embedded Intelligence for Health Care & Wellbeing, Univeristy of Augsburg, Germany2GLAM – Group on Language, Audio, & Music, Imperial College London, U. K.maurice.gerczuk@uni-a.deABSTRACTEmotion is an essential aspect of music, and its recognition is aprevalent research topic in the field of computer audition. Machinelearning-based Music Emotion Recognition (MER) systems couldboost the accessibility of music collections by providing standard-ised methodologies of music categorisation. In this paper, we intro-duce our (team name: AugsBurger) machine learning architecturesequentially composed of a convolutional feature extractor withblock attention modules and a recurrent stack with self-attentionfor automatic MER. We train 5 models and conduct various latefusion experiments. Utilising a Convolutional Recurrent Neural Net-work (CRNN) with convolutional block attention applied through-out a 18-layer ResNet and a single recurrent layer with a GatedRecurrent Unit cell, a ROC-AUC of 73.9% can be achieved on thetest partition of the MediaEval 2020 Emotion & Themes in Musictask. Applying late fusion on the individual model predictions andanother challenge submission, this result is further increased to75.3 % ROC-AUC.1 INTRODUCTIONThe ability of music to express emotions is a demonstrable andeminent fact [18]. Emotional experiences of music are complexand dependent on factors related to the states and traits of thelistener, the performer, and the listening context, with researchsuggesting that musical structure alone is a key determinant ofthe emotional indication of music [25]. Different music emotioncategories can induce emotional states, such as happiness, sadness,hope, excitement, and joy in listeners [16, 19]. This is primarilydue to the affective information encoded in musical parameters,including melody, timbre, rhythm, and dynamics which are implic-itly decoded by listeners [13, 17]. Conventional feature extractionmethods (e. g., openSMILE [15]) have shown their suitability to ex-tract such features from music recordings [28, 32]. However, thestate-of-the-art for MER is defined by contemporary machine learn-ing approaches which utilise convolutional and recurrent neuralnetworks and learn data representations directly from the audiosignals (or spectrograms) instead of extraction of pre-defined hand-crafted features [3, 20, 21, 29]. Moreover, the integration of anattention mechanism in such systems has shown promise for var-ious audio recognition tasks [6, 26]. Motivated by our previousworks with CRNNs [2, 3, 5] and the success of attention mech-anisms [4, 6, 26, 31], in this paper, we introduce an end-to-endCopyright 2020 for this paper by its authors. Use permitted under Creative CommonsLicense Attribution 4.0 International (CC BY 4.0).MediaEval’20, December 14-15 2020, Onlineframework composed of two attention blocks: a ConvolutionalNeural Network (CNN) with Convolutional Block Attention Mod-ules (CBAMs), and a recurrent block with self-attention for the taskof emotion and theme recognition in music [8–10].2 APPROACHA high-level overview of our approach is depicted in Figure 1. Theframework consists of a CNN feature extractor enhanced byCBAMsand an RNN with self-attention. The convolutional block aims tolearn high-level shift-invariant features, whilst long(er)-term tem-poral dependencies of music data [8–10] are mainly extracted bythe recurrent block [1]. For all experiments in this paper, the MTG-Jamendo dataset [10] is solely used. The 18 486 audio tracks ofthe MTG-Jamendo dataset [10] are annotated in 56 distinct moodand theme categories, with every track having at least one tag.The dataset provides 60-20-20 % splits for training, validation, andtesting. A full description of the dataset can be found in [10].2.1 Pre-Processing and AugmentationOur model uses the pre-computed mel-spectrograms that are partof the challenge dataset. Furthermore, we only use random win-dows of 8 seconds (500 timesteps) during training, reducing thememory footprint of our models and also serving as a form of dataaugmentation. Additionally, we apply SpecAugment [22], randomlyapplying masks of a maximum width of 10 (timesteps or frequencybands) to both the frequency and time domains of the spectrograms.We do not, however, use warping. During validation and testing,we take an 8 second chunk from the middle of each spectrogramand do not apply SpecAugment.2.2 Convolutional Block Attention ModuleIn the CNN part of our modes, we use CBAMs [30] to refine thelearn feature maps. CBAMs sequentially apply channel and spatialattention to the max and average pooled outputs of a convolutionallayer. The attention maps are applied by element-wise multipli-cation. As these modules are a very lightweight extension, andshow consistent performance increases for a wide range of popu-lar image-recognition benchmarks [30], we evaluate their efficacywhen added to a CRNN for music emotion and theme recognition.2.3 Recurrent Self-AttentionIn the RNN head of our CRNN framework, we use an attentionmechanism to help themodel focus on important parts of the featuresequences extracted by the CNN. This is done by applying self-attention [12] to the RNN outputs and states at each time step,MediaEval’20, December 14-15 2020, Online M. Gerczuk et al.predictionsRaw AudiosMel-spectrograms SpecAugmentGRU/LSTMGRU/LSTMselfattentionrefinedfeaturesinputfeatures channelattentionmodulespatial attention modulex xConvolutional Block Attention ModuleinputfeaturesinputfeaturesinputfeaturesrefinedfeaturesrefinedfeaturesrefinedfeaturesFigure 1: Overview of our system composed of a CBAM enhanced convolutional feature extractor followed by a RecurrentNeural Network (RNN) stack with self-attention. A detailed account of the framework is given in Section 2.finally forming a compact representation for the whole sequence.We use the scaled dot product attention of Vaswani et al. [27].2.4 Attention CRNNCombining a CNN feature extractor with CBAMs and an RNN headwith self-attention leads to our final attention CRNN. Specifically,we use an 18-layer ResNet architecture and replace the global pool-ing layer with an RNN stack. In the ResNet, we apply CBAMs inevery convolutional block right before adding the residual. For theRNN, we evaluate using Gated Recurrent Unit (GRU) and LongShort-Term Memory (LSTM) cells. We started with a single layerwith 256 units for both cell types. As we found GRUs to performbetter, we additionally trained a model containing two recurrentlayers of this type with 128 units each. Finally, the step-wise outputsand the final hidden state are used in the self-attention mechanismas keys and values, and query, respectively. Finally, a fully con-nected layer with sigmoid activation is used to perform the themeand mood tagging of the input audio samples. We train all of ourmodels for a maximum of 100 epochs with an Adam optimiser withthe learning rate set to 0.0003 but stop the training early if thevalidation ROC-AUC does not improve for 20 epochs. We use theweights from the best epoch (measured in validation ROC-AUC)for evaluation on the held-out test set.2.5 Fusion ExperimentsWe apply late fusion to the results achieved by our attention CRNNs(models with CBAMs) through averaging prediction scores. Further-more, we fuse our predictions with another system submitted to thechallenge [23] which uses standalone self-attention an attention-based Rectified Linear Units (ReLUs) [11] added to the challengebaseline’s vggish model [9].3 RESULTS AND ANALYSISThe results of our experiments are shown in Table 1. The bestindividual model can be found with a CBAM enhanced CRNN witha single GRU layer. This model reaches 73.9% ROC-AUC on thetest partition, compared to the challenge baseline of 72.5 %. Usingthe same architecture but without applying CBAMs, only 69.4%are achieved on test. Furthermore, we observe that models with asingle GRUs layer outperform their LSTM counterparts. Fusing thetwo best CRNNs – CBAMs in the CNN and GRU cells in the RNN –further leads to a slight performance boost to 74.1 %. More effectiveis fusing with the attention enhanced CNN from [23], achievingour best result of 75.3 % ROC-AUC and 13.1 % PR-AUC on test. Thishints at complementarity of the two systems.Table 1: Performance of our proposed approaches measuredin macro ROC-AUC. Baseline ROC-AUC on test is 72.5 % [9].CRNNconv. attention RNN cell RNN units validation testno attention GRU 256 69.9 69.4no attention LSTM 256 70.0 69.3block attention GRU 2 × 128 67.9 71.6block attention GRU 256 71.8 73.9block attention LSTM 256 69.4 69.4attention CNN [23] - - 72.8 72.8Fusionblock attention with GRU 74.1block attention with GRU+ [23] 75.3A noteworthy characteristic of all the systems used in our sub-mission to the challenge is that they do not make use of any externaldata and only train on short extracts of the songs (about 10 secondslong). Furthermore, none of the models were trained for more than50 epochs, against the challenge baseline’s 1 000 epochs. In this way,our attention models reduce data, memory and time requirementswhile achieving stronger performance than the baseline. Comparedto the other challenge submissions, only one submission that doesnot rely on external data outperforms our best fusion model1.4 DISCUSSION AND OUTLOOKWe have introduced a CRNN architecture with attention modulesfor both convolutional (cf. Section 2.2) and recurrent blocks (cf. Sec-tion 2.3) for emotions and themes recognition in music. In thepre-processing step, in order to achieve a better model generalisa-tion, we have augmented the spectrograms from the music record-ings with SpecAugment [22] and trained our models with bothchallenge and augmented data (cf. Section 2.1). Furthermore, as apost-processing step, we have conducted a set of late (decision-level)fusion experiments to check the complementary of the predictionsfrom each trained model (cf. Section 2.5). The results indicate theefficacy of our applied methodologies for this challenge(cf. Sec-tion 3). Considering the performance increase achieved by utilis-ing attention-based ReLUs with the baseline’s vggish architecturein [23], it is worth investigating this activation mechanism in com-bination with the CBAM enhanced CRNNs presented herein. Asour models only make use of the challenge data itself, one shouldalso consider exploiting external data, such as the Million SongDataset [7], Music4All [24] or NSynth [14] for possible improve-ments in model accuracy.1https://multimediaeval.github.io/2020-Emotion-and-Theme-Recognition-in-Music-Task/resultsMediaEval’20, December 14-15 2020, OnlineEmotion and Themes in MusicREFERENCES[1] Shahin Amiriparian. 2019. Deep representation learning techniques foraudio signal processing. Ph.D. Dissertation. Technische UniversitätMünchen.[2] Shahin Amiriparian, Alice Baird, Sahib Julka, Alyssa Alcorn, SandraOttl, Suncica Petrović, Eloise Ainger, Nicholas Cummins, and BjörnSchuller. 2018. Recognition of Echolalic Autistic Child VocalisationsUtilising Convolutional Recurrent Neural Networks. In Proceedings ofINTERSPEECH 2018, 19th Annual Conference of the International SpeechCommunication Association. ISCA, Hyderabad, India, 2334–2338.[3] Shahin Amiriparian, Maurice Gerczuk, Eduardo Coutinho, Alice Baird,Sandra Ottl, Manuel Milling, and Björn Schuller. 2019. Emotion andthemes recognition in music utilising convolutional and recurrentneural networks. InMediaEval Benchmarking Initiative for MultimediaEvaluation. Sophia Antipolis, France.[4] Shahin Amiriparian, Maurice Gerczuk, Sandra Ottl, Alice Baird, LukasStappen, Lukas Koebe, and Björn Schuller. 2020. Towards Cross-Modal Pre-Training and Learning Tempo-Spatial Characteristics forAudio Recognition with Convolutional and Recurrent Neural Net-works. EURASIP Journal on Audio, Speech, and Music Processing 2020(2020), to appear.[5] Shahin Amiriparian, Sahib Julka, Nicholas Cummins, and BjörnSchuller. 2018. Deep Convolutional Recurrent Neural Networksfor Rare Sound Event Detection. In Proceedings 44. Jahrestagung fürAkustik, DAGA 2018. DEGA, Deutsche Gesellschaft für Akustik e.V.(DEGA), Munich, Germany.[6] Shahin Amiriparian, Pawel Winokurow, Vincent Karas, Sandra Ottl,Maurice Gerczuk, and Björn Schuller. 2020. Unsupervised Representa-tion Learning with Attention and Sequence to Sequence Autoencodersto Predict Sleepiness From Speech. In Proceedings of the 1st Interna-tional on Multimodal Sentiment Analysis in Real-life Media Challengeand Workshop. 11–17.[7] Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and PaulLamere. 2011. The million song dataset. (2011).[8] Dmitry Bogdanov, Alastair Porter, Philip Tovstogan, and Minz Won.2019. MediaEval 2019: Emotion and Theme Recognition in MusicUsing Jamendo. In MediaEval Benchmarking Initiative for MultimediaEvaluation. Sophia Antipolis, France.[9] Dmitry Bogdanov, Alastair Porter, Philip Tovstogan, and Minz Won.2020. MediaEval 2020: Emotion and Theme Recognition in MusicUsing Jamendo. In MediaEval Benchmarking Initiative for MultimediaEvaluation. Online.[10] Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, andXavier Serra. 2019. The MTG-Jamendo Dataset for Automatic MusicTagging. In Machine Learning for Music Discovery Workshop, Interna-tional Conference on Machine Learning (ICML 2019). ICML, Long Beach,CA, United States.[11] Dengsheng Chen and Kai Xu. 2020. AReLU: Attention-based RectifiedLinear Unit. (2020). arXiv:arXiv:2006.13858[12] Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long Short-Term Memory-Networks for Machine Reading. In Proceedings of the2016 Conference on Empirical Methods in Natural Language Processing.Association for Computational Linguistics, Austin, Texas, 551–561.https://doi.org/10.18653/v1/D16-1053[13] Eduardo Coutinho and Björn Schuller. 2017. Shared acoustic codesunderlie emotional communication in music and speech—Evidencefrom deep transfer learning. PloS one 12, 6 (2017), e0179289.[14] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Moham-mad Norouzi, Douglas Eck, and Karen Simonyan. 2017. Neural audiosynthesis of musical notes with wavenet autoencoders. In InternationalConference on Machine Learning. PMLR, 1068–1077.[15] Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile:the munich versatile and fast open-source audio feature extractor. InProceedings of the 18th ACM international conference on Multimedia.1459–1462.[16] Bruce Ferwerda and Markus Schedl. 2014. Enhancing Music Recom-mender Systems with Personality Information and Emotional States:A Proposal.. In Umap workshops.[17] Alf Gabrielsson and Erik Lindström. 2010. The role of structure in themusical expression of emotions. In Handbook of music and emotion:Theory, research, applications, Patrik N. Juslin and John Sloboda (Eds.).Oxford University Press, Oxford, 367–400.[18] Patrik N. Juslin and John Sloboda (Eds.). 2011. Handbook of music andemotion: Theory, research, applications. Oxford University Press.[19] Ai Kawakami, Kiyoshi Furukawa, Kentaro Katahira, and KazuoOkanoya. 2013. Sad music induces pleasant emotion. Frontiers inpsychology 4 (2013), 311.[20] Khaled Koutini, Shreyan Chowdhury, Verena Haunschmid, HamidEghbal-zadeh, and Gerhard Widmer. 2019. Emotion and Theme Recog-nition in Music with Frequency-Aware RF-Regularized CNNs. arXivpreprint arXiv:1911.05833 (2019).[21] Maximilian Mayerl, Michael Vötter, Hsiao-Tzu Hung, Bo-Yu Chen,Yi-Hsuan Yang, and Eva Zangerle. 2019. Recognizing Song Mood andTheme Using Convolutional Recurrent Neural Networks. InMediaEvalBenchmarking Initiative for Multimedia Evaluation. Sophia Antipolis,France.[22] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, BarretZoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simpledata augmentation method for automatic speech recognition. arXivpreprint arXiv:1904.08779 (2019).[23] Srividya Tirunellai Rajamani, Kumar Rajamani, and Björn Schuller.2020. Emotion and Theme Recognition in Music using Attention-based Methods. In MediaEval Benchmarking Initiative for MultimediaEvaluation. Online.[24] Igor André Pegoraro Santana, Fabio Pinhelli, Juliano Donini, LeonardoCatharin, Rafael Biazus Mangolin, Valéria Delisandra Feltrim, Mar-cos Aurélio Domingues, and others. 2020. Music4All: A New MusicDatabase and Its Applications. In 2020 International Conference onSystems, Signals and Image Processing (IWSSIP). IEEE, 399–404.[25] Klaus R. Scherer and Eduardo Coutinho. 2013. How music createsemotion: a multifactorial process approach. In The Emotional Power ofMusic: Multidisciplinary Perspectives on Musical Arousal, Expression,and Social Control, T Cochrane, B Fantini, and K R Scherer (Eds.).Number 10. Oxford University Press, 121–145.[26] Lorenzo Tarantino, Philip N Garner, and Alexandros Lazaridis. 2019.Self-Attention for Speech Emotion Recognition.. In INTERSPEECH.2578–2582.[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At-tention is all you need. In Advances in neural information processingsystems. 5998–6008.[28] Felix Weninger, Florian Eyben, and Björn Schuller. 2013. The TUMapproach to the MediaEval music emotion task using generic affectiveaudio features. In MediaEval Benchmarking Initiative for MultimediaEvaluation. Barcelona, Spain.[29] Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. 2020.Evaluation of CNN-based Automatic Music Tagging Models. arXivpreprint arXiv:2006.00751 (2020).[30] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.2018. CBAM: Convolutional Block Attention Module. In Proceedingsof the European Conference on Computer Vision (ECCV).MediaEval’20, December 14-15 2020, Online M. Gerczuk et al.[31] Seunghyun Yoon, Seokhyun Byun, Subhadeep Dey, and Kyomin Jung.2019. Speech emotion recognition using multi-hop attention mecha-nism. In ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP). IEEE, 2822–2826.[32] Fan Zhang, Hongying Meng, and Maozhen Li. 2016. Emotion extrac-tion and recognition from music. In 2016 12th International Confer-ence on Natural Computation, Fuzzy Systems and Knowledge Discovery(ICNC-FSKD). IEEE, 1728–1733.

Emotion and themes recognition in music with convolutional and recurrent attention-blocks

https://opus.bibliothek.uni-augsburg.de/opus4/files/116965/paper60.pdf

Emotion and themes recognition in music with convolutional and recurrent attention-blocks

Abstract

Similar works

Full text

Available Versions

OPUS Augsburg