Deleu, Johannes

Demeester, Thomas

Develder, Chris

Mertens, Laurent

Sterckx, Lucas

English

Ghent University Academic Bibliography

Assessing Quality of Unsupervised Topicsin Song LyricsLucas Sterckx, Thomas Demeester,Johannes Deleu, Laurent Mertens, and Chris DevelderGhent University - iMinds, Belgium<firstname>.<lastname>@intec.ugent.beAbstract. How useful are topic models based on song lyrics for ap-plications in music information retrieval? Unsupervised topic models ontext corpora are often difficult to interpret. Based on a large collection oflyrics, we investigate how well automatically generated topics are relatedto manual topic annotations. We propose to use the kurtosis metric toalign unsupervised topics with a reference model of supervised topics.This metric is well-suited for topic assessments, as it turns out to bemore strongly correlated with manual topic quality scores than existingmeasures for semantic coherence. We also show how it can be used for adetailed graphical topic quality assessment.1 IntroductionThis paper presents an analysis of how well topic models can be used to detectlyrical themes for use in Music Information Retrieval (MIR), an interdisciplinaryscience developing techniques including music recommendation.Probabilistic topic models are a tool for the unsupervised analysis of text, pro-viding both a predictive model of future text and a latent topic representationof the corpus. Latent Dirichlet Allocation (LDA) is a Bayesian graphical modelfor text document collections represented by bags-of-words [1]. In a topic model,each document in the collection of documents is modeled as a multinomial distri-bution over a chosen number of topics, each topic is a multinomial distributionover all words. We evaluate the quality and usefulness of topic models for newmusic recommendation applications.Although lyricism and themes are undeniably contributing to a musical identity,they are often treated as mere secondary features, e.g., for obtaining music orartist similarity, which are dominantly determined by the audio signal. Never-theless, previous works have analyzed lyrics, mainly aimed at determining themajor themes they address. Mahadero et al. [2] performed a small scale eval-uation of a probabilistic classifier, classifying lyrics into five manually appliedthematic categories. Kleedorfer et al. [3] focused on topic detection in lyrics us-ing an unsupervised statistical model called Non-negative Matrix Factorization(NMF) on 32,323 lyrics. After clustering by NMF, a limited evaluation was per-formed by a judgment of the most significant terms for each cluster. We expandon this work by performing a large-scale evaluation of unsupervised topic modelsusing a smaller dataset of labeled lyrics and a supervised topic model.While state-of-the-art unsupervised topic models lead to reasonable statisticalmodels of documents, they offer no guarantee of producing results that are in-terpretable by humans and require a thorough evaluation of the output. Whenconsidering lyrics, there is no general consensus on the amount and nature ofthe main themes, as opposed to news-corpora (sports, science,. . . ). A usefultopic model for MIR, appends the music with a representation of the thematiccomposition of the lyrics. For use in applications like music recommendation,playlist generation, . . . , the topics should be interpretable. Evaluation method-ologies based on statistical [1] or coherence [4] measures are not optimal for thispurpose since they do not account for interpretability and relevance to the appli-cation. Chuang et al. [5] introduced a framework for the large-scale assessmentof topical relevance using supervised topics and alignment between unsupervisedand supervised topics.Our contributions presented in this paper apply and build on aforementionedwork, by assessing quality of unsupervised topics for use in MIR, and by in-troducing a new method for measuring and visualizing the quality of topicalalignment, based on the kurtosis of the similarity between unsupervised topicsand a reference set of supervised topics.In Section 2, we present the data and our experimental set-up. The main topicmodel analysis is presented in Section 3, followed in Section 4 by conclusions.We used the results presented below to create an online demo that demonstratesthe use of high-quality topics for MIR with an application which automaticallygenerates playlists based on preference of lyrical themes. This demo can be foundat http://users.ugent.be/~lusterck (Login = demo:ldamir).2 Experimental SetupThe main dataset used for this research is the ‘Million Song Dataset’(MSD) [6],with metadata for one million songs, and lyrics as bags-of-words for a subset of237,662 songs from a commercial lyrics catalogue, ‘musiXmatch’.LDA was applied on the set of lyrics, using the java-based package MALLET [7].Three topic models were inferred from the subset of 181,892 English lyrics forevaluation1, one with 60 (T60), 120 (T120) and 200 (T200) topics. A manualquality assessment of all of these topics was performed, with scores ranging from1 (useless) to 3 (highly useful).As an additional resource, a clean dataset of labels was provided by the website,‘GreenbookofSongs.com R©’2 (GOS), a searchable database of songs categorizedby subject. This dataset contains 9,261 manually annotated song lyrics matchedwith the MSD (a small subsample of the GOS’ complete database), with multiplelabels from a large class-hierarchy of 24 super-categories with a total of 877 sub-categories. Labeled Latent Dirichlet Allocation (L-LDA) is a variation of LDA1 Track-id’s can be provided upon request2 http://www.greenbookofsongs.com, the authors would like to thank Lauren Vir-shup and Jeff Green for providing access to the GOS-databasefor labeled corpora by incorporating user supervision in the form of a one-to-onemapping between topics and labels [8]. An L-LDA model with 38 supervisedtopics was inferred from the much smaller set of GOS-labeled lyrics, based onthe GOS super-categories (but with the omission of minor categories like ‘Tools’,and splitting up of major categories like ‘Love’). These are high-quality topics,but because of the limited size of the GOS data set, less representative for theentire scope of themes in the complete MSD lyrics collection.3 Topic Model AssessmentThe suitability of topic models for use in MIR is determined by the amount ofrelevant and interpretable topics they produce. We first introduce suitable met-rics to evaluate to what extent unsupervised topics can be mapped to supervisedtopics obtained from tagged documents. We then show how these can be used asa better measure for the interpretability of topics than an existing alternative,and provide a visual analysis of topical alignment.3.1 Measuring Topical AlignmentWe define high-quality topics as topics for which a human judge finds a clear se-mantic coherence between the relevant terms in relation to an underlying concept(such as ‘Love’, or ‘Christmas’). Such concepts are made explicit by an L-LDAmodel based on tagged documents, and we detect high-quality LDA-topics asthose that bear a strong resemblance with L-LDA topics. For an unsupervisedtopic to represent a highly distinctive theme, ideally it should be highly similarto only a single supervised topic. For each of the unsupervised LDA-topics, thecosine similarity between the word-topic probability distribution is calculatedwith the distribution of each L-LDA topic.We introduce two metrics to assess the distribution of these similarities per LDA-topic, which measure how strongly the variance of the mean cosine similaritydepends on extreme values (in this case, because of similarities that are muchhigher than the average). The first is the excess kurtosis (γ2), traditionally usedto detect peakedness and heavy tails. The second is the normalized maximumsimilarity (zmax), used in several outlier detection strategies.γ2 =µ4σ4− 3, zmax = Xmax − µσ(1)with µ4 the fourth moment about the mean µ, σ the standard deviation, andXmax the maximum similarity. Figure 1 shows the similarities with the unsuper-vised topics for the high-quality LDA-topic 29, with a clearly matched supervisedtopic (and high values for γ2 and zmax), and for the low-quality LDA-topic 39(with low γ2 and zmax). The insets show the histograms of the similarities. Var-ious other metrics were evaluated as well, but with a lower ability of detectingthe interesting distributions.3.2 Semantic CoherenceA second evaluation was performed using metrics presented in [4], where theauthors show that measures for semantic coherence are highly correlated withChristmas FamilyMusic/RockingPeopleNightLoveSeasonsHome/HouseLaw/CrimeNatureTimeTravelling/MovingNumbersSpace/Moon andStarsFoodAnimalsWar/PeaceSports/GamesDrugs/AlcoholHeartbreakReligionCommunicationSleepDreamsAnatomyMedia/Showbiz LifePoliticalSexColors0.00.10.20.30.40.50.60.70.8CosineSimilarityTopic 29: christmas bells snow santa ring merry years tree bright sleighγ2 = 25, 91zmax = 5.750.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Cosine Distance024681012FrequencyPlaces/CitiesWaterNatureFamilyLaw/CrimePoliticalHome/HouseWar/PeaceTravelling/MovingAnimalsLove LifeSeasonsDrugs/AlcoholReligionFoodHeartbreakAnatomyTimeNumbersColorsWeatherSpace/Moon andStarsSociety/ClassesSports/GamesCommunicationSexMusic/RockingChristmas0.00.10.20.30.40.50.60.70.8CosineSimilarityTopic 39: love man tree young hand home land mother day menγ2 = 0, 15zmax = 2.700.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Cosine Distance01234567FrequencyFig. 1: Kurtosis measure and Normalized Maximum Similarity for topic evalua-tionhuman quality judgments. These metrics use WordNet, a lexical ontology, toscore topics by measuring the average distance between words of a topic usinga variety of distance metrics based on the ontology. The best performing metricwas reported to be the LESK-metric [9], based on lexical overlap in dictionarydefinitions. Table 1 shows the Spearman rank correlation between the LESKscore for each topic and the manually assigned quality scores. For comparison,the rank correlation between the manual quality scores and γ2 and zmax (ascalculated in Section 3.1) are shown as well, and lead to significantly highercorrelation values than with the LESK metric.Table 1: Spearman correlations with manual quality-scores for the three topicmodelsEvaluation Metric T60 T120 T200Semantic Coherence using Wordnet (LESK) 0,35 0,23 0,31Kurtosis (γ2) 0,49 0,49 0,56Normalized Maximum Similarity (zmax) 0,49 0,50 0,533.3 Graphical Alignment of TopicsWe can visualize the alignment between the supervised and unsupervised topicsby calculating the kurtosis on the similarities between both topic sets. Theseare shown in Fig. 2, a correspondence chart similar to the one presented in [5],for the 60 topics LDA-model (T60). Our chart differs from the one presentedin [5] in that it uses topics from an L-LDA model for the matching of unsuper-vised topics instead of a list of words generated by experts, and uses bar-chartsto display the automatically calculated kurtosis scores instead of likelihoods ofhuman-assisted matching. The size of the circles denotes the cosine similaritybetween the corresponding supervised and unsupervised topics, and the coloringshows which concepts are matched in a one-to-one fashion by the unsupervisedand supervised topics using the harmonic mean of both kurtosis’ values. Notethat the detection of topics is dependent on the labels included in the super-vised data. High-quality LDA-topics, not present in the supervised set, are notdetected. The chart shows that topics involving Christmas, Fire and Water areall very distinguishable by statistical models and human-assisted labeling, orresolved. Other topics are linked to more labels and contain fused concepts orjunk. Another use of this chart is evaluating the reference topics by the expertsof GOS. Some concepts devised by experts may be chosen too broadly. For ex-ample, the supervised topic of Music/Rocking is close in cosine-distance to Topic6 and to Topic 54, which in turn is close to the supervised theme Dancing/Party.This indicates that labeling for Music/Rocking should be confined more to mu-sic and exclude songs about dancing. Topics like Love and Heartbreak correlatewith many LDA-topics which demonstrate their dominance in lyrical themes.4 ConclusionThis paper provides insights into the quality of topic models constructed fromsong lyrics. We showed that the kurtosis is a suitable metric to align unsuper-vised topics with supervised reference topics, which allows detecting high-qualitytopics in accordance to manual quality assessments.References1. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. the Journal ofmachine Learning research 3 (2003) 993–10222. Logan, B., Kositsky, A., Moreno, P.: Semantic analysis of song lyrics. In: Interna-tional Conference on Multimedia and Expo, ICME 2004. Volume 2., IEEE (2004)827–8303. Kleedorfer, F., Knees, P., Pohle, T.: Oh oh oh whoah! towards automatic topicdetection in song lyrics. In: Proceedings of the 9th ISMIR. (2008) 287–2924. Newman, D., Karimi, S., Cavedon, L.: External evaluation of topic models. In:Australasian Document Computing Symposium (ADCS). (2009) 1–85. Chuang, J., Gupta, S., Manning, C.D., Heer, J.: Topic model diagnostics: Assessingdomain relevance via topical alignment. In: ICML. (2013)6. McFee, B., Bertin-Mahieux, T., Ellis, D.P., Lanckriet, G.R.: The million songdataset challenge. In: Proceedings of the 21st international conference companionon World Wide Web, ACM (2012) 909–9167. McCallum, A.K.: Mallet: A machine learning for language toolkit.http://mallet.cs.umass.edu (2002)8. Ramage, D., Hall, D., Nallapati, R., Manning, C.D.: Labeled lda: A supervisedtopic model for credit attribution in multi-labeled corpora. In: Proceedings of the2009 Conference on EMNLP, ACL (2009) 248–2569. Lesk, M.: Automatic sense disambiguation using machine readable dictionaries:how to tell a pine cone from an ice cream cone. In: Proceedings of the 5th annualinternational conference on Systems documentation, ACM (1986) 24–26419546445128Drugs/AlcoholNumbersSleepDreamsSexTimeEducation/AdviceCommunicationLaw/CrimeTravelling/MovingLifeHeartbreakLoveSociety/ClassesPoliticalAnatomyMedia/ShowbizColorsPeopleFoodAnimalsSports/GamesHome/HouseSpace/MoonandStarsNightNatureFamilyReligionWorkMoneyPlaces/CitiesWeatherSeasonsWar/PeaceDancing/PartyMusic/RockingWaterFireChristmas294195464451284531495927224611235155756114525042344038134392573647431053026378583024335516121932483171852320210 5 10 15 20 25 3005101520253035Fig.2:CorrespondenceChartbetweentopics.Thesizeofcirclesdepictscosinesimilaritybetweencorrespondingsupervisedandunsupervisedtopics.Barsonthesidesofthegraphshowthekurtosisscoresfortheircorrespondingtopics.Highscoresshowthattopicsarealigned,lowscoresmeantopicsarenotmatchedorjunkinthecaseofLDA-topics.CirclecoloringmeansastrongmatchbetweenLDAandL-LDAtopics,thusausefulunsupervisedtopic.

Assessing quality of unsupervised topics in song lyrics

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography

Crossref