International audienceIn this paper, we present a web-based interactive tool for exploring a collection of two million commercially released songs. It gathers song information from a large number of heterogeneous sources, web-based and audio-based, and integrates work from multiple research groups. The resulting tool can be used to request information about a specific song such as lyrics, metadata and chords; to navigate further on to linked external resources such as Discogs, AllMusic, Mu-sicBrainz or a number of streaming providers; or to browse the collection by artist's discographies or band membership. Several Web Audio applications are integrated and use the dataset to enrich the experience

Buffa, Michel

Lebrun, Jerome

Pauwels, Johan

Pellerin, Guillaume

INRIA a CCSD electronic archive server

HAL Id: hal-02366730https://hal.inria.fr/hal-02366730Submitted on 18 Nov 2019HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.A 2 Million Commercial Song Interactive NavigatorMichel Buffa, Jerome Lebrun, Johan Pauwels, Guillaume PellerinTo cite this version:Michel Buffa, Jerome Lebrun, Johan Pauwels, Guillaume Pellerin. A 2 Million Commercial SongInteractive Navigator. WAC 2019 - 5th WebAudio Conference 2019, Dec 2019, Trondheim, Norway.￿hal-02366730￿A 2 Million Commercial Song Interactive NavigatorMichel BuffaJerome LebrunUniversité Côte d’AzurCNRS, INRIA{buffa, lebrun}@i3s.unice.frJohan PauwelsQueen Mary University ofLondonj.pauwels@qmul.ac.ukGuillaume PellerinIRCAMguillaume.pellerin@ircam.frABSTRACTIn this paper, we present a web-based interactive tool forexploring a collection of two million commercially releasedsongs. It gathers song information from a large number ofheterogeneous sources, web-based and audio-based, and in-tegrates work from multiple research groups. The resultingtool can be used to request information about a specific songsuch as lyrics, metadata and chords; to navigate further onto linked external resources such as Discogs, AllMusic, Mu-sicBrainz or a number of streaming providers; or to browsethe collection by artist’s discographies or band membership.Several Web Audio applications are integrated and use thedataset to enrich the experience.Figure 1: a schema of the WASABI network archi-tecture and workflow.Licensed under a Creative Commons Attribution 4.0 International License (CC BY4.0). Attribution: owner/author(s).Web Audio Conference WAC-2019, December 4–6, 2019, Trondheim, Norway.c© 2019 Copyright held by the owner/author(s).1. INTRODUCTIONFigure 2: the metadata of the WASABI databaseis complemented by computational analysis of lyricsand audio, both on-demand and precomputed.Since 2017, a 2M song database consisting of metadatacollected from the Web of Data [4] has been constructed inthe context of the WASABI research project1. The hetero-geneous sources that have been consulted are displayed onthe right of Figure 1.These metadata include the identifiers of the correspond-ing audio on a variety of audio platforms, which allowed toenrich the database with computational analyses of the lyricsand audio data (see Figure 2). Several research groups havecontributed to the analysis and have built interactive WebAudio applications on top of the output. For example, IR-CAM linked their TimeSide analysis and annotation tool tomake on-demand audio analysis possible. Queen Mary Uni-versity of London, through the FAST project2, performed anoffline chord analysis of 442k songs, and built an online en-hanced audio player [15] and chord search engine [16] aroundit. The I3S laboratory, responsible for the design and imple-mentation of the database, extracted song structure basedon lyrics analysis [10,12]. Furthermore, they have developeda rich set of Web Audio applications and plugins [6, 7] thatallow, for example, songs to be played along with soundssimilar to those used by artists.These metadata, computational analyses and Web Audioapplications have now been gathered in one easy-to-use webinterface, the WASABI Interactive Navigator, which is pre-sented in this paper.1http://wasabihome.i3s.unice.fr/2http://www.semanticaudio.ac.uk2. RELATED WORKSOther research projects aimed to collect metadata on alarge set of commercial songs, such as The Million SongDataset project from 2011, which is mainly based on audiodata [3] and did not exploit the availability of large struc-tured data sources from the Web of Data to the full extent.MusicWeb and its successor MusicLynx [1] link musicartists within a Web-based application for discovering con-nections between them and provides a browsing experienceusing extra-musical relations. The project shares some ideaswith WASABI, but works on the artist level, and does notperform analyses on the audio and lyrics content itself. Itreuses, for example, MIR metadata from AcousticBrainz.The WASABI project has been built on a broader scopethan these projects and mixes a wider set of metadata, in-cluding ones from audio and natural language processing oflyrics. In addition, as presented in this paper, it comes witha large set of Web Audio enhanced applications.3. THE WASABI INTERACTIVE NAVIGA-TOR HOMEPAGEThe primary starting point for the interactive navigatoris the home screen as seen in Figure 3. From here, a textualsearch can be performed for one of the 2 million songs, 200kalbums or 77k artists included in the database [4]. Theresulting page displays information aggregated from a hugeset of online sources, as shown in Figure 4.A set of tabs at the top link directly to the tools based onthe computational analysis of audio and lyrics, which willbe discussed next.4. SONG STRUCTURE DERIVED FROMLYRICS AND AUDIOOn the pages for individual songs, the musical structure isindicated by grouping the lyrics in blocks, as shown in Fig-ure 5. Song lyrics contain repeated patterns that facilitateautomated segmentation, with the detection of constitutiveelements of a song text (e.g., intro, chorus) as goal.We proposed to segment lyrics by applying a convolutionalneural network to a synchronized audio-text representationof a song. First, we created a corpus projecting the segmen-tation of the lyrics of the WASABI corpus onto a synchro-nized lyric-audio corpus (DALI corpus3). We have shownthat the information in the text enriched with the charac-teristics of the audio signal allows our segmentation modelto surpass the state of the art method, which is based solelyon textual characteristics [10,12].5. CHORD PLAYER AND SEARCHChord transcriptions can be requested from the songpages, and are presented as an interactive player showingthe chords in sync with the music, such that musicians canplay along with them (see Figure 6, top). As an alterna-tive entrypoint to the WASABI database, a chord searchinginterface is available to find specific songs that contain a cer-tain set of chords (bottom of Figure 6). This type of searchinterface has been previously tested with the Jamendo musiccollection [16].3https://github.com/gabolsgabs/DALIFigure 3: main page of the WASABI navigator in-terface.For the search interface to work, the songs need to beindexed offline. So far, 442k files have been analysed withthe chord analysis algorithm proposed in [14]. In order tocomply with copyright requirements, a remote processingtoolchain has been set-up in which algorithms packaged as aDocker container are sent to Deezer for processing on theirservers. The output is then returned to us and stored in theWASABI database. Calculation of further music descriptorssuch as tempo and key is ongoing.6. ON-DEMAND AUDIO ANALYSISOn-demand audio analysis is complementary to pre-computed analysis in the sense that it avoids large, up-front computational costs and scales easily to changes indata and algorithms. It comes with its own challenges, how-ever. Building a web platform that depends on various ex-ternal services requires that the underlying software archi-tecture and data model must be robust against disappearingsources. For example, if a YouTube video gets removed, westill want its computational analysis to remain available incase any of the web services built on top refer to the trackand its related metadata. The TimeSide framework4 hasbeen designed to provide a RESTful API as well as pluginbased core library dedicated to audio processing that can4https://github.com/Parisson/TimeSide/Figure 4: a large set of online databases has beenexploited for building the metadata part of theWASABI database.provide this resilience.In the context of the WASABI project, we demonstratehow the service (see Figure 1) can be used on demand by amaster semantical application that will feed TimeSide withURLs coming from various providers (YouTube, Deezer,etc.) and then dynamically return the analysis data to theclient application to feed Web Audio based tools. A num-ber of audio-based characteristics can be requested from asong’s page in this way.7. MULTITRACK PLAYER AND EFFECTSIn order to assist music schools, which is one of the targetaudiences of the WASABI project, some Web Audio toolsFigure 5: song lyric structure detection [10].have been integrated into the Navigator. One possible sce-nario is that music teachers pick a song from the databasefor their students to learn. An enhanced multi-track player(Figure 7) is presented to the users, which displays the song’ssheet music and allows to play back or selectively mute thedifferent instruments in the song. Furthermore, a simplifiedDAW is available in the browser for recording and playingback student or teacher performances. It includes real-timeaudio effects, with ready to use presets, to attain a realisticand attractive studio sound.The in-browser audio effects have been implemented us-ing a Web Audio plugin standard [8,9] (including SDK, on-line validation tools and examples of host applications), forwhich a large set of plugins5 already exists. An online IDEfor designing these plugins [11, 13] and a guitar tube ampsimulator designer [6, 7] are also available for developers.8. FUTURE ENHANCEMENTSFine-tuning the parameters in many of our Web Audioapplications (tube amplifiers, pedals, etc. . . ) has been quitetedious and time-consuming with a high level of expertise5https://github.com/micbuffa/WebAudioPluginsFigure 6: augmented player and search engine, in-tegrated in the navigator interface.Figure 7: a multitrack player capable of displayingGuitar Pro music tabs in sync with the audio [5].required [6, 7]. To ease this process of presetting for musicschool students, we are currently exploring machine-learningbased approaches (with scattering wavelet based convolu-tional neural networks [2]) to learn and extract the relevantfeatures from large datasets of songs and to match thesewith presets leading to similar subjective timbre in simu-lated instruments and also to automatically classify songsin the WASABI database.Furthermore, offline indexation of more audio-based char-acteristics is ongoing.9. ACKNOWLEDGMENTSThis work was supported by the French Research NationalAgency (ANR) and the WASABI team (contract ANR-16-CE23-0017-01). Part of this work has been funded by UKEPSRC grant EP/L019981/1.Figure 8: Web Audio embedded DAW and effectchain for recording and playback.10. REFERENCES[1] A. Allik, F. Thalmann, and M. Sandler. MusicLynx:Exploring music through artist similarity graphs. InCompanion Proc. (Dev. Track) The Web Conf.(WWW 2018), 2018.[2] J. Anden and S. Mallat. Deep scattering spectrum.IEEE Trans. Sig. Proc., 62(16), 2014.[3] T. Bertin-Mahieux, D. Ellis, B. Whitman, andP. Lamere. The Million Song Dataset. In Proc. 12thInt. Soc. Music Information Retrieval (ISMIR 2011),2011.[4] M. Buffa et al. WASABI: a two million song databaseproject with audio and cultural metadata plusWebAudio enhanced client applications. In Proc. 3rdWeb Audio Conf. (WAC 2017), 2017.[5] M. Buffa, A. Hallili, and P. Renevier. MT5: a HTML5multitrack player for musicians. In Proc. 1st WebAudio Conf. (WAC2015), 2015.[6] M. Buffa and J. Lebrun. Real time tube guitaramplifier simulation using webaudio. In Proc. 3rd WebAudio Conference (WAC 2017), 2017.[7] M. Buffa and J. Lebrun. Web audio guitar tubeamplifier vs native simulations. In Proc. 3rd WebAudio Conf. (WAC 2017), 2017.[8] M. Buffa, J. Lebrun, J. Kleimola, O. Larkin, andS. Letz. Towards an open web audio plugin standard.In Companion Proc. (Dev. Track) The Web Conf.(WWW 2018), 2018.[9] M. Buffa, J. Lebrun, J. Kleimola, O. Larkin, S. Letz,and G. Pellerin. WAP: Ideas for a Web Audio plug-instandard. In Proc. 4th Web Audio Conf. (WAC 2018),2018.[10] M. Fell, Y. Nechaev, E. Cabrio, and F. Gandon.Lyrics segmentation: Textual macrostructuredetection using convolutions. In Proc. 27th Int. Conf.on Computational Linguistics (COLING2018), 2018.[11] S. Letz, Y. Orlarey, and D. Fober. Compiling Faustaudio DSP code to WebAssembly. In Proc. 3rd WebAudio Conf. (WAC 2017), 2017.[12] G. Meseguer-Brocal, A. Cohen-Hadria, and G. Peeters.DALI: A large dataset of synchronized audio, lyricsand notes, automatically created using teacher-studentmachine learning paradigm. In Proc. 19th Int. Soc. ofMusic Information Retrieval (ISMIR 2018), 2018.[13] Y. Orlarey, D. Fober, and S. Letz. Syntactical andsemantical aspects of Faust. Soft Computing, 8(9),2004.[14] J. Pauwels, K. O’Hanlon, G. Fazekas, and M. Sandler.Confidence measures and their applications in musiclabelling systems based on hidden Markov models. InProc. 18th Int. Soc. Music Information Retrieval(ISMIR 2017), 2017.[15] J. Pauwels and M. Sandler. A web-based system forsuggesting new practice material to music learnersbased on chord content. In Joint Proc. 24th ACM IUIWorkshops (IUI2019), 2019.[16] J. Pauwels, A. Xambó, G. Roma, M. Barthet, andG. Fazekas. Exploring real-time visualisations tosupport chord learning with a large music collection.In Proc. 4th Web Audio Conf. (WAC 2018), 2018.

A 2 Million Commercial Song Interactive Navigator

https://hal.inria.fr/hal-02366730/file/WAC2019__Wasabi_demo_poster_wasabi_project.pdf

A 2 Million Commercial Song Interactive Navigator

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server