Bullock, J.

Jillings, N.

Stables, Ryan

Birmingham City University Open Access Repository

JS-Xtract: A Realtime Audio Feature Extraction Library for the Web

BCU Open Access

JS-XTRACT: A REALTIME AUDIO FEATURE EXTRACTION LIBRARYFOR THEWEBNicholas JillingsDigital Media Technology LabBirmingham City UniversityBirmingham, UKnicholas.jillings@mail.bcu.ac.ukJamie BullockIntegra LabBirmingham City UniversityBirmingham, UKjamie.bullock@bcu.ac.ukRyan StablesDigital Media Technology LabBirmingham City UniversityBirmingham, UKryan.stables@bcu.ac.ukABSTRACTJS-Xtract is an efficient modular JavaScript library for au-dio feature extraction, capable of operating on arbitrarytime-series data, or being bound to Web Audio objects.The library implements an extensive range of vector andscalar feature extractors, and allows both procedural andobject-oriented function calls. We show it performs wellacross a range of desktop and mobile browsers, and is ca-pable of extracting audio features in realtime.1. INTRODUCTIONWith the introduction of the Web Audio API [1], high reso-lution audio can now be handled natively in the web browser.This allows for the development of tools such as listen-ing test platforms [5] and online audio processing environ-ments [13], which are developed natively using JavaScriptand HTML 5. These tools often require audio feature ex-traction to perform analysis on time-series data, howeverlibraries that are developed for compiled languages such asLibXtract [2] for C, YAAFE [7] for Python or jAudio [8]for Java can be difficult to deploy due security limitationsimposed by the browser. As an alternative, libraries suchas Meyda [12] allow for inline Javascript implementation,with a limited feature set, bound to Web Audio API ob-jects. This limits the extent to which non-realtime analysiscan be implemented.Typically these libraries utilise a subset of features de-scribed by the CUIDADO [14] and MPEG 7 [6] featuresets, where extensive groups of descriptors are presented.These have been used as a benchmark for inclusion in re-view papers [9] incorporating temporal, spectrotemporaland abstracted feature representations. Here, we presentJS-Xtract, 1 a light-weight JavaScript library for featureextraction, which is agnostic of the data’s type or origin.1 Library available at http://www.semanticaudio.co.uk/projects/js-xtract/c© Nicholas Jillings, Jamie Bullock, Ryan Stables. Li-censed under a Creative Commons Attribution 4.0 International License(CC BY 4.0). Attribution: Nicholas Jillings, Jamie Bullock, Ryan Sta-bles. “JS-Xtract: A Realtime Audio Feature Extraction Library for theWeb”, Extended abstracts for the Late-Breaking Demo Session of the17th International Society for Music Information Retrieval Conference,2016.2. LIBRARY DESIGNWe follow the design philosophy of LibXtract [2] by defin-ing low-level, modular function calls which can be com-bined to create complex feature extraction graphs. The li-brary is written in JavaScript following the ECMAScript 5standard for maximum support and functionality. 2 Func-tions can be passed an arbitrary datasource, providing itmeets the dimensionality criteria, and can be bound to WebAudio objects for real-time processing. The library sup-ports both procedural and object-oriented function calls.2.1 Feature SetWe extend the LibXtract feature set, which includes sta-tistical moments and shape descriptors such as centroid,spread and roll-off, each of which can be extracted from arange of input representations such as a magnitude spec-trum, peak- and harmonic-peak spectrum (HPS), MFCCsand Bark coefficients. Input representations can be cate-gorised as temporal, spectrotemporal, and abstracted, whereabstracted representations typically model an external sys-tem (including MFCCs, Bark or Chroma). Two additionalinput representations are included, namely chroma features(included in the Chroma Toolbox [3]) which characterisespectral energy distributed across pitch classes, and Equiv-alent Rectangular Bandwidth (ERB) filters (included in theTimbre Toolbox [11]), which represent auditory filters pro-posed by Moore and Glasberg [10]. In addition, we in-corporate temporal envelope, spectral flux and filter-bankdeltas, which were omitted from LibXtract due to the lackof frame-based decomposition. From this, we can extracta range of features from the amplitude envelope, such asLog-attack time and temporal centroid. The library primar-ily follows the structure in Figure 1, in which most scalarfeatures can be extracted for an arbitrary input representa-tion, including the original time-series frame of audio, withthe exception of f0, loudness, HPS and noisiness, whichrely on specific input representations.2.2 Web Audio API IntegrationThe Web Audio API [1] provides audio processing func-tionality for every major mobile and desktop browser. 32 Current ECMAScript 5 support can be found at http://kangax.github.io/compat-table/es5/3 At time of publication, only Opera Mini is not supported http://caniuse.com/#feat=audio-apiInputAudioFrameOutputFeatureVectorInputRepresentationFeatureExtractorFigure 1: The JS-Xtract feature extraction topologyThe API defines several nodes to enable processing andis configured using client-side JavaScript. JS-Xtract usesan Analyser node to extract the current time and frequencydomain blocks for analysis, then adds a prototype functionto set up an accurate interval callback using a ScriptPro-cessorNode to call a user-supplied function on each frame.This causes multiple function calls to occur inside the mainJavaScript thread. Using the Web Audio API’s AudioWorkernode would be preferable for this task since it would bepossible to run extraction functions in a separate JavaScriptthread, however at the time of publication there are nobrowsers that implement this node, since the underlyingWebWorker standard is incomplete [4]. To ensure sepa-rability, the current version of JS-Xtract includes the WebAudio API specific functions in a separate file (jsXtract-wa.js), allowing sites or projects not using the Web AudioAPI to still use the library.2.3 Additional FunctionalityFrames: The library implements frame-based decomposi-tion with variable frame and hop sizes for customisableoverlap. This converts a vector representing a stream ofaudio into sub-regions. The decomposed array is iterableand JS-Xtract supplies a prototype to simplify processing.Each frame is iterated over by calling the user suppliedfunction with the current frame, previous frame and theprevious computed value. This allows for the extractionof deltas and delta-deltas without recomputing features.Typed arrays: For most analysis blocks, the source datais read into a JavaScript Typed Array. These allow mul-tiple ‘views’ on the memory, allowing for shallow copies.Therefore data is stored efficiently as the same memoryspace is referred to twice, rather than copied, saving sys-tem memory. JavaScript performs all floating point calcu-lations in double precision, therefore the only cost to usingdouble is the increased memory footprint and conversionbetween single and double.Output: The library supports simple derivation of mul-tiple output formats. In JS-Xtract, data is returned as aJavaScript Object which can be converted to a JSON stringor other data store such as XML for transmission.2.4 ImplementationWhen using the JSXtract object, multiple instances ofitems such as the DCT, MFCC and Wavelets can be man-aged by the class. Calling new jsXtract(); will buildan object containing the initialisers and references to thefunction calls. Features can then be extracted usingobj.features.xtract [Feature Name]. TheFloat32Array and Float64Array objects have the followingFeature FF C S ETonality 0.791 0.712 0.411 0.277SSD 1.022 1.094 0.415 0.334SD 0.819 0.627 0.377 0.248ASDF 2,095 4,280 5,453 28,185AMDF 2,319 4,252 2,476 3,102DCT 56,395 184,412 50,618 142,830Table 1: Feature performance in ns on up-to-date (July-2016) desktop browsers, where FF: Firefox, C: Chrome,S: Safari, E: EdgeFeature iPhone iPad Nexus LinxTonality 0.621 1.287 1.324 1.046SSD 1.108 1.429 1.741 1.048SD 0.517 1.263 1.192 0.959ASDF 7,246 18,779 9,096 45,319AMDF 4,491 9,971 8,824 8,305DCT 64,297 169,967 315,077 347,625Table 2: Feature performance in ns on mobile browserstwo prototype functions applied:xtract get data frames andxtract process frame data to enable the frame de-composition and frame iteration on any floating point ar-ray data type, such as those returned for the Web AudioAPI buffer. Alternatively, for C-like procedural functioncalls, xtract [Feature Name] can be used, which isaligned with LibXtract syntax.3. PERFORMANCETo demonstrate the performance of the library, feature ex-traction was performed on a sine wave polluted with Gaus-sian white noise, which was 1024 samples long. Each fea-ture was iterated 3,000 times on 41 computer - browserpairs. The fastest and slowest 3 features per call are pre-sented in Table 1 for each of the major desktop browsers,and in Table 2 for mobile platforms.Millisecond accurate timestamps are obtained throughthe W3C High Resolution Time Level 2, 4 where most ma-jor desktop and mobile browsers, support the use of theAPI natively. 5 A timestamp is taken before and after theiterations, giving the total time to execute the functions.Firefox showed the best overall performance, althoughthe slowest for the scalar features. Chrome and Edge bothshowed unstable performance for the DCT calculations whilstFirefox and Safari proved consistent results for the vectorfeatures (slowest three). The results show the library out-performs other JavaScript feature extraction libraries forreal-time performance, and exhibits relative consistencyacross platforms when using scalar features. Given the ef-ficiency, the library has the capacity to support real-timeaudio applications on both desktop and mobile interfaces.4 At time of publication, latest draft 25th February 20165 See http://caniuse.com/#feat=high-resolution-time4. REFERENCES[1] Paul Adenot, Chris Wilson, and Chris Rogers. Web Au-dio API. W3C, October, 10, 2013.[2] Jamie Bullock. Libxtract: A lightweight library for au-dio feature extraction. In Proceedings of the Interna-tional Computer Music Conference, volume 43, 2007.[3] Sebastian Ewert. Chroma toolbox: Matlab implemen-tations for extracting variants of chroma-based audiofeatures. In Proc. ISMIR, 2011.[4] Ian Hickson. Web workers, 2015. Available at http://www.w3.org/TR/workers/.[5] Nicholas Jillings, David Moffat, Brecht De Man,Joshua D Reiss, and Ryan Stables. Web audio evalu-ation tool: A framework for subjective assessment ofaudio. In The 2nd Web Audio Conference (WAC). Geor-gia, US, 2016.[6] Bangalore S Manjunath, Philippe Salembier, andThomas Sikora. Introduction to MPEG-7: multimediacontent description interface, volume 1. John Wiley &Sons, 2002.[7] Benoit Mathieu, Slim Essid, Thomas Fillon, JacquesPrado, and Gae¨l Richard. Yaafe, an easy to use andefficient audio feature extraction software. In ISMIR,pages 441–446, 2010.[8] Cory McKay, Ichiro Fujinaga, and Philippe Depalle.jaudio: A feature extraction library. In Proceedings ofthe International Conference onMusic Information Re-trieval, pages 600–3, 2005.[9] David Moffat, David Ronan, and Joshua D Reiss.An evaluation of audio feature extraction toolboxes.In International Conference on Digital Audio Effects(DAFx), 2016.[10] Brian CJ Moore and Brian R Glasberg. Suggested for-mulae for calculating auditory-filter bandwidths andexcitation patterns. The Journal of the Acoustical So-ciety of America, 74(3):750–753, 1983.[11] Geoffroy Peeters, Bruno L Giordano, Patrick Susini,Nicolas Misdariis, and Stephen McAdams. The tim-bre toolbox: Extracting audio descriptors from musicalsignals. The Journal of the Acoustical Society of Amer-ica, 130(5):2902–2916, 2011.[12] Hugh Rawlinson, Nevo Segal, and Jakub Fiala. Meyda:an audio feature extraction library for the web audioapi. In The 1st Web Audio Conference (WAC). Paris,Fr, 2015.[13] Ryan Stables, Sean Enderby, Brecht De Man, Gyo¨rgyFazekas, and Joshua Reiss. Safe: A system for the ex-traction and retrieval of semantic audio descriptors. In15th International Society for Music Information Re-trieval Conference (ISMIR 2014), 2014.[14] Hugues Vinet, Perfecto Herrera, and Franc¸ois Pachet.The cuidado project. In International Conference onMusic Information Retrieval, pages 197–203, 2002.

http://www.open-access.bcu.ac.uk/4099/1/JS%20Xtract.pdf

JS-Xtract: A Realtime Audio Feature Extraction Library for the Web

Abstract

Similar works

Full text

Available Versions

Birmingham City University Open Access Repository

BCU Open Access