The Web Audio Evaluation Tool is an open-source, browser-

based framework for creating and conducting listening tests.

It allows remote deployment, GUI-guided setup, and analy-

sis in the browser.  While currently being used for listening

tests in various fields, it was initially developed specifically

for the study of music production practices.  In this work,

we highlight some of the features that facilitate evaluation

of such content

De Man, Brecht

Jillings, N.

Moffat, David

Reiss, Joshua

Stables, Ryan

BCU Open Access

Proceedings of the 2nd AES Workshop on Intelligent Music Production, London, UK, 13 September 2016SUBJECTIVE COMPARISON OF MUSIC PRODUCTION PRACTICESUSING THEWEB AUDIO EVALUATION TOOLBrecht De Man1, Nicholas Jillings2, David Moffat1, Joshua D. Reiss1 and Ryan Stables21 Centre for Digital MusicQueen Mary University of London{b.deman,d.j.moffat,joshua.reiss}@qmul.ac.uk2 Digital Media Technology LabBirmingham City Universitynicholas.jillings@mail.bcu.ac.uk, ryan.stables@bcu.ac.ukABSTRACTThe Web Audio Evaluation Tool is an open-source, browser-based framework for creating and conducting listening tests.It allows remote deployment, GUI-guided setup, and analy-sis in the browser. While currently being used for listeningtests in various fields, it was initially developed specificallyfor the study of music production practices. In this work,we highlight some of the features that facilitate evaluationof such content.1. INTRODUCTIONPerceptual evaluation of audio is an important part of re-search in virtually every audio-related field, and music pro-duction research is no exception. To facilitate listening testdesign without time-intensive development, compatibilityissues, and limited interface options, the Web Audio Eval-uation Tool was developed [1]. Initially developed with theevaluation of mixes in mind [2], the tool has since evolved toan all-round listening test platform, used and shaped by re-searchers from a wide range of audio research topics. Someprominent characteristics relevant to intelligent music pro-duction are discussed here.2. FEATURESIn many cases, tests are run on one or more dedicated com-puters in listening rooms, sometimes in different countries.As these computers may have different operating systems orsupporting software, developing an interface that works onall machines can be a challenge. This led to the choice fora browser-based tool, which only needs a browser that sup-ports the Web Audio API and is therefore platform-indepen-dent. Equally important is the opportunity for ‘remote’ teststhis offers, meaning no installation or setup of the tool isrequired, but participants can simply go to a website.The comparison of differently processed musical sig-nals is further made possible by instantaneous switching be-tween stimuli, and synchronised playback of time-alignedaudio fragments. This leads to seamless transitions wherethe relevant sonic characteristics change immediately whilethe source signal seemingly continues to play, and avoidsexcessive focus on the first few seconds of long fragments.This is especially useful for the comparison of different sig-nal processing algorithms or parameters. Optional cross-fading or inter-fragment silence accommodates other typesof tests.Whereas most available software still requires a sub-stantial amount of programming or tedious configuration onbehalf of the user, the Web Audio Evaluation Tool allowsanything from test setup to visualisation of results to happenentirely in the browser, making it attractive to researcherswith less technical backgrounds as well. The code or testconfiguration files only need to be altered when advancedmodifications need to be made. Even for users proficient inweb design, the GUI allows very quick setup and an imme-diate overview of test results, see Figure 1.Figure 1: The test creation GUI allows quick and easy setup of any but the most advanced and customised interfacesBased on experience from several researchers workingon different topics, various other features have been addedto address common difficulties and issues when designing,Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).Proceedings of the 2nd AES Workshop on Intelligent Music Production, London, UK, 13 September 2016conducting and analysing tests. Surveys are integrated inthe interfaces so that answers are collected immediately be-fore or after the test, or even before or after specific pagesin the test. These include validation of responses and vari-ous question formats such as checkboxes, radio buttons, andembedded videos.The test timeline and ratings can be visualised imme-diately upon finishing the test, to spot errors or misunder-standings quickly, or even do basic analysis of the resultsthus far in the browser, see Figures 2 and 3.0 50 100 150 200 250 300 350 400Time [seconds]BadPoorFairGoodExcellent PREFERENCE    AABBCCDDEEFFGGHHHIFigure 2: Timeline for a specific subject and test page,showing playback of the stimuli (red) and movement of theassociated sliders as a function of timeFigure 3: Box plot visualisation of results from several sub-jects3. INTERFACESEvery common interface type for perceptual evaluation issupported by the Web Audio Evaluation Tool, and each canbe customised to a large extent. For a modest number ofstimuli, a multiple stimulus interface is preferred over sin-gle stimulus or pairwise comparison [3, 4]. As there is usu-ally no reference when comparing differently processed ver-sions of a musical signal, the popular MUSHRA interface[5] may not be appropriate. Furthermore, modified versionsof MUSHRA featuring a single rating axis with multiplemarkers have been shown to be more accessible [6], akin tothe APE-style interface [7] which typically includes com-ment boxes as well - see Figure 4.Figure 4: Example multi-stimulus test interface4. CONCLUDING REMARKSThe Web Audio Evaluation Tool is a versatile, browser-based listening test platform, currently used for various ap-plications including the evaluation of music production prac-tices and audio processing algorithms. Apart from the timesaved by using an off-the-shelf, feature-rich tool, researchalso benefits from an experimental apparatus that is well-documented and widely used.The authors highly welcome any feedback and contri-butions on the GitHub page1. Source code is available as aZIP or through git1 or Mercurial2.5. REFERENCES[1] N. Jillings, D. Moffat, B. De Man, J. D. Reiss andR. Stables, “Web Audio Evaluation Tool: A frameworkfor subjective assessment of audio,” in 2nd Web AudioConf., April 2016.[2] N. Jillings, D. Moffat, B. De Man, and J. D. Reiss,“Web Audio Evaluation Tool: A browser-based listen-ing test environment,” in 12th SMC Conf., July 2015.[3] B. Cauchi et al., “Perceptual and instrumental evalua-tion of the perceived level of reverberation,” in ICASSP,pp. 629–633, Mar 2016.[4] B. De Man and J. D. Reiss, “A pairwise and multi-ple stimuli approach to perceptual evaluation of micro-phone types,” in AES Convention 134, May 2013.[5] Method for the subjective assessment of intermediatequality level of coding systems. Recommendation ITU-R BS.1534-1, 2003.[6] C. Vo¨lker and R. Huber, “Adaptions for the MUltiStimulus test with Hidden Reference and Anchor(MUSHRA) for elder and technical unexperienced par-ticipants,” in DAGA, Mar 2015.[7] B. De Man and J. D. Reiss, “APE: Audio PerceptualEvaluation toolbox for MATLAB,” in AES Convention136, Apr 2014.1github.com/BrechtDeMan/WebAudioEvaluationTool2code.soundsoftware.ac.uk/projects/webaudioevaluationtoolLicensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

Subject comparison of music production practices using the web audio evaluation tool

Birmingham City University Open Access Repository

http://www.open-access.bcu.ac.uk/4110/1/subjective%20comparison.pdf

Subject comparison of music production practices using the web audio evaluation tool

Abstract

Similar works

Full text

Available Versions

BCU Open Access

Birmingham City University Open Access Repository