This paper examines the effect of sensor performance on speaker diarisation in meetings and investigates the use of more advanced beamforming techniques, beyond the typically employed delay-sum beamformer, for mitigating the effects of poorer sensor performance. We present superdirective beamforming and investigate how different time difference of arrival (TDOA) smoothing and beamforming techniques influence the performance of state-of-the-art diarisation systems. We produced and transcribed a new corpus of meetings recorded in the instrumented meeting room using a high SNR analogue and a newly developed low SNR digital MEMS microphone array (DMMA.2). This research demonstrates that TDOA smoothing has a significant effect on the diarisation error rate and that simple noise reduction and beamforming schemes suffice to overcome audio signal degradation due to the lower SNR of modern MEMS microphones. Index Terms — Speaker diarisation in meetings, digital MEMS microphone array, time difference of arrival (TDOA), superdirective beamforming 1

Lincoln, M.

Renals, S.

Zwyssig, E.

English

This paper examines the effect of sensor performance on speaker diarisation in meetings and investigates the use of more advanced beamforming techniques, beyond the typically employed delay-sum beamformer, for mitigating the effects of poorer sensor performance. We present super-directive beamforming and investigate how different time difference of arrival (TDOA) smoothing and beamforming techniques influence the performance of state-of-the-art diarisation systems. We produced and transcribed a new corpus of meetings recorded in the instrumented meeting room using a high SNR analogue and a newly developed low SNR digital MEMS microphone array (DMMA.2). This research demonstrates that TDOA smoothing has a significant effect on the diarisation error rate and that simple noise reduction and beamforming schemes suffice to overcome audio signal degradation due to the lower SNR of modern MEMS microphones

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      On the effect of SNR and superdirective beamforming in speakerdiarisation in meetingsCitation for published version:Zwyssig, E, Renals, S & Lincoln, M 2012, 'On the effect of SNR and superdirective beamforming in speakerdiarisation in meetings'. in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE InternationalConference on. pp. 4177-4180, ICASSP 2012, Kyoto, Japan, 26-30 March.,10.1109/ICASSP.2012.6288839Digital Object Identifier (DOI):10.1109/ICASSP.2012.6288839Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Author final version (often known as postprint)Published In:Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference onGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015ON THE EFFECT OF SNR AND SUPERDIRECTIVE BEAMFORMING IN SPEAKERDIARISATION IN MEETINGSErich Zwyssig1,2, Steve Renals1 and Mike Lincoln11Centre for Speech Technology Research, University of Edinburgh, Edinburgh, EH8 9AB, Scotland UK2EADS IW, Appleton Tower, 6th Floor, Edinburgh, EH8 9LE, Scotland UKABSTRACTThis paper examines the effect of sensor performanceon speaker diarisation in meetings and investigates the useof more advanced beamforming techniques, beyond the typ-ically employed delay-sum beamformer, for mitigating theeffects of poorer sensor performance. We present super-directive beamforming and investigate how different timedifference of arrival (TDOA) smoothing and beamformingtechniques influence the performance of state-of-the-art diar-isation systems. We produced and transcribed a new corpusof meetings recorded in the instrumented meeting room us-ing a high SNR analogue and a newly developed low SNRdigital MEMS microphone array (DMMA.2). This researchdemonstrates that TDOA smoothing has a significant effecton the diarisation error rate and that simple noise reductionand beamforming schemes suffice to overcome audio sig-nal degradation due to the lower SNR of modern MEMSmicrophones.Index Terms— Speaker diarisation in meetings, digitalMEMS microphone array, time difference of arrival (TDOA),superdirective beamforming1. INTRODUCTIONSpeaker diarisation is the process of determining who spokewhen during a conversation. Diarisation systems typicallyidentify both the number of speakers in the recording andthe time intervals during which each individual is speaking.Speaker diarisation has recently been used in the analysis ofmeeting recordings which has shown that accurate diarisationis crucial to the performance of subsequent processes, such asspeaker recognition and transcription [1].Meetings are usually recorded using microphone arraysconsisting of a number of high quality analogue microphoneswhich provide a high signal to noise ratio (SNR). However,microphone arrays have recently been developed using digitalMEMS microphones (so-called silicon microphones). MEMSmicrophone arrays have a number of advantages (e.g. lowerprice, smaller size), however the MEMS sensors have the dis-advantage of significantly lower SNRs than their analoguecounterparts.In this paper we study the effect of the sensor perform-ance on the diarisation task and investigate the use of super-directive beamforming to mitigate the effects of poorer sensorperformance.2. BACKGROUNDTime delay of arrival (TDOA) estimation seeks to identify thetime difference between signals from a given source arrivingat two different sensors in a sensor array and is an essentialfirst step in most beamforming techniques. An establishedmethod for performing TDOA estimation is the generalisedcross correlation with phase transform (GCC-PHAT [2],[3])which can be used to determine the relative delay betweensignals arriving at two microphones in a microphone array.The GCC-PHAT of two signals is defined asGˆPHAT (f) =Xi(f)[Xj(f)]∗|Xi(f)[Xj(f)]∗| (1)where xi(t) and xj(t) are two discrete signals in the timedomain and Xi(f) and Xj(f) their discrete Fourier trans-form. The TDOA dˆPHAT (i, j) of the two signals xi(t)and xj(t) is estimated as the maximum value of the inverseFourier transform RˆPHAT of GˆPHAT :dˆPHAT (i, j) = arg maxd(RˆPHAT (d)) (2)GCC-PHAT does not produce stable delay estimates whenused in acoustically noisy environments (such as a typicalmeeting room), and smoothing techniques, such as Viterbidelay selection, can be used to obtained better estimates [4].In our experiments we compare the performance of smoothedand un-smoothed delay estimates for beamforming in termsof the achieved diarisation error rate.For acoustic beamforming, the delays between eachmicrophone and a reference channel (typically taken as thechannel with the highest energy level) are calculated, andthese can be directly used for delay-sum beamforming. Theoutput of a delay-sum beamformer is the weighted sum ofall the microphone signals, with each channel delayed by itsZwyssig, E., Renals, S., & Lincoln, M. (2012). On the effect of SNR and superdirective beamforming in speaker diarisation in meetings.  In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. (pp. 4177-4180). 10.1109/ICASSP.2012.6288839corresponding delay estimate:y(n) =1MN∑m=1xm(n− τm) (3)where y(n) is the output signal of the beamformer, M thenumber of microphones, xm the input signal at microphonem and τm the delay of that input signal.One commonly used measure of the performance ofbeamforming techniques is the array gain, G, which showsthe improvement of the signal to noise ratio of the arraycompared to an individual sensor:G =SNRarraySNRsensor(4)Delay-sum beamforming achieves a signal amplificationof 3dB for every doubling of the number of microphones. En-hancement is achieved by constructively adding the signalsfrom the look direction and suppressing interference fromother sources. By optimising the array gain, more sophist-icated methods, known as superdirective beamformers, canbe used to improve the beamformers directional selectivity atlower frequencies, further cancelling undesired sources. Anumber of superdirective beamformers have been developed.Examples include filter-sum, differential, eigen, generalisedsidelobe cancelling, and minimum variance distortionless re-sponse (MVDR) beamformers, each being differentiated bythe method employed to optimise G.In this work we employ an MVDR superdirective beam-former [5]. The aim of MVDR beamforming is to minimisethe power of the output signal of the array, while maintain-ing unity gain in the look direction and additional constraints(such as maximum white noise gain). MVDR beamformingis based on filter-sum beamforming and its frequency domainoutput signal, Yb, is defined as:Yb(ejΩ) =M−1∑m=0Wm∗(ejΩ)Xm(ejΩ) = WHX (5)where Wm(ejΩ) denotes the filter coefficients of the beam-former for sensor m at frequency Ω, Xm(ejΩ) are the micro-phone input signals and [·]H denotes the matrix transpose con-jugate.3. DMMA.2Most microphone arrays to date have been composed of highquality, but expensive and relatively bulky, analogue micro-phones. A digital MEMS (micro electro mechanical system)microphone is a microphone on a chip containing a pressuresensitive membrane, a matched pre-amplifier, and integratedanalogue-digital conversion (ADC) and downsampling. Wehave previously constructed a prototype digital MEMS micro-phone array, DMMA.1 [6], and preliminary experiments pro-duced promising results for a task based on the adaptation(a) Microphones on daughter boards (b) Complete microphone arrayFig. 1. The digital MEMS microphone arrayof WSJ acoustic models. DMMA.1 has a number of limit-ations, most significantly the inability to directly record allchannels individually at 48kHz sample rate. In order to over-come this problem, a second microphone array has been con-structed which allows the recording of 8 microphone channelsat sample rates from 8 kHz to 48 kHz.In this work we have designed a new array, DMMA.2(Figure 1), which like DMMA.1 is an 8 channel circularmicrophone array with a diameter of 20 cm. It is built usingADI ADMP441 omnidirectional MEMS microphones1 withbottom port and I2S output and the Rigisystems USBPAL2, aUSB 2.0 multi-channel audio interface for Windows PC andMAC OS X.Digital MEMS microphones have significantly lower in-trinsic signal to noise ratios compared to analogue micro-phones. Initial tests on the microphones used in the DMMA.2suggest that this sensor noise is not white as would be expec-ted. While SNR and THD measurements carried out showthe microphones to be within specification, the MEMS micro-phones output a non-white chirping noise, which we suspectoriginates from the DSP built into the microphones. Furthertests, including sensor measurements in a vacuum enclosureare being conducted.4. AD IMR CORPUSThe DMMA.2 and an array with identical geometry con-structed using high signal to noise ratio analogue micro-phones have been used to simultaneously record six researchmeetings of around one hour in length. The recordings weremade in a typical meeting room at the University of Edin-burgh. The analogue array is identical to that used in theAMI meeting corpus recordings and is fully documented in[1]. From each of the recordings, a continuous ten to fifteenminute segment containing lively discussion has been selec-ted, creating a total of approximately 78 minutes of record-ings. These extracts were transcribed to show speech/non-1http://www.analog.com/en/mems-sensors/microphones/admp441/products/product.html2http://www.rigisystems.net/Zwyssig, E., Renals, S., & Lincoln, M. (2012). On the effect of SNR and superdirective beamforming in speaker diarisation in meetings.  In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. (pp. 4177-4180). 10.1109/ICASSP.2012.6288839Recording Length [s] # of speakersrec14june2011 825 5rec15june2011 804 7rec21june2011 630 4rec22june2011 856 4rec28june2011 607 4rec29june2011 914 6Table 1. Summary of AD IMR recordingsMicrophoneArrayNoiseReductionNoiseReductionTDOAEstimation(Viterbi smoothing)TDOAEstimationDelay-sumBeamformingSuperdirective Beamforming VAD and DiarisationGWmdm toolsqio-fe BeamformItSHOuTICSIFig. 2. Data flow for the experimentsspeech events and for each speech segment the speaker IDwas annotated. Both overlapping speech (where more thanone speaker is talking simultaneously) and back channels(short interjections from listeners, typically indicating agree-ment or disagreement with the main speaker) were includedin the transcription. The transcription was formatted using theRTTM specification, as defined by NIST 3, allowing scoringof automatically generated diarisation annotations using thestandard NIST evaluation tools. Details of the meeting re-cordings contained in the corpus, named AD IMR, are listedin Table 1.5. METHODSExperiments were conducted to investigate the effect on thediarisation task of using the digital array and superdirectivebeamforming. Using two state-of-the-art diarisation systems,we compared the error rates achieved using the low SNRrecordings from the DMMA.2 with recordings of the samemeeting from the analogue array. Using both smoothed andun-smoothed delay estimates, we then compared diarisationerrors using the MVDR beamformer and the currently useddelay-sum beamformer.Figure 2 shows the data flow for the experiments. Initially,Wiener-filter-based noise reduction is applied to the analogueand digital microphone signals [7] and both smoothed andunsmoothed TDOA values for each of the channels calcu-lated [4]. Enhanced signals are then generated using threetechniques: (1) Delay-sum beamforming using smootheddelay estimates; (2) Superdirective beamforming using un-smoothed delay estimates; (3) Superdirective beamforming3http://www.itl.nist.gov/iad/mig/tests/rt/using smoothed delay estimates. We used the open sourceBeamformIt toolkit4 [8] and the AMI project beamformingtools [9].Speaker diarisation is then performed on the three en-hanced signals using two diarisation systems—SHOUTspeech recognition toolkit [10] and the ICSI speaker diar-isation system [11]5.6. RESULTSTwo metrics are used to verify the performance of speakerdiarisation systems—the voice activity detection error rate(VER) and the diarisation error rate (DER). The VER is cal-culated from missed speech and false alarms—that is, speechsegments that are not detected as speech and non-speech seg-ments that are identified as speech. In addition to missedspeech and false alarms, DER (see [8], chapter 6.1.3, page162ff) also takes into account the speaker to whom eachsegment is assigned, and penalises segments assigned to thewrong speaker. In order to account for errors in the refer-ence labels and slight variations in automatic processing, atolerance of ±250ms is permitted at the edge of each speechsegment.The VER and DER results for the six meetings in theAD IMR corpus are given in Table 2. The results show that,for diarisation, the new digital microphone array compareswell with the analogue array despite the reduced SNR, pro-ducing only marginally increased error rates. This result sug-gests that MEMS microphone technology provides a viablealternative to expensive analogue devices for speech data cap-ture, and further experiments will be conducted on a varietyof speech processing tasks using the DMMA.2Table 2 also shows that Viterbi smoothing of the TDOAcoefficients and delay-sum beamforming provide betterresults than superdirective beamforming using either smoothedor un-smoothed delays. This may in part be due to the fact thatthe TDOA smoothing method was optimised for diarisationperformance using a delay-sum beamformer, and alternativeoptimisation may be required in the superdirective case. Also,it is possible that the superdirective beamformer actually re-moves vital acoustic information from the sidelobes, leadingto an increased DER due to the diarisation tools being tunedto acoustic output from a delay-sum beamformer. Analysingthe effect of the superdirective beamformer white noise gainconstraint GW on the diarisation error rate, it was found that,by tuning GW , the performance gap between the digital andanalogue arrays could be reduced. In general, reducing theGW leads to a decrease of the difference in the DER betweenthe analogue and digital arrays, as shown in Figure 3, with4http://www.xavieranguera.com/beamformit/5The implementation of the ICSI system evaluated here only uses acousticfeatures, in contrast to the system used in the ICSI submission to the NISTRT09 evaluation which incorporates TDOA features directly as an input tothe diarisation system.Zwyssig, E., Renals, S., & Lincoln, M. (2012). On the effect of SNR and superdirective beamforming in speaker diarisation in meetings.  In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. (pp. 4177-4180). 10.1109/ICASSP.2012.6288839Table 2. % VER and DER for delay-sum (DSB) and superdirective (SDB) beamforming using the ICSI and SHOUT diarisationsystems, for analogue and digital arrays. FA denotes false alarms, MS denotes missed speech.SHOUT ICSIDER VER FA MS DER VER FA MSDSB analogue 20.54 2.3 1.3 1 22.49 2.2 1.3 0.9(TDOA smoothing) digital 21.89 3 1.5 1.5 22.81 2.9 1.5 1.4SDB analogue 29.21 4.8 3.5 1.3 28.17 4.7 3.5 1.2GW=0.6 digital 35.16 4.9 3 1.9 30.31 4.8 3.1 1.7modified SDB analogue 23.11 3.6 1.9 1.7 21.58 3.5 1.9 1.6GW=0.6 (TDOA smoothing) digital 25.45 3.7 1.6 2.1 28.82 3.7 1.7 2Fig. 3. Effect of white noise gain constraint GW on DERbest performance achieved by setting GW < 0.15.7. CONCLUSIONS AND FUTURE WORKIn this paper we have presented the development of a newdigital MEMS microphone array. We have recorded a newcorpus of 6 meetings using both the digital array and an ana-logue array, and annotated 78 minutes of data extracted fromthe recordings for speech/non-speech and speaker identifica-tion. We have compared the performance of two state-of-the-art diarisation systems using both the analogue and digital re-cordings, and a number of delay estimation and beamformingtechniques.We found that the digital MEMS microphone array ap-proaches the performance of the analogue array when usingsuperdirective beamforming, if the white noise gain constraintof the beamformer is correctly adjusted. In addition, we foundthat superdirective beamforming, even when using delay es-timation smoothing, is unable to match the diarisation per-formance of delay-sum beamforming and believe this may becaused by mismatch between the beamformer output and thediarisation systems used.Future work will investigate optimising the TDOA estim-ation and diarisation system for such a beamformer in order toincrease performance. We also plan to record more meetingswith the DMMA.2, some with improved speaker tracking andothers in an anechoic chamber, to investigate the effects ofSNR and reverberation on diarisation performance.8. REFERENCES[1] T. Hain, L. Burget, J. Dines, P.N. Garner, A.E. Hannani,M. Huijbregts, M. Karafiat, M. Lincoln, and V. Wan, “TheAMIDA 2009 meeting transcription system,” in Interspeech,2010.[2] C. Knapp and G. Carter, “The generalized correlation methodfor estimation of time delay,” IEEE Transactions on Acoustics,Speech and Signal Processing, 1976.[3] M. S. Brandstein and H. F. Silverman, “A robust method forspeech signal time-delay estimation in reverberant rooms,” inICASSP, 1997.[4] X. Anguera, C. Wooters, and J. Hernando, “Acoustic beam-forming for speaker diarization of meetings,” IEEE Transac-tions on Audio, Speech, and Language Processing, 2007.[5] J. Bitzer and K. Uwe Simmer, “Superdirective microphonearrays,” in Microphone arrays: signal processing techniquesand applications, M. Brandstein and E Ward, Eds. SpringerVerlag, 2001.[6] E.P. Zwyssig, M. Lincoln, and S. Renals, “A digital micro-phone array for distant speech recognition,” in ICASSP, 2010.[7] A. Adami, L. Burget, S. Dupont, H. Garudadri, F. Grezl,H. Hermansky, P. Jain, S. Kajarekar, N. Morgan, and S. Siva-das, “Qualcomm-ICSI-OGI front end archive,” http://www.icsi.berkeley.edu/Speech/papers/qio/,2002.[8] X. Anguera, Robust Speaker Diarization for Meetings, Ph.D.thesis, Universitat Polite`cnica de Catalunya, 2006.[9] G. Lathoud, I. McCowan, and D. Moore, “Segmentingmultiple concurrent speakers using microphone arrays,” inEurospeech, 2003.[10] M. Huijbregts, “SHOuT Speech Recognition Toolkit,” http://shout-toolkit.sourceforge.net/, 2006.[11] G. Friedland, A. Janin, D. Imseng, X. Anguera, L. Gottlieb,M. Huijbregts, M. Knox, and O. Vinyals, “The ICSI RT-09 speaker diarization system,” IEEE Transactions on Audio,Speech, and Language Processing, 2012.Zwyssig, E., Renals, S., & Lincoln, M. (2012). On the effect of SNR and superdirective beamforming in speaker diarisation in meetings.  In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. (pp. 4177-4180). 10.1109/ICASSP.2012.6288839

A digital microphone array for distant speech recognition,” in ICASSP,

A robust method for speech signal time-delay estimation in reverberant rooms,” in ICASSP,

Acoustic beamforming for speaker diarization of meetings,”

meeting transcription system,” in Interspeech,

On the effect of SNR and superdirective beamforming in speaker diarisation in meetings.

Qualcomm-ICSI-OGI front end archive,” http:// www.icsi.berkeley.edu/Speech/papers/qio/,

Robust Speaker Diarization for Meetings, Ph.D. thesis, Universitat Polit` ecnica de Catalunya,

Segmenting multiple concurrent speakers using microphone arrays,” in Eurospeech,

SHOuT Speech Recognition Toolkit,” http: //shout-toolkit.sourceforge.net/,

Superdirective microphone arrays,” in Microphone arrays: signal processing techniques

The generalized correlation method for estimation of time delay,”

On the effect of SNR and superdirective beamforming in speaker diarisation in meetings

Erich Zwyssig

Steve Renals

Mike Lincoln

Crossref

On the effect of snr and superdirective beamforming in speaker diarisation in meetings

http://www.cstr.inf.ed.ac.uk/downloads/publications/2012/zwyssig-dmma2-icassp12.pdf

On the effect of SNR and superdirective beamforming in speaker diarisation in meetings

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Crossref