Automatic detection of different types of acoustic events is an interesting problem in soundtrack processing. Typical approaches to the problem use short-term spectral features to describe the audio signal, with additional modeling on top to take temporal context into account. We propose an approach to detecting and modeling acoustic events that directly describes temporal context, using convolutive non-negative matrix factorization (NMF). NMF is useful for finding parts-based decompositions of data; here it is used to discover a set of spectro-temporal patch bases that best describe the data, with the patches corresponding to event-like structures. We derive features from the activations of these patch bases, and perform event detection on a database consisting of 16 classes of meeting-room acoustic events. We compare our approach with a baseline using standard short-term mel frequency cepstal coefficient (MFCC) features. We demonstrate that the event-based system is more robust in the presence of added noise than the MFCC-based system, and that a combination of the two systems performs even better than either individually

Cotton, Courtenay Valentine

Ellis, Daniel P. W.

Columbia University Academic Commons

2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics October 16-19, 2011, New Paltz, NYSPECTRAL VS. SPECTRO-TEMPORAL FEATURESFOR ACOUSTIC EVENT DETECTIONCourtenay V. Cotton and Daniel P. W. EllisLabROSA, Dept. of Electrical EngineeringColumbia University1300 S. W. Mudd, 500 West 120th Street,New York, NY 10027, USA{cvcotton,dpwe}@ee.columbia.eduABSTRACTAutomatic detection of different types of acoustic events is an inter-esting problem in soundtrack processing. Typical approaches to theproblem use short-term spectral features to describe the audio sig-nal, with additional modeling on top to take temporal context intoaccount. We propose an approach to detecting and modeling acous-tic events that directly describes temporal context, using convolutivenon-negative matrix factorization (NMF). NMF is useful for findingparts-based decompositions of data; here it is used to discover a setof spectro-temporal patch bases that best describe the data, withthe patches corresponding to event-like structures. We derive fea-tures from the activations of these patch bases, and perform eventdetection on a database consisting of 16 classes of meeting-roomacoustic events. We compare our approach with a baseline usingstandard short-term mel frequency cepstral coefficient (MFCC) fea-tures. We demonstrate that the event-based system is more robust inthe presence of added noise than the MFCC-based system, and thata combination of the two systems performs even better than eitherindividually.Index Terms— Acoustic signal processing, acoustic event de-tection, acoustic event classification, non-negative matrix factoriza-tion1. INTRODUCTIONDetection and classification of acoustic events is important in anumber of applications. In particular, we are motivated to exam-ine this problem in the context of automatically finding and taggingevents that occur in an unconstrained environmental audio stream,such as the soundtrack of a YouTube video.Standard approaches to acoustic event detection utilize se-quences of feature vectors that each capture spectral informationover a very short time window, e.g., 30 ms. Additional modeling oftemporal structure, for example a hidden Markov model (HMM),may then be used to reincorporate the temporal context that hasbeen lost. In contrast, we are interested in taking advantage oftemporal context directly when detecting and comparing acousticevents, rather than trying to describe events only by the statistics oftheir component frames. Intuitively, we would think of an acousticevent as something that is defined by both its spectral energy and itscharacteristic temporal shape.This work was supported Eastman Kodak Corp., by the NationalGeospatial Intelligence Agency, and by the IARPA Aladdin program.In [1] we explore an approach to discovering and comparingacoustic events with their temporal context. We do this by extract-ing spectro-temporal patches around “transient” points that exhibit alarge increase in spectral energy. However, this method can be verysensitive to the parameters of the transient detector, and thereforehave poor robustness to environmental factors. The extracted fea-tures also suffer because there is no separation between an event’senergy and the background noise floor, which will also be capturedin the surrounding spectro-temporal patch.To address these problems, we present an algorithm based onnon-negative matrix factorization (NMF). NMF is an algorithm fordescribing data as the product of a set of bases and a set of acti-vations, both of which are non-negative; it was first described inits current form in [2]. It is useful for finding parts-based decom-positions of data; since all components are non-negative, each basiscontributes only additively to the whole. This promotes a solution inwhich high-energy foreground events and constant low-level back-ground energy may be described by different bases, and thereforeseparated in the feature representation.Most applications of NMF to audio processing decomposespectral magnitude frames (e.g., columns of a spectrogram), andhave each NMF bases consist of a single short time frame [3]. Sincewe are interested in learning bases that correspond to entire events,we use the convolutive formulation of NMF [4, 5]. In this version,bases consist of spectro-temporal patches of a number of spectralframes stacked together. The pattern described by each frame isthen activated as a whole to contribute to the reconstruction of thedata. Additionally we would like to ensure some level of sparsityin the activations of these bases. This is in order to encourage thebases to learn more foreground event patterns and fewer patternsthat mimic the background, which would be activated non-sparselyover large segments of the data.This NMF algorithm allows us both to locate transients in time,and to build a dictionary of event-patch codewords, within a singleoptimization framework, avoiding the separate transient detectionand patch clustering of our earlier approach.2. EVENT DETECTION WITH CONVOLUTIVE NMFOur algorithm for the detection of acoustic events is as follows. Wedownsample all data to 12 kHz, and take a standard STFT or spec-trogram of the entire signal, using 32 ms frames and 16 ms hops.We warp the frequency axis to mel-frequency as though we weretaking MFCC features, using 30 mel-frequency bands from 0 to 6kHz. We then concatenate all training data spectrograms and per-2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics October 16-19, 2011, New Paltz, NYform convolutive NMF across the entire set of training data, using20 basis patches which are each 32 frames wide (approximately 500ms). This produces a set of basis patches W and a set of basis acti-vations H . In our experiments we use the implementation of sparseconvolutive NMF described in [4], with KL divergence as the ob-jective and a sparsity parameter λ = 1. For testing, we then use thefixed set of learned patch bases and perform NMF to find the corre-sponding activation values for the test files. An example set of 20patch bases learned from training data is shown in figure 1. Figure 2shows an example event (’door knock’) and the activation patternsof the three bases that contributed the most energy in representingit. It also shows the event as reconstructed by NMF using the full setof bases. Note that the activations appear to occur somewhat beforethe onset of energy; this is because the patch activation is placed atthe left-hand edge of the 500 ms patch.We believe the NMF algorithm captures a relevant set of event-like patches, but we still need a reasonable way to represent the(continuous) activation patterns of these bases as discrete event-likefeatures. In order to do this, we use a sliding window of 1 s, withhops of 250 ms. Within this window, we summarize the local basisactivation pattern by taking the log of the maximum of each activa-tion dimension, producing a set of 20 features per window. Theseactivation values are pre-normalized such that each basis has a max-imum activation of 1 over the entire dataset.In order to perform event detection with these features, we use asimple HMM. The dataset we use for evaluation (described in sec-tion 5) labels specific time intervals as containing acoustic eventsof a given class. Our HMM consists of a single state for each ofthe 16 classes and a 17th state for the background. The observationmatrix is trained using the interval labels of the training data, withthe simple assumption that the observations for each event class canbe modeled as a single Gaussian. The transition matrix is trained onthe stream of labels, which in practice prohibits direct transitionsbetween two classes other than background (since the training datahas no overlap or adjacency between different classes). A streamof predicted labels is produced for each test file and scored as ex-plained in section 6. Finally, in order to produce a reliable eventstream, events shorter than 6 frames are removed.3. BASELINE MFCC EVENT DETECTORIn order to evaluate our algorithm, we compare it with a baselinewhich has a similar HMM structure, but that employs a standardset of short-term MFCC features. We extract MFCC features fromall data (25 ms frames with 10 ms hops, 40 mel-frequency bands)and retain 25 coefficients as features. We feed this into an HMM,trained in the same way described above. Again, we use a singlestate for each class, plus a background state, and observations aremodeled as a single Gaussian for each class. We generate a seriesof predicted labels at the frame level, as above.Because the MFCC frame spacing is much shorter than theNMF system’s frame spacing (10 ms vs. 250 ms), we need to post-process the predicted label stream to evaluate it fairly. We first usea median filter across 250 frames (2.5 sec), and then we remove anyremaining events that are less than 100 frames long. Both these pa-rameters were selected to optimize performance on clean test data.4. COMBINED SYSTEMSince we have built two event detectors based on different sets offeatures, we were interested to see if they could be combined infreq (Hz)442139834744421398347444213983474442139834740.0 0.5 0.0 0.5time (s)0.0 0.5 0.0 0.5 0.0 0.5Patch 1 Patch 2 Patch 3 Patch 4 Patch 5Patch 6 Patch 7 Patch 8 Patch 9 Patch 10Patch 11 Patch 12 Patch 13 Patch 14 Patch 15Patch 16 Patch 17 Patch 18 Patch 19 Patch 20Figure 1: 20 NMF patch bases learned on training data.a complementary way to produce better performance. We did thisby taking the two predicted event streams and requiring that theyboth agree on a predicted event label for some overlapping periodof time. If this is true, then we consider that predicted event to ex-tend to the entire period of time that either system has predictedthe event. This yields a combined prediction event stream that canbe evaluated alongside the two individual systems. Requiring bothcomponent systems to agree tends to reduce insertions while in-creasing deletions; however, it turns out that insertions were thedominant problem in noise, so this approach can be beneficial.5. DATABASEIn order to focus on the task of detecting specific acoustic eventsother than speech, we tested our approaches on the FBK-Irstdatabase of isolated meeting-room acoustic events [6]. This is adataset which was originally collected under the CHIL (Computerin the Human Interaction Loop) project. Event detection and clas-sification using data of this type has been extensively examined byTemko and others [7].The data we used consisted of 9 sessions, each around 7 min-utes long. Each session was recorded by multiple microphones,although we only use one channel in our experiments. This is be-cause we would like to develop algorithms that will also work onless controlled data, such as video soundtracks which would onlyhave one or two channels available.The database contains 16 semantic classes of acoustic events:door knock; door open; door slam; steps; chair moving; cough;paper wrapping; falling object; laugh; keyboard clicking; key jingle;spoon, cup jingle; phone ring; phone vibration; MIMIO pen buzz;and applause. Each session contains around 4 repetitions of each ofthe 16 classes of events, so there are around 36 examples of eachevent in the database. Approximately 50 repetitions per event classwere recorded.The data labels consist of short intervals that contain instances2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics October 16-19, 2011, New Paltz, NYfreq (Hz)Event: "door knock"44213983474patch#Activation values of 3 highest bases8318time (s)freq (Hz)Reconstruction using all bases5.584 6.064 6.544 7.024 7.504 7.98444213983474Figure 2: Example of a ’door knock’, the top 3 bases used to repre-sent it, and a reconstruction of the event.of the labeled sound. The events of different classes do not overlapwith each other.In our experiments, we use two folds of the data. In each fold,the data split is 6 training files and 3 test files. For efficiency rea-sons, in the NMF algorithm only 3 of the training files are used toactually learn the patch bases; the remaining 3 are added back inand used to train the HMM and learn the observation distributions.6. EXPERIMENT AND METRICWe are interested in examining the ability of our NMF algorithm todiscover acoustic events in the quiet meeting room environment inwhich this data was recorded, but also in the midst of noisy environ-ments. Our hope is that the additive nature of the NMF algorithmwill allow it to represent acoustic events that occur in noise moreconsistently than standard MFCC features, which will be corruptedby added noise.In order to test this idea, we performed all experiments withvarying levels of additive noise. The noise added was a short clipof background chatter and activity recorded in a cafeteria. For eachnoise level, the test data was left clean while this noise clip wasadded to the training data at the specified SNR.For evaluation we use the acoustic event error rate (AEER) thatwas used in CHIL evaluations for the event detection task, as de-scribed in [7]. This is defined as: AEER = 100(D + I + S)/N ,where D is the number of deletions, I the number of insertions, Sthe number of substitutions, and N the total number of events thatoccur in the ground truth labels.To evaluate a stream of predicted labels, it is broken into pre-10 15 20 25 30050100150200250300SNR (dB)Acoustic Event Error Rate (AEER%)Error Rate in Noise  MFCCNMFcombinedFigure 3: Acoustic event error rate results in noise.dicted events. A predicted event is any string of consecutive frameswith the same label. If the center of a predicted event (of the correctclass) falls anywhere within the true event’s label interval, then itis considered a correctly predicted event. Any predicted event thatdoes not fall within a true event of the same class is considered aninsertion or substitution; we count these errors together. Any trueevent that does not have a (correctly) predicted event fall within itis considered a deletion.7. RESULTSFigure 3 shows the performance of the three algorithms under theAEER metric. Table 1 breaks these results down into deletions,insertions, and again the overall AEER for each algorithm. Eachsystem has been tuned (by balancing insertions and deletions) tooptimize its performance at 30 dB SNR (nearly clean noise condi-tions). In clean conditions, the MFCC-based system performs muchbetter than the event-based one, and about the same as the combi-nation of the two.We then examine how each system breaks down in the presenceof added noise. All three systems produce deletions at a roughlycomparable rate as noise increases. The MFCC-based system pro-duces a large number of insertions in high noise conditions, whilethe NMF-based system largely does not (insertion and deletion ratesfor NMF stay roughly balanced). The combination system achieveseven better performance as the noise level increases. This is mostlytrue because it is limiting the number of insertions by requiring thatthe two systems agree on events.8. DISCUSSION AND CONCLUSIONSDespite the relative crudeness of our NMF-based features, wedemonstrate that these type of large-scale event features can be us-able in the detection and classification of acoustic events. Althoughour system is not competitive with a conventional short-frame-basedsystem in clean conditions, it proves useful when the test data iseven slightly more noisy than the training data. Features based onNMF basis activations seem to be fairly robust under moderate noise2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics October 16-19, 2011, New Paltz, NYSNR (dB)10 15 20 25 30DeletionsMFCC 41 28 17 13 13NMF 55 40 32 26 22Combined 54 41 27 21 18InsertionsMFCC 133 105 91 29 6NMF 43 50 29 22 18Combined 33 32 15 9 3AEERMFCC 282 216 175 68 30NMF 158 147 100 77 64Combined 140 118 69 48 34Table 1: Average number of deletions and insertions contributing toAEER.conditions (i.e. both systems using NMF features do not degrademuch between 30 and 20 dB SNR). The MFCC-based system, onthe other hand, performs much more poorly under moderate noiseconditions. Presumably this is because the MFCC features are be-ing corrupted by background noise while the NMF-based system isallowing prominent events to be represented by the same bases asthey would have in the clean test data. This would therefore yieldfeature descriptions that are theoretically more constant as the back-ground noise increases.Since our interest in event detection extends to varied types ofdata and recording conditions, it is important for an algorithm tobe able to detect similar events that occur in the midst of differenttypes of noise. Event modeling based on convolutive NMF basesseems promising for developing noise-robust of the type necessaryto detect acoustic events in all types of unconstrained videos andother audio data.9. REFERENCES[1] C. Cotton, D. Ellis, and A. Loui, “Soundtrack classification bytransient events,” in Proc. IEEE ICASSP, Prague, 2011, p. (toappear).[2] D. Lee and H. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 204, pp. 788–791,1999.[3] P. Smaragdis and J. Brown, “Non-negative matrix factorizationfor polyphonic music transcription,” in Proc. IEEE WASPAA,Mohonk, 2003, pp. 177–180.[4] P. D. O’Grady and B. A. Pearlmutter, “Convolutive non-negative matrix factorisation with a sparseness constraint,” inProc. IEEE MLSP, Maynooth, Sept. 2006, pp. 427–432.[5] P. Smaragdis, “Convolutive speech bases and their applica-tion to supervised speech separation,” IEEE Tr. Audio, Speech,Lang. Process., vol. 15, no. 1, pp. 1–12, 2007.[6] CHIL, “FBK-Irst database of isolated meeting-room acousticevents,” http://catalog.elra.info/product info.php?products id=1093, 2008.[7] A. Temko, Acoustic Event Detection and Classification (Ph.D.Thesis). Barcelona, Spain: Department of Signal Theory andCommunications, Universitat Politecnica de Catalunya, 2007.

Spectral vs. spectro-temporal features for acoustic event detection

https://academiccommons.columbia.edu/doi/10.7916/D8PZ5K4M/download

Spectral vs. spectro-temporal features for acoustic event detection

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons