Arthur Flexer

Martin Gasser

Thomas Grill

English

ZENODO

ON COMPUTING MORPHOLOGICAL SIMILARITY OF AUDIO SIGNALSMartin GasserAustrian Research Institutefor Artificial Intelligencemartin.gasser@ofai.atArthur FlexerAustrian Research Institutefor Artificial Intelligencearthur.flexer@ofai.atThomas GrillAustrian Research Institutefor Artificial Intelligencethomas.grill@ofai.atABSTRACTMost methods to compute content-based similarity be-tween audio samples are based on descriptors represent-ing the spectral envelope or the texture of the audio signalonly. This paper describes an approach based on (i) theextraction of spectro-temporal profiles from audio and (ii)non-linear alignment of the profiles to calculate a distancemeasure.1. INTRODUCTIONMany real-world applications in the field of audio similar-ity would greatly benefit from an approach that explicitlymodels the temporal evolution of certain aspects of the sig-nal and derives similarity values from this high-level de-scription of the signal. Apart from being able to calculatesimilarities between signals that would be totally indistin-guishable under a bag-of-frames-type [1, 2, 3] approach,such a method would also facilitate the classification ofaudio signals according to morphological descriptions.Schaeffer [4, 5] proposed the description of sound ob-jects according to a set of criteria, which include descrip-tions of the sound matter, the sound shape, and variationcriteria. The notion of such sound-shapes appears in ev-eryday applications e.g. with so-called up- and downliftingsounds as used in fields of sound-design and jingle pro-duction. Recently, some of those criteria have been im-plemented [6, 7, 8]. These works focus on classificationof audio signals based on simple categories like “ascend-ing” or “descending” based on standard audio features (e.g.loudness, spectral centroid, pitch).The contribution of this paper is two-fold: First, we showsome deficiencies of a standard descriptor (the spectralcentroid [9]) for modeling spectral evolution over time. Inparticular, the spectral centroid is very sensitive to back-ground noise if the noise level exceeds the level of thesignal, whereas humans can easily spot spectral move-ment, even in the presence of strong (stationary) noise.We present a noise-robust and efficient approach to modelspectro-temporal evolution based on tracking window-to-window cross-correlations of Constant-Q magnitude spec-tra over time.Copyright: c©2011 Martin Gasser et al. This is an open-access article distributedunder the terms of the Creative Commons Attribution 3.0 Unported License, whichpermits unrestricted use, distribution, and reproduction in any medium, providedthe original author and source are credited.(a) Log spectrogram of test signal (b) Log spectrogram of noisy signal(c) Spectral centroid of test signal (d) Spectral centroid of noisy signal(e) Shifts from cross correlation (f) Shifts from cross correlation(g) Integrated shifts (h) Integrated shiftsFigure 1. Profiles calculated from a clean test signal and anoisy signal (SNRdb = 1)We also propose to calculate similarities between spec-tral evolution trajectories reconstructed from the aforemen-tioned descriptor by using Dynamic TimeWarping [16] andbriefly evaluate our approach on a set of synthetic audiosamples.2. FEATURE EXTRACTIONA natural candidate for capturing the spectral evolution ofa signal is the spectral centroid [9]. The spectral centroidis calculated as a weighted mean of the frequencies presentin the signal. Thus, the lower the signal-to-noise ratio of asignal is, the less meaningful is the spectral centroid value.Figure 1(d) shows a typical result of a spectral centroidcalculation on a noisy signal.Another approach to capturing the spectral movement ofsignals is F0-tracking [10]. Since we want to be able tocapture spectral evolutions of highly inharmonic sounds aswell, F0-tracking approaches have been rejected in the firstplace.t t+1Shiftt+2Shiftt+3ShiftfrequencytimeFigure 2. Shifted log-scaled magnitude spectraInstead of trying to find instantaneous descriptors likeF0 or the spectral centroid, we take a different route:Based on the observation that successive logarithmicallyscaled magnitude spectra calculated from a coherent spec-tral movement are cross-correlated (see figure 2), we cross-correlated Constant-Q magnitude spectra, from the cross-correlations we derive the hypothetical shift values to opti-mally align two magnitude spectra, and finally we calculatethe final trajectory by cumulative summing of the shift val-ues. Figure 1 demonstrates the weakness of spectral cen-troid under noisy conditions (Fig. 1(c) and Fig. 1(d)) andhow we derive the final spectral profile from the shift val-ues (Fig. 1(e)–Fig. 1(h)).To calculate the logarithmically scaled short-time spectraof the input signal, we apply a Constant-Q transform [11](CQT) and use the absolute value of the result.For the Constant-Q analysis, we use a frequency reso-lution of 32 bins per octave, with a minimum frequencyof 50 Hz and a maximum frequency corresponding to theNyquist frequency (the sampling rate of the signals being44.1 kHz). The hop size for the analysis is set to 1.451 msand the time domain basis functions of the CQT are taperedwith Hann windows. The window size resulting from theaforementioned parameters is 371.5 ms. We use the CQTalgorithm as described by Brown and Puckette [12] anddiscard 98% of the coefficients in the spectral kernel SKby thresholding it with 0.01 ∗max(SK ), taking advantageof sparse matrix multiplications.2.1 Mathematical backgroundAccording to the correlation theorem [13], the cross-correlation r of two signals can be calculated efficientlyin the Fourier domain as:Ga = F{ga} , Gb = F{gb}R =GaG∗b|GaG∗b |, r = F−1{R}where Ga and Gb are the Fourier transforms of the sig-Estimated peakMeasured peaky α y βy 0 y p-1 10 pAmplitudeFrequencyFigure 3. Fitting a parabola to the correlation functionnals ga and gb, respectively, and G∗b denotes the complexconjugate ofGb.By peak-picking the cross-correlation function, the shiftfactors can be recovered.One problem that we encountered was that the frame-to-frame shift factors were potentially smaller than the resolu-tion of the spectral analysis. Therefore, the resulting shiftfactors were much too coarse and in many cases, the corre-lation function always peaked at lag 0. Our solution to theproblem was to use correlation interpolation [14] by fittinga parabola to the peak and the two surrounding points inthe cross-correlation function (see figure 3).p =12yα − yβyα − 2yo + yβ (1)By adding the value p to the index of the peak index ofthe cross correlation function, the final shifting factor for apair of magnitude spectra is computed.3. SIMILARITY COMPUTATIONWe assume that similar audio signals show a similar spec-tral evolution in time. In order to measure the similaritybetween two signals, we (i) align a low dimensional repre-sentation of the spectral evolution of the signals and we (ii)derive a similarity measure from the quality of the optimalalignment.A well-researched method that solves the problem ofnon-linear alignment of time series is Dynamic TimeWarping (DTW) [15, 16]. DTW is a dynamic program-ming algorithm, that is, it calculates a matrix of partial op-timal solutions to sub-problems and finds the optimal solu-tion of the problem in a back-tracking manner. The resultof DTW is a list of matchings (ai, bj)k, where a match intime step k relates the elements ai and bj from the timeseries a and b (ai and bj are the elements at positions i andj in the time series a and b, respectively). By summingand normalizing the distances between matched elements,a distance measure for the time series is derived.Figure 4 shows a simple example of DTW (the first col-umn shows the distance matrix and the optimal path, thesecond column shows the resulting point-to-point align-(a) Optimal path through distance matrix (b) Alignment of time series(c) Optimal path through distance matrix (d) Alignment of time seriesFigure 4. DTW examplement). Whereas in figure 4(a)/figure 4(b) the two time se-ries are identical, figure 4(c)/figure 4(d) shows two slightlydifferent series, yielding a non-zero distance.In order to be independent of translations along the y axis,we used a variant of DTW called Derivative Dynamic TimeWarping [17], that is, instead of directly matching valuesof time series, we matched their first order differences.Our algorithm for calculating morphological similaritiescan be summed up as follows:1. Calculate the Constant-Q magnitude spectrum of thesignals2. For each pair of spectral frames, calculate the cross-correlation, pick the peaks and derive the shift value3. Integrate shifts (compute cumulative sums)4. To account for different signal lengths and to re-duce computation time, resample the integrals of thespectral evolution trajectories to length n (we usedn = 100)5. Align the trajectories with DTW and compute thedistances4. EVALUATIONTo evaluate our approach, we generated short audio sam-ples consisting of a sinusoidal oscillator following a set ofpitch envelopes {ei(x)|i ∈ [0, 17]}modeling up-down anddown-up movements.tα(x) ={sin( xα · pi2 ), x ≤ αsin(pi2 +x−α1−α · pi2 ), x > α(2)ei(x) ={t(i+1)/10, 0 ≤ i < 91− t(i−8)/10, 9 ≤ i ≤ 17(3)See figure 5 for a plot of the resulting 18 functions.As a proof of concept, we calculated DTW distances di-rectly on the pitch envelopes without synthesizing audioand extracting the trajectories in the first place. Figure 6demonstrates the theoretical applicability of the DTW ap-proach to our problem: For each pair of pitch envelopes,the DTW distance is mapped to a color (dark correspondsto low, light to high values). As can be seen, the distancesvary correspondingly to the choice of the α parameter inequation 2.From the profiles in figure 5, we generated short au-dio samples by applying a bank of resonators with time-varying filter frequencies to an excitation signal consist-ing of white noise. The resonator frequencies were set to(env ∗ 1.1, env ∗ 1.6, env ∗ 2.1, env ∗ 2.2), where env isthe value of a prototype envelope. We also generated twosets of signals with added white noise (SNRdb = 6 and 1,respectively) and calculated their distances to the clean sig-nals. Figures 7(a)- 7(c) show that the distance measure isFigure 5. Pitch envelopes used for the evaluationFigure 6. DTW distance matrix calculated from the 18prototypesvery robust, even under very noisy conditions.5. CONCLUSIONS AND FUTUREWORKWe have presented an approach to calculating similaritybetween audio samples based on morphological criteria.We model the spectro-temporal evolution of signals bycalculating the pairwise interpolated cross-correlation be-tween successive log-scaled short-time magnitude spectraand integrating the resulting shift values to a trajectory. Tocalculate similarities, trajectories are aligned with Deriva-tive Dynamic Time Warping and a distance value is derivedfrom the resulting DTW path.In this paper, we have only considered spectral evolutionof signals; we plan on using the same methodology to cal-culate similarities from loudness functions as well.We also think that the method has potential for interactiveretrieval of audio samples, since it should be straightfor-ward for users to directly draw the desired spectral evolu-tion trajectory. We will investigate this possibility in futureresearch.6. ACKNOWLEDGMENTSThis research is supported by the Austrian Science Fund(FWF, P21247, and Z159) and the Vienna Science andTechnology Fund (WWTF, project “Audiominer”)7. REFERENCES[1] M. Mandel and D. Ellis, “Song-level features and sup-port vector machines for music classification,” in Pro-ceedings of the 6th International Conference on Mu-sic Information Retrieval, London, United Kingdom,2005.[2] E. Pampalk, “Computational models of music sim-ilarity and their application in music informationretrieval,” Ph.D. dissertation, Vienna University ofTechnology, Vienna, Austria, March 2006. [On-line]. Available: http://www.ofai.at/∼elias.pampalk/publications/pampalk06thesis.pdf[3] J.-J. Aucouturier, B. Defreville, and F. Pachet, “Thebag-of-frames approach to audio pattern recognition:A sufficient model for urban soundscapes but not forpolyphonic music,” Journal of the Acoustical Societyof America, vol. 122, no. 2, pp. 881–891, 2007.[4] P. Schaeffer, Traite´ des objets musicaux. Editions duSeuil, Paris, 1966.[5] M. Chion, Guide des objets sonores, Pierre Schaefferet la recherche musicale. Ina-GRM/Buchet-Chastel,Paris, 1983.[6] J. Ricard and P. Herrera, “Morphological sounddescription computational model and usability eval-uation,” in AES 116th Convention, 2004. [Online].Available: files/publications/AES116-jricard.pdf[7] J. Bloit, N. Rasamimanana, and F. Bevilacqua,“Modeling and segmentation of audio descriptorprofiles with segmental models,” Pattern Recogn.Lett., vol. 31, pp. 1507–1513, September 2010.[Online]. Available: http://dx.doi.org/10.1016/j.patrec.2009.11.003[8] G. Peeters and E. Deruty, “Sound indexing using mor-phological description,” IEEE Transactions on Audio,Speech & Language Processing, vol. 18, no. 3, pp.675–687, 2010.[9] G. Peeters, “A large set of audio features forsound description (similarity and classification) in theCUIDADO project,” IRCAM, Tech. Rep., 2004.[10] A. de Cheveigne and H. Kawahara, “Yin, a funda-mental frequency estimator for speech and music,”The Journal of the Acoustical Society of America, vol.(a) Reference profiles to themselves(b) Reference profiles to noisy version (SNRdb = 6)(c) Reference profiles to noisy version (SNRdb = 1)Figure 7. Distance matrices of reference profiles to noisyversions111, no. 4, pp. 1917–1930, 2002. [Online]. Available:http://link.aip.org/link/?JAS/111/1917/1[11] J. C. Brown, “Calculation of a constant q spectral trans-form,” Journal of the Acoustical Society of America,vol. 89, no. 1, pp. 425–434, 1991.[12] J. Brown and M. Puckette, “An efficient algorithm forthe calculation of a constant q transform,” Journal ofthe Acoustical Society of America, vol. 92, no. 5, pp.2698–2701, 1992.[13] A. V. Oppenheim and R. W. Schafer, Discrete-TimeSignal Processing, 3rd ed. Upper Saddle River, NJ,USA: Prentice Hall Press, 2009.[14] Q. Tian and M. N. Huhns, “Algorithms for subpixelregistration,” Computer Vision, Graphics, and ImageProcessing, vol. 35, no. 2, pp. 220–233, 1986.[15] L. Rabiner and B.-H. Juang, Fundamentals of speechrecognition. Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 1993.[16] D. J. Berndt and J. Clifford, “Finding patterns intime series: a dynamic programming approach,”pp. 229–248, 1996. [Online]. Available: http://portal.acm.org/citation.cfm?id=257938.257961[17] E. J. Keogh and M. J. Pazzani, “Derivative Dy-namic Time Warping,” in In First SIAM InternationalConference on Data Mining (SDM’2001, 2001. [On-line]. Available: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.23.6686

On Computing Morphological Similarity of Audio Signals

https://zenodo.org/record/849968/files/smc_2011_053.pdf

On Computing Morphological Similarity of Audio Signals

Abstract

Similar works

Full text

Available Versions

ZENODO