We propose a concept for evaluating signal transformations for music signals with respect to an individual hearing
deficit by using an auditory model. This deficit is simulated in the
model by changing specific model parameters. Our idea is extracting the musical attributes rhythm, pitch, loudness and timbre and
comparing the modified model output to the original one. While
rhythm, pitch, and loudness estimation are studied in previous
works the focus in this paper concentrates on timbre estimation.
Results are shown for the original auditory model and three models, each simulating a specific hearing loss

Friedrichs, Klaus

Weihs, Claus

Eldorado - Ressourcen aus und für Lehre, Studium und Forschung

SFB 823 Comparing timbre estimation using auditory models with and without hearing loss Discussion Paper  Klaus Friedrichs, Claus Weihs      Nr. 51/2012        COMPARING TIMBRE ESTIMATION USINGAUDITORY MODELS WITH AND WITHOUTHEARING LOSSKLAUS FRIEDRICHS AND CLAUS WEIHSAbstract. We propose a concept for evaluating signal transfor-mations for music signals with respect to an individual hearingdeficit by using an auditory model. This deficit is simulated in themodel by changing specific model parameters. Our idea is extract-ing the musical attributes rhythm, pitch, loudness and timbre andcomparing the modified model output to the original one. Whilerhythm, pitch, and loudness estimation are studied in previousworks the focus in this paper concentrates on timbre estimation.Results are shown for the original auditory model and three mod-els, each simulating a specific hearing loss.1. IntroductionFor fitting and tuning a hearing aid for an individual patient as wellas for fundamental research of hearing aid algorithms a method forautomatic assessment based on a specific hearing deficit is very valu-able. To take the knowledge of a specific hearing deficit into accountMeddis proposed implementing this deficit in a widely recognized com-puter model of the human auditory periphery (Meddis et al., 2009). Inthis paper we pursue this suggestion for evaluating arbitrary complexhearing aid algorithms for music signals. This approach is shown inFigure 1. Music signals are processed by an auditory model withouthearing loss and simultaneously by another model in which the simu-lated hearing deficit is implemented. Since the auditory model outputcan not be interpreted directly the musical information has to be ex-tracted by an auralization procedure. Hence the recognized musicalattributes of the two models can be compared and an evaluation ofsignal transformations is feasible.12 KLAUS FRIEDRICHS AND CLAUS WEIHSFigure 1. Assessment of Hearing Aid Algorithms for Music SignalsHere, auralization means a technique of decoding the auditory modeloutput into a representation which can be understood and evaluatedby humans. This can be an audio signal or in case of music the musicalattributes of the signal: rhythm, pitch, loudness and timbre. For sim-plified auditory models the auralization task can be solved by using agenuine model inversion (Feldbauer et al. 2005). Instead, in this studyan auralization approach for the model described in Meddis (2006)is presented. This model enables a more realistic implementation ofdifferent types of hearing impairments (Meddis et al., 2009). Since forMeddis’ model an analytical inversion is not possible we have developeda statistical auralization approach for music signals using classificationand regression methods to estimate the musical attributes: rhythm(onsets), pitch (key tones), loudness and timbre.Meddis’ auditory model is a computational simulation model of thehuman auditory periphery. 40 auditory nerve fibres are simulated bya 40 channels filter bank. Each channel has an individual best fre-quency which defines which frequencies are stimulated the most. Thebest frequency is between 250 Hz for the first and 7500 Hz for the 40thchannel. In Figure 2 an exemplary output of the model can be seen.While the 40 channels are located on the vertical axis and the timeresponse on the horizontal axis, the color indicates the spiking activityper second. For the auditory model with hearing loss we consider thethree examples, called hearing dummies, which are described in Med-dis et al. (2009). These are modified models based on the auditorymodel described above. The first hearing dummy simulates a bilateralmoderate-severe sensory-neural hearing loss with normal middle earfunction. In the model this is implemented by retaining the channelwith the best frequency of 250 Hz only and by disabling the nonlin-ear path. The second hearing dummy simulates a bilateral, moderate,sensory-neural, sloping hearing loss with normal middle ear function.TIMBRE ESTIMATION WITH AND WITHOUT HEARING LOSS 3For the model this means that the endocochlear potential is reducedfrom -0.1 V to -0.09 V and the channels with best frequencies above2700 Hz are disabled. The third hearing dummy is a moderate bi-lateral, sensory-neural, ski-slope loss with normal middle ear functionbut no detectable acoustic reflex. This is implemented by disabling allchannels with best frequencies above 1800 Hz.Figure 2. Exemplary Output of Meddis Auditory Model (Normal Hearing)2. Auralization approachFirst the auditory model output has to be separated into the singletones by using tone onset detection. Subsequently for each tone thekey tone frequency, the loudness and the timbre are estimated usingstatistical learning procedures. During training mode the first step isskipped and instead all onset times are known.While onset detection and frequency and loudness estimation aredealt in our previous works the focus of this paper is timbre estima-tion. In Bauer et al. (2012) a tone onset detection using Meddis’ audi-tory model is proposed and compared to another approach, which usesthe original signal instead. It is shown that both used representationsperform altogether roughly equal. In Weihs et al. (2012) an approachfor frequency detection based on Meddis’ model using classificationmethods is introduced which solves the problem almost error free. Ad-ditionally, a method for loudness estimation for each partial tone usingregression methods is presented. While in that work each partial tone isestimated separately, for the improved approach described in this paperwe just need the key tone frequency and the loudness of the complete4 KLAUS FRIEDRICHS AND CLAUS WEIHStone, since we want to estimate also timbre. This makes these othertasks even somewhat easier.3. Timbre estimationTimbre is a combination of all acoustical attributes by which two mu-sical tones which are identical in pitch, loudness and length can be dis-tinguished (Emiroglu et al., 2007). While there are many approachesfor mathematical representations for timbre many of them are highlycontroversial. Thus, an objective definition of timbre appears to beproblematic. Therefore, we simplify timbre definition: Here, timbredistiguishes the tones of different instruments identical in pitch, loud-ness and length. By this we can define timbre estimation as a classifica-tion task, which can also be compared with human auditory perception.3.1. Experimental Design. We use randomly generated tone sequenceswith known onset times which contain tones of two different musicalinstruments from RWC data base (Goto et al., 2003). The classifica-tion task is to identify for each tone which instrument is playing. Weconsidered three classification tasks with different level of difficulty:piano versus clarinet, clarinet versus trumpet and piano versus guitar.While piano and clarinet can be distinguished relatively easy, the clas-sification task for piano versus guitar is even for humans a very hardchallenge, at least for the tones used in this study. For each classifica-tion task 10 randomly generated tone sequences are used. Each tonesequence consists of 20 tones, thus altogether we have 200 observationsper experiment. All tones have the same duration of 0.5 seconds whilethe sound itensities and the pitches are randomly chosen using uniformdistributions. The sound intensities have a range of [70,90] in MIDI-coding. The pitch range is dependent on the common pitch range ofthe respective instruments. This means [466, 4187] Hz for piano vs.clarinet, [523, 3730] Hz for clarinet vs. trumpet and [261, 2093] Hz forpiano vs. guitar. For each tone one of the two instruments is randomlychosen.All tone sequences are separately processed through Meddis’ audi-tory model respectively each of the three hearing dummies describedin Section 1. Since past observations have a significant impact on theauditory model output the same tones at different locations often pro-duce significant different model outputs. Figure 3 shows an exemplaryTIMBRE ESTIMATION WITH AND WITHOUT HEARING LOSS 5output of a tone sequence piano vs. clarinet using the auditory modelwithout hearing loss.Figure 3. Exemplary Auditory Output of Piano vs. Clarinet (Normal Hearing)3.2. Feature Generation. Contrary to other instrument recognitionexperiments which are based on the acoustic signal, here an additionalissue is combining the outputs of up to 40 channels. In this study a rel-atively simple approach is chosen by using only two channel combiningfeatures. All other features are generated for each channel indepen-dently. For further simplification features are computed over the wholetone. Possibly, better results can be achieved by windowing, thereby,e.g., distinguishing the attack time from the rest. The two combiningfeatures are the average firing rate over all channels and the varianceof the mean firing rates of the channels. While the first one shouldbe more connected to loudness, the second one should be more relatedto pitch. Both, however, might also include some information abouttimbre, e.g. the amount of high frequency.From each channel output the following features are generated. Theyall originate from music information retrieval for analyzing acousticsignals with respect to timbre (Lartillot et al., 2007). Thus, we can notsay for sure that all of them are also an indicator for predicting timbreon the auditory model output.a) Shannon EntropyThe entropy H(X) is defined as:H(X) = −N∑i=1p(xi)log2p(xi),whereX is the Discrete Fourier Transform of a signal and p(xi)is the energy share of the i-th frequency component. Shannon’s6 KLAUS FRIEDRICHS AND CLAUS WEIHSentropy is a measure of randomness in an acoustic signal andoften proposed as a measure for music complexity (Madsen etal., 2006). By calculating the entropy of each channel output itcan be estimated if the firing activity is just random or if it isstimulated by certain frequencies.b) Zero-cross (Here: “One-cross”)Zero-cross is an indicator for noisiness of an acoustic signal bycounting how often it changes the sign. Since the channels’ firingactivities are of cause always positive, we count how often eachchannel output crosses the 1-value, which might also indicate thenoisiness to some degree.c) Roll-off 85Roll-off 85 is defined as the minimal value R such thatR∑i=1xi ≥ 0.85N∑i=1xi,where xi is the amplitude of the i-th frequency component. Itestimates the amount of high frequency in an acoustic signal bycalculating the frequency below which 85% of the total energyis contained. Since each channel can be considered simply as abandpass filter, this feature might be very noisy for the auditorymodel output.d) BrightnessThis is another feature estimating the amount of high fre-quency in a signal. It calculates the amount of energy above1500 Hz with respect to the total energy:N∑i=f1500xi/N∑i=1xi,where xf1500 is the 1500 Hz frequency component. This mea-sure should be discussed in the same way als Roll-off 85.e) IrregularityIrregularity measures the degree of variation of adjoining par-tials in a tone:N∑k=1(ak − ak+1)2/N∑k=1a2k,where ak is the amplitude of the k-th partial and aN+1 is sup-posed to be zero. Since the ratio of adjoining partial tones canalso be seen very clearly in the auditory model output (Weihs etal., 2012), Irregularity should also indicate timbre in the auditorymodel output.TIMBRE ESTIMATION WITH AND WITHOUT HEARING LOSS 7f) Mel-Frequency Cepstral Coefficients (MFCCs)MFCCs describe the spectral shape of the sound (Davis et al.,1980). For this study the first 13 cepstral coefficients are takenas features from each channel output. However, like Roll-off 85and Brightness, MFCCs might be very noisy due to the bandpasscharacteristic of the channels.Altogether this means that we consider 2+ 40 ∗ 18 = 722 features incase of the original auditory model (without hearing loss).3.3. Results. Using the features described above classification meth-ods can be applied. In this study this is done by a Linear SupportVector Machine (SVM). Ten times repeated ten-fold cross validationis performed for getting significant results. These results are listed inTable 1.Table 1. Misclassification rates (10 times repeated 10-fold cross validation):Piano vs. Clarinet Clarinet vs. Trumpet Piano vs. GuitarNormal Hearing 0.0% 0.7% 4.2%Hearing Dummy 1 31.4% 32.0% 31.5%Hearing Dummy 2 0.6% 2.4% 6.0%Hearing Dummy 3 2.0% 6.2% 13.4%As expected the smallest classification error appears for the modelwithout hearing loss and the worst one for the first hearing dummywhich simulates a very strong hearing loss. Since the second hearingdummy scores better than the third one it can be supposed that a re-duced endocochlear potential does not have a high impact on timbreestimation. Furthermore, the difficulties of the level of the three clas-sification tasks is consistent with informal listening tests for all modelsexcept for the first hearing dummy, for which all tasks remarkably havenearly the same bad results.4. ConclusionIn this paper an approach for timbre estimation respectively musicinstrument recognition using an auditory model is proposed. The uti-lized features, which are almost all originally developed for analyzingacoustic signals instead of auditory model outputs, seem already to besufficient to produce satisfactory results. Nevertheless, we suppose thatresults could even be better with improved features which give moreregard to the auditory model output. Additionally, in future studies we8 KLAUS FRIEDRICHS AND CLAUS WEIHSwill optimize the classification by conducting feature selection and test-ing different classification methods. While subsequently, a comparisonwith state of the art instrument recognition systems could be very in-teresting, our primary intention is to generate a method for automaticassessment for hearing aid algorithms. Therefore, the classifier doesnot need to be perfect as long as the results are consistent with thehuman auditory perception. However, this has to be proven by a lis-tening test which measures the degree of correlation between statisticallearning and human perception. Finally, our purpose is combining thetimbre estimation with our previous studies about onset detection andpitch and loudness estimation to an overall measure, thereby gaining amethod for evaluating hearing aid algorithms for music signals.acknowledgementThis work was supported by the Collaborative Research Center "Sta-tistical modeling of nonlinear dynamic processes" (SFB 823) of theGerman Research Foundation (DFG).ReferencesN. Bauer, K. Friedrichs, D. Kirchhoff, J. Schiffner and C. Weihs (2012):"Tone onset detection using an auditory model", SFB 823 DiscussionPaper 50/12, TU Dortmund.S.B. Davis und P. Mermelstein (1980): "Comparison of ParametricRepresentations for Mono-syllabic Word Recognition in Continu-ously Spoken Sentences", IEEE Transactions on Acoustics, Speech,and Signal Processing 28/4, 357-366.S. Emiroglu, B. Kollmeier (2007): "Timbre discrimination in normal-hearing and hearing-impaired listeners under different noise condi-tions", Brain Res.C. Feldbauer, G. Kubin and W.B. Kleijn (2005): "AnthropomorphicCoding of Speech and Audio: A Model Inversion Approach", inEURASIP Journal on Applied Signal Processing, Volume 2005.M. Goto, H. Hashiguch, T. Nishimura, and R. Oka (2003): "RWCMusic Database: Music Genre Database and Musical InstrumentSound Database", Proceedings of the 4th International Conferenceon Music Information Retrieval (ISMIR 2003), 229-230.TIMBRE ESTIMATION WITH AND WITHOUT HEARING LOSS 9O. Lartillot, P. Toiviainen (2007): "MIR in Matlab (II): A toolbox formusical feature extraction from audio", in International Conferenceon Music Information Retrieval.S. T. Madsen, G. and Widmer (2006): "Music complexity measurespredicting the listening experience", in Proceedings of the 9th Inter-national Conference on Music Perception and Cognition, Bologna.R. Meddis (2006): "Auditory-nerve first-spike latency and auditoryabsolute threshold: A computer model", Journal of the AcousticalSociety of America 119, 406-417.R. Meddis, W. Lecluyse, C.W. Tan, and M.R. Panda (2009): "Be-yond the audiogram: identifying and modelling patterns of hearingdeficits", in The Neurophysiological Bases of Auditory Perception,Proc. of the International Symposium on Hearing.C. Weihs, K. Friedrichs und B. Bischl (2012): "Statistics for hearingaids: Auralization", in J. Pociecha, R. Decker (Hrsg.): Data AnalysisMethods and its Applications, 183-196.Chair of Computational Statistics, TU DortmundE-mail address: {friedrichs, weihs} @statistik.tu-dortmund.de   

Comparing timbre estimation using auditory models with and without hearing loss

https://eldorado.tu-dortmund.de/bitstream/2003/29768/1/DP_5112_SFB823_Friedrichs_Weihs.pdf

Comparing timbre estimation using auditory models with and without hearing loss

Abstract

Similar works

Full text

Available Versions

Eldorado - Ressourcen aus und für Lehre, Studium und Forschung