Colloque avec actes et comité de lecture. internationale.International audienceWe study in this work the Missing Data Recognition (MDR) framework applied to a large vocabulary continuous speech recognition (LVCSR) task with cepstral models when the speech signal is corrupted by musical noise. We do not propose a full system that solves this difficult problem, but we rather present some of the issues involved and study some possible solutions to them. We focus in this work on the issues concerning the application of masks to cepstral models. We further identify possible errors and study how some of them affect the performances of the system

Cerisara, Christophe

INRIA a CCSD electronic archive server

HAL Id: inria-00107648https://hal.inria.fr/inria-00107648Submitted on 19 Oct 2006HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Towards Missing Data Recognition with CepstralFeaturesChristophe CerisaraTo cite this version:Christophe Cerisara. Towards Missing Data Recognition with Cepstral Features. 8th European Con-ference on Speech Communication and Technology - EUROSPEECH’03, Sep 2003, Geneva, Switzer-land, 4 p. ￿inria-00107648￿Towards Missing Data Recognition with Cepstral FeaturesChristophe CerisaraLORIA - UMR 750354500 Vandoeuvre-les-Nancy - FRANCEcerisara@loria.frAbstractWe study in this work the Missing Data Recognition(MDR) framework applied to a large vocabulary continuousspeech recognition (LVCSR) task with cepstral models whenthe speech signal is corrupted by musical noise. We do notpropose a full system that solves this difficult problem, but werather present some of the issues involved and study some pos-sible solutions to them. We focus in this work on the issues con-cerning the application of masks to cepstral models. We furtheridentify possible errors and study how some of them affect theperformances of the system.1. IntroductionThe two main issues to solve when using missing data recogni-tion are:  To find the masks that represent the corrupted or unreli-able spectro-temporal regions;  To compute the likelihoods that the speech models havegenerated the acoustic observations, given the fact thatsome spectro-temporal regions are masked.In this work, we will not consider the former problem atall. We rather focus on the latter issue. However, the fact thatthe masks must segregate reliable and corrupted coefficients im-plies that a frequency-localized noise must also be localized inthe feature domain. This is certainly not the case for cepstral co-efficients, and this is why most of the previous work in the fieldof MDR has been realized with “spectrographic-like” models[1, 2]. On the other hand, it is well-known that MFCCs areless sensitive to amplitude mismatch than spectrographic coef-ficients, and are also more robust to noise. We study here thepossible use of MDR techniques with MFCC models. Our othermain contributions concern the use of MDR in a large vocabu-lary continuous speech recognition task where the speech signalis contaminated by musical noise.2. Missing data recognition with cepstralmodels2.1. General considerationsThere are two main solutions to solve the second issue men-tioned in section 1:  Data imputation: The system tries to estimate somehowthe unknown value of the “corrupted” observations;  Marginalization: The recognizer marginalizes the likeli-hoods over the unknown observations, which is an opti-mal decision given the masks in the Bayes’ sense [3].The best results are reported in the literature for themarginalization process [1], but this technique can only be ap-plied to spectral features at a reasonable complexity. On theother hand, data imputation can more easily be applied to cep-stral models, as once the noisy observations have been replacedby some estimated uncorrupted values, further processing ofthese vectors (such as a discrete cosine transform) can be re-alized and recognition in the cepstral domain can thus be per-formed. This is why we have chosen to describe our approachwithin the data imputation framework.A few other works such as [4, 5] deals with the use ofMFCC models within the MDR framework. We extend heresuch studies by (i) proposing and testing a new system that usesMFCC models in a LVCSR task corrupted by background mu-sic and (ii) identifying and studying specific recognition errorsthat may result from the use of cepstral features.2.2. Class-conditional imputationIn the following, the upper indexrepresents log-spectral vec-tors. Whenis omitted, MFCC vectors are considered. Let be the observation at time  . We decompose    	 	 (1)where  	 and  	 respectively represent the masked and un-masked coefficients of .Class-conditional imputation [2] (or mean imputation) sim-ply consists to impute   	    	 where    	 represents themasked coefficients of the mean of the Gaussian model alignedwith . This technique assumes that   	 is independent of 	 . It is reported in [1] that using the correlation betweenthe masked and unmasked coefficients improves the results, butat the cost of a much higher complexity. We thus assume inthe next sections that  	 and  	 are independent and wepropose some improvements of this method.2.3. Vocabulary sizeVery good results are reported for small-size vocabulary MDRtasks [6]. A few other works deal with medium-size vocabulary[3, 2], but the results seem to be less good. To our knowledge,there is no result reported for large vocabulary, and we proposehere to study some MDR techniques in a LVCSR task.2.4. Baseline algorithmWe assume in the next experiments that the masks are given.Such masks are usually called oracle masks [2]. This approachdoes not give a clear understanding of how MDR might per-form in real recognition tasks, but it is very useful to isolate theproblems when we study a new approach in the field of MDR.We should then keep in mind that in a more realistic use of oursystem, these masks will have to be estimated and the perfor-mances will probably decrease.The algorithm we use is based on the classical Viterbi algo-rithm, modified as follows:  For a given alignment between the (static) cepstral frameand the (static) cepstral clean speech distribution   	, both vectorsand    are transformed intothe log-power-spectral domain: For, this simply con-sists to compute   whereis the discretecosine matrix. For    , we know that:            	 (2)Therefore,            .    	 is replaced by:  	 !    	 if   	#"    	  	 if   	#$    	&% (3)  The unmasked coefficients  	 are not modified: 	   	 (4)The imputed observation vector is now  ('   	 	*) (5)  This vector is then transformed back into the cepstral do-main:  +,   (6)  Dynamic cepstral parameters are usually computed asa linear combination of static parameters of successiveframes such as 1:- .  	 +/ . 10 /2	 0  3054 	76 .  6 4 	768/ .  69/2	(7)In the speech models, only- is known (and not the in-dividual terms that appear in the right-hand side of equa-tion 7). To impute- .  	 , we thus have to assume thatthe mask defined at time  is also valid at times  6:/ , 6 4 , ;0<4 and ;0 / . Therefore, the same coefficients of  6=/2	 , .  6 4 	 , . >0?4 	 and . >0 /2	 are maskedand we can decompose-     	  -   	   	-   	   	  (8)based on the mask defined at time  . Unbounded impu-tation can then be applied to this vector:-     	   -    	-  	   	  (9)The same principle is used to impute-@ .  	 .  The emission likelihood is computed with this modifiedobservation and the Viterbi algorithm repeats this proce-dure for every possible alignment.The complexity of this algorithm can be made minimal by stor-ing in memory a log-spectral version of the speech models andby using a front-end that directly computes the observations inthe log-spectral domain before the DCT.1Notation: A is omitted for static vectors but is kept in this derivationconcerning the dynamic parameters2.5. Improvement of the algorithmThe bounded imputation technique [2] is an extension of theimputation procedure presented in 2.2 and is used in the algo-rithm described in section 2.4. Its idea consists to replace theobservation  	 by the model’s mean log-power    	 if andonly if  	 "    	 . This is reasonable because the energyof the speech plus noise should be greater than the energy ofthe speech alone. We propose here to further extend this prin-ciple by considering that when  	 is close to    	 , then it isunlikely that the noise has a great influence on the recognitionaccuracy. Replacing these masked values by some approxima-tions may be worse than doing nothing. Therefore, we decideto replace  	 by    	 iff   	#"CB    	 , where B is estimatedon a development corpus.On the other hand, preliminary experiments suggested thateven when masking is really needed, it may actually be better toimpute an intermediate value between  	 and    	 . We thuspropose to replace  	 by D    	 rather than by    	 .To summarize, equation 3 becomes:  	 E D    	 if   	 "CB    	  	 if   	#$ B    	 % (10)The role of B is to reduce the number of coefficients effec-tively masked by adjusting the threshold used in the “bounded”paradigm, while the role of D is to “soften” the effect of mask-ing by finding a compromise between completely masking theobservation and not masking it at all.3. First experimental results3.1. Experimental setupExperiments are realized on the BREF80 evaluation task [7]. Itis a large vocabulary (20000 words) continuous speech recog-nition task in French, quite similar to the English Wall StreetJournal task. 40 french monophones are modeled with 3 statesleft-to-right HMMs. We have adapted the julius recognition en-gine [8] to handle missing data as explained above. This en-gine makes use of two passes with a words bigram in the firstpass and a trigram in the second pass. 39 MFCC parameters(included c0, delta and acceleration) are computed every 10ms. 24 filters are used in the log-spectral domain. 300 sen-tences pronounced by 20 speakers are used for testing and 2000sentences pronounced by 70 speakers for training. All the testsentences are “corrupted” by a background music, in that caseBach’s Chaccone. The oracle masks are computed by compar-ing the corrupted and clean log-spectral observations: when thedifference between them is greater than a given threshold, thenthe coefficient is masked.3.2. Bounded factorsThe first experiments reported here concern the estimation ofthe factor B described in 2.5. Figure 1 gives the word error ratein function of B on the 10 sentences of the development corpus.We can observe that, as supposed, some improvement maybe obtained by enforcing the “bounded” paradigm, i.e. byslightly increasing the masking / nomasking threshold. Notethat, due to the size of the development corpus, the differencein Word Error Rate (WER) is not significant. But as the sys-tems and corpus are exactly the same and only differ in B , westill choose B based on this experiment.For the next experiments, we set B  42FG4 . We then evaluatethe “optimal” value of D on the same development corpus. The5253545556575859600.8 1 1.2 1.4 1.6 1.8 2WERAlphaFigure 1: Influence of B on the WER (development corpus).WER is represented in figure 2 in function of D . Note that whenD   , the system behaves like a classical recognition enginewithout missing data, while when D  4 , the basic boundedmissing data algorithm is used without any modification.It is interesting to observe that the “optimal” value of D is  F  , which is far from the basic imputation scheme obtainedwith D  47F   . This supports the idea that it might be better tofind a good “compromise” between the missing data paradigmand the classical one, than to simply use one or the other. Thisis also a possible interpretation of the “soft mask” proceduredescribed in [1].485052545658600 0.2 0.4 0.6 0.8 1WERBetaFigure 2: Influence of D on the WER (development corpus).3.3. Number of masksThe next experiment is realized with B  47F 4 and D   F  onthe whole test set (300 sentences). Its role is to evaluate the in-fluence of the number of coefficients masked on the recognitionaccuracy.Figure 3 gives the word error rate in function of the numberof masked coefficients (this number is increased by decreasingthe threshold used to compute the oracle masks). This numberis computed after the test described in section 2.5 and representsthe ratio of imputed coefficients in all the Viterbi paths.As expected, when the number of masks increases toomuch, the WER increases dramatically. But it is surprising than636465666768697071720 1 2 3 4 5 6 7 8WERPercentage of coefficients maskedFigure 3: Influence of the number of masks on the WER (testcorpus).the optimal number of imputed coefficients is only about 3 %,which is very few. As discussed in [1], part of the explana-tion may be related to the difficulty to have clearly discriminantacoustic models in a LVCSR task. We investigate in next sec-tions other possible explanations.3.4. Possible causes of recognition errorsFor the baseline system (not MDR), the errors originate from:(i) The mismatch between the corrupted observations and theclean models.For MDR recognition, the errors originate from:(i) The mismatch between the corrupted observations that arenot masked and the clean models;(ii) The lost of acoustic information in the masked coefficients(or equivalently the mistakes realized when imputing the miss-ing coefficients).The more masks are used, the more errors (ii) and the lesserrors (i) occur. The difficulty of MDR consists to find a goodcompromise between both kinds of errors. We study separatelyeach kind of error in section 4.4. A study of MDR errors4.1. Study of the cepstral filtering effectOne of the issues in the case of cepstral parameterization is re-lated to the filtering effect of the DCT. Indeed, as the numberof cepstral parameters is usually not the same as the number offilterbanks in the log-spectral domain, some smoothing is in-troduced when transforming the imputed observations into thecepstral domain. This smoothing has two effects:  It modifies the imputed values, that are not any moreequal to the means of the models ;  It modifies the unmasked values.Both effects are not desired and may introduce new errors oftypes (i) and (ii). To test this issue, we have made some exper-iments by setting the number of filterbanks ( 	 ) equal to thedimension of cepstral vectors  4 . Figure 4 compares 	  4 with  	 +/ .We can note that the WER increases when  	  4 . thiscan be explained by the fact that the description of the observa-tion in that case is less precise than with 	 +/ . But moreimportantly, this shows that the smoothing effect introduced by64656667686970717273744 4.5 5 5.5 6 6.5 7 7.5 8WERPercentage of coefficients masked"NFILT=13""NFILT=24"Figure 4: Influence of the smoothing effect of the DCT on theWER.the DCT is not a problem and this validates the use of cepstralmodels in the framework of MDR.4.2. Study of each kind of errorsIn the next experiment, we try to remove all errors (i) in order toisolate errors (ii). To achieve this, just after the oracle masks arecomputed, the signal is processed by replacing every unmaskedlog-spectral coefficient by its uncorrupted (clean) value. Re-versely, we also remove all errors (ii) by replacing every maskedlog-spectral coefficient by its clean value, and then realizing aclassical (not MDR) recognition. This is equivalent at using anideal imputation method.Table 1 compares this system with the previous one and thebaseline when  F 4  of the coefficients are masked.Table 1: Respective influence of each kind of errors on WER.system WERbaseline 68.9 %MDR: errors (i) & (ii) 63.4 %MDR: only errors (ii) 52.4 %MDR: only errors (i) 44.5 %baseline without noise 36.7 %We can note that both types of errors strongly affect therecognition rate. Consequently, we can imagine the followingimprovements for the system:  The “oracle” masks are not that good, probably becausethe energy criterion on which they are based is not ade-quate. A better criterion would be to choose the masksthat have a strong influence on the recognition accuracy.The corresponding spectro-temporal regions may haveonly a small energy mismatch due to the noise, and re-versely other regions with an important mismatch maynot really influence the WER. Such masks would cer-tainly reduce errors (i).  Better imputation procedures are needed to reduce errors(ii). Previous results show that reducing this kinds oferrors may lead to an important improvement. Therefore,it might be worth to adapt solutions such as conditionalimputation to cepstral models.5. ConclusionsThe original contributions of the work presented here are thefollowing:  Proposal of a system to test MDR in a LVCSR task cor-rupted by background music;  Proposal and justification of a framework to use MDRwith cepstral models;  Extension of the “bounded imputation” procedure to (i)reduce the number of unneeded masks, and (ii) soften theeffect of masking by adjusting the compromise betweenthe MDR and classical frameworks;  Analysis of the errors realized by a MDR system andexperimental study of each type of error separately.We believe the MDR paradigm might be more widely usedif it becomes compliant with standard MFCC models. The mainobjective of this work is thus to create and strengthen a sanebasis for MDR with cepstral models. But there are still manyproblems to solve before such solutions can be used in real-lifesituations. The most important is to adapt or create a better im-putation procedure than the one presented here. Then, researchobjectives shall focus on the estimation of the masks.6. References[1] Cooke, M. and Green, P. and Josifovski, L. and Vizinho,A., “Robust automatic speech recognition with missingand unreliable acoustic data”, Speech Communication,Vol. 34, N. 3, June 2001.[2] Ramakrishnan, B. R., “Reconstruction of incompletespectrograms for robust speech recognition”, PhD. the-sis, Department of Electrical and Computer Engineering,Carnegie Mellon University, April 2000.[3] Morris, A. and Cooke, M. and Green, P., “Some solu-tions to the missing feature problem in data classification,with applications to noise robust ASR”, in proceedingsICASSP, Seattle, Washington, USA, 1998, p. 737–740.[4] Häkkinen, J. and Haverinen, H., “On the Use of Miss-ing Feature Theory with Cepstral Features”, CRAC Work-shop, Aalborg, Denmark, September 2001.[5] Renevey, P., “Speech recognition in noisy conditions us-ing missing feature approach”, PhD. thesis, École Poly-technique Fédérale de Lausanne, 2000.[6] Barker, J. and Cooke, M. and Ellis, D., “Decoding speechin the presence of other sound sources”, in proceedingsICSLP, Beijing, China, 2000.[7] Lamel, L. F., Gauvain, J. L. and Eskenazi, M., “BREF, alarge vocabulary spoken corpus for French”, in proceed-ings EUROSPEECH, 1991.[8] Lee, A., Kawahara, T. and Shikano, K., “Julius – anopen source real-time large vocabulary recognition en-gine”, in proceedings EUROSPEECH, 2001, p. 1691–1694, http://winnie.kuis.kyoto-u.ac.jp/dictation/.

Towards Missing Data Recognition with Cepstral Features

HAL Descartes

Hal-Diderot

https://hal.inria.fr/inria-00107648/file/A03-R-078.pdf

Towards Missing Data Recognition with Cepstral Features

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

HAL Descartes

Hal-Diderot