We propose a novel method for clustering allophones called Feature-Dependent Allophone Clustering (FD-AC) that determines feature-dependent HMM topology automatically. Existing methods for allophone clustering are based on parameter sharing between the allophone models that resemble each other in behaviors of feature vector sequences. However, all the features of the vector sequences may not necessarily have a common allophone clustering structures It is considered that the vector sequences can be better modeled by allocating the optimal allophone clustering structure to each feature. In this paper, we propose Feature-Dependent Successive State Splitting (FD-SSS) as an implementation of FD-AC. In speaker-dependent continuous phoneme recognition experiments, HMMs created by FD-SSS reduced the error rates by about 10% compared with the conventional HMMs that have a common allophone clustering structure for all the features

Matsuda, Shigeki

Nakai, Mitsuru

Sagayama, Shigeki

Shimodaira, Hiroshi

Edinburgh Research Archive

FEATURE-DEPENDENT ALLOPHONE CLUSTERINGShigeki Matsuda, Mitsuru Nakai, Hiroshi Shimodaira, and Shigeki SagayamaJapan Advanced Institute of Science and TechnologyTatsu-no-Kuchi, Ishikawa, 923-1292 JapanE-mail:  matsuda,mit,sim,sagayama@jaist.ac.jpAbstractWe propose a novel method for clustering allophones calledFeature-Dependent Allophone Clustering (FD-AC) that deter-mines feature-dependent HMM topology automatically. Existingmethods for allophone clustering are based on parameter sharingbetween the allophone models that resemble each other in be-haviors of feature vector sequences. However, all the features ofthe vector sequences may not necessarily have a common allo-phone clustering structures. It is considered that the vector se-quences can be better modeled by allocating the optimal allo-phone clustering structure to each feature. In this paper, we pro-pose Feature-Dependent Successive State Splitting (FD-SSS) asan implementation of FD-AC. In speaker-dependent continuousphoneme recognition experiments, HMMs created by FD-SSS re-duced the error rates by about 10% compared with the conven-tional HMMs that have a common allophone clustering structurefor all the features.1. INTRODUCTIONIn recent speech recognition techniques, hidden Markov models(HMMs) are one of the most powerful techniques for modelingthe time sequential data. HMM models non-stationary featurevector sequences by switching the finite number of stationaryvector signal sources. However, being limited amount of train-ing data available, more than enough number of free parametersleads to over-fitting or over-learning and resulting in poor gener-alization. Hence it is essential to devise an efficient representationscheme of acoustic features with less number of free parameters.Parameter tying is a typical manner that reduces the excess freeparameters. The parameter tying techniques can be classified gen-erally into the following four levels.Level 1: allophone clustering [1, 2]Level 2: state tying [1, 2, 3]Level 3: tied mixtures [4]Level 4: distribution parameter tying [5]The allophone clustering shares the parameters of the allophonemodels that are similar each other in the behaviors of feature vec-tor sequences. The total number of parameters can be reducedefficiently with minimum loss of information by the tying tech-nique. A number of clustering algorithms have been proposed inthis framework. But none has tried to relax the tacit constraint onacoustic features, i.e., treating the features as a vector. In speechrecognition, feature vector sequences often consist of MFCCs, a2nd MFCC 8th MFCC-20-10010-20-1001020-20-10010-20-1001020ka i/ /ka i/ /ka a/ /ka a/ /Figure. 1: Trajectories of 2nd and 8th MFCC of allo-phones /aka/ and /aki/power and its time derivatives. Since these individual featureshave different behaviors and different features may have differentcontext dependencies, all the features may not necessarily havea common allophone clustering structure. It is considered thatthe feature vector sequences can be modeled more efficiently byclustering allophones for each feature separately.In this paper, we propose Feature-Dependent Allophone Cluster-ing (FD-AC) that clusters allophones for each feature separately.In other word, FD-AC determines the tying structure of distri-bution parameters for each feature separately. In section 2, wedescribe a principle of FD-AC and Feature-Dependent Succes-sive State Splitting as an implementation of FD-AC. In section3, HMMs with different allophone clustering structures createdby FD-SSS are evaluated in the continuous phoneme recognitionexperiment. The last section is a conclusion.2. FEATURE-DEPENDENTALLOPHONE CLUSTERINGIn speech recognition, MFCCs and their time-derivatives (deltaMFCCs) are commonly used as acoustic feature vectors. It isgenerally considered that the lower-order MFCCs have more sig-nificant information than the higher-order MFCCs. Therefore, as-signing more number of HMM states to the lower-order MFCCsthan to the higher-order MFCCs may result in better recogni-tion performance. Fig. 1 shows an example of feature vectorsequences of a phoneme /aka/ and a phoneme /aki/. Here, /aki/denotes the phoneme /k/ with the preceding phoneme /a/ and thefollowing phoneme /i/. We can see that the both phonemes havesimilar trajectories for the 8th MFCC, but different for the 2ndMFCC. In this case, it is better to assign a common HMM stateto the 8th MFCC of those two allophones so that the number offree HMM parameters is reduced, while, for the 2nd MFCC, they1st feature 2nd featureAllophone modelsAllophone clustering structuresa-k+a a-k+i8th featureFigure. 2: Feature-Dependent Allophone Clustering (FD-AC) that creates different allophone clustering structuresfor individual featuresneed different HMM states each other. Therefore, allocating theoptimal number of HMM free parameters and determining the ty-ing structure for each individual feature by means of clusteringwill lead to a better modeling that shows higher precisions androbustness than the existing methods.Fig. 2 illustrates the idea of Feature-Dependent Allophone Clus-tering. In this figure, each feature space consists of different num-ber of clusters: there are six clusters in the 1st feature, five clustersin the 2nd feature and four clusters in the 8th feature. Allophones/aka/ and /aki/ share a common state only for the 8th feature.2.1. Feature-Dependent SuccessiveState SplittingIn order to obtain the feature-dependent allophone clusters and todecide the optimal numbers of free HMM parameters, we extendSuccessive State Splitting (SSS)[2] that is one of the most popu-lar algorithms for creating the hidden Markov networks (HMnet).The SSS algorithm increases the model likelihood successfullyby splitting a state in HMnet that represents a state tying struc-ture and an allophone clustering structure. We utilize the SSSalgorithm for the proposed feature-dependent allophone cluster-ing and call the new algorithm as FD-SSS. There are two possibleimplementations of FD-SSS as follows.Synchronous type: Create a HMnet for each feature with theconstraint that the state transition of each feature’s HMnettakes place synchronously with those of other feature’s HM-nets (Fig. 3).Asynchronous type: Create a HMnet for each individualfeature separately, and combine the HMnets into theAsynchronous-Transition HMMs [6, 7], in which state tran-sition occurs asynchronously for all the features but thesame efficient decoding algorithm such as one-pass Viterbisearch can be still applied with the conventional HMMs.In this paper, we discuss only the asynchronous type. The outlineof the FD-SSS algorithm is shown in Fig. 4.Step 1: Train a single state HMM (HMnet) for each feature withall the training phone samples, where the output probabilityfor each feature is modeled by a single Gaussian distributionwith two scalar parameters, i.e., a mean and a variance.1st feature2nd featuren-th featureFigure. 3: FD-SSS (synchronous type) for generating aFD-HMnet. Each vertical dotted-line indicates the syn-chronized state-transitions for all the features.Step 2: Among the states of all the HMnets (i.e., HMnet forthe 1st feature through the HMnet for the  -th feature), findthe one that will earn the largest likelihood gain when beingsplit into two states. The state splitting gains are examinedboth in contextual and temporal domains.Step 3: Split the state and re-train all the states that are affectedby the split using the corresponding data subsets.Step 4: Repeat steps 2 and 3 until the number of all statesreaches a preset number.Step 5: Create AT-HMMs [6, 7] with the allophone clusteringstructures of the obtained HMnets by clustering the statetransition timings.The state splitting gain  in Step 2 can be calculated by the sameequation used in ML-SSS. In case of splitting state  into twostates,   and , the gain is defined by                                                    (1)     	   (2)             (3)where   is the variance of state , 	   represents the prob-ability of being in state   at time , and       denotes theprobability of being in state   at time  and state    at time 	.3. EXPERIMENTSTo evaluate the proposed method, the ATR word speech databaseof Japanese important 5240 words uttered by 2 male speakers(MHT, MAU) and 2 female speakers (FMS, FFS) was used. Outof them, the odd-numbered 2620 words and the phoneme bal-Initial modelSplit in the     temporal domainSplit in the     contextual domainSelect a state to splitRetrain the model2nd feature n-th feature1st feature2nd feature1st feature n-th featuren-th feature2nd feature1st featureFigure. 4: FD-SSS (asynchronous type) for generating aFD-HMnet with the optimal state tying structureanced 216 words were used for training the speaker-dependentHMnet, and half of the even-numbered 1310 words were used fortesting. 12 MFCCs, 12 MFCCs, log-power and log-powerextracted with 5ms frame period and 25ms frame length wereused as an acoustic feature vector. The phoneme categories forrecognition were          	                   3.1. Model Generation using FD-SSSFig. 5 and Fig. 6 show the examples of the HMnet topologies forphoneme /k/ of a male speaker (MHT), where the former (Fig. 5)was created by ML-SSS, while the latter (Fig. 6) was built bythe proposed FD-SSS algorithm. The upper row in each roundbox denotes the left context of the phoneme shown in the mid-dle row, and the lower row represents the right context. As isshown in Fig. 6, the HMnets by FD-SSS (FD-HMnet) have dif-ferent state-tying structures for different features, which supportsour assumption that different features may have different allo-phone clustering structures.Fig. 7 shows the number of states that were assigned to each in-kaiueoykaiueoy_aiueoNqaiueoNq_kaiueoyaiueoNq_kaiueoNq_kaaiueoNq_kieyaiueoNq_kuoFigure. 5: Example of the HMnet topologies by ML-SSSfor /k/kaiey_kuo_kaiueoyaiueoNqaiueoNq_kiyaiueoNq_kaueoaiueoNq_keiyaiueoNq_kauokaiueoyaiueoNqkaiey_kuo_aiueoNq_koNkuaiueq_kuaiueoNq_koaiueoNq_keyaiueoNq_kiaaiueoNq_kieyaiueoNq_ko aiueoNq_kaiueoyaiueoNq_kaiueoNq_keaiueoNq_kaiyaiueoNq_kuokieoyaiueNqkeioyo_kauo_aiueoNq_kaiueoNq_kiaiueoNq_koyaiueoNq_kauekaiueoyaiueoNqkau_kieoy__kieoyaiueoNq_kauaiueoNqkoaiueoNqkeiyHMnet for 1st MFCCHMnet for 3rd MFCC HMnet for 4th MFCCHMnet for 2nd MFCCFigure. 6: Examples of the HMnet topologies by FD-SSSfor /k/dividual feature in the FD-HMnet under the condition that totalnumber of distribution parameters is fixed to 20000. It can beseen from the figure that the numbers of assigned states are dif-ferent among the features, implying that the proposed FD-SSSsuccessfully allocated the optimal number of free-parameters tothe allophone structure for each individual feature in the sense ofmaximum-likelihood estimation.Fig. 8 illustrates the average variances of the output distributionsof the FD-HMnet in comparison with those of ML-SSS basedHMnet (ML-HMnet). We can see that the FD-HMnet has smallervariances than the ML-HMnet. It is considered that the FD-HMnet represents the information of the training data more ef-ficiently than ML-HMnet.3.2. Continuous Phoneme RecognitionFor the evaluation of FD-HMnet generated by FD-SSS, contin-uous phoneme recognition experiments were performed in com-parison with the conventional ML-HMnet created by ML-SSS.For decoding, the one-pass Viterbi algorithm was used with thephonotactic constraints in Japanese. The number of states forphoneme HMM was fixed to 3 or 5, and the SSS algorithm wasNumber of states for individual features0200400600800Coefficients1st to 12th MFCCsPower1st to 12th delta MFCCsDelta powerFigure. 7: Numbers of states of FD-HMnet created by FD-SSSAverage variance for individual features0102030ML-SSSFD-SSSCoefficients1st to 12th MFCCs and a power 1st to 12th delta MFCCs anda delta powerFigure. 8: Average variances of models created by ML-SSS and FD-SSSTable. 1: Phoneme error rates of models generated by FD-SSS and ML-SSS#parameters #states method %error %reductionML-SSS 7.8 –3FD-SSS 6.6 15.410400ML-SSS 6.4 –5FD-SSS 5.4 15.6ML-SSS 6.3 –3FD-SSS 5.4 14.320800ML-SSS 5.3 –5FD-SSS 4.8 9.4stopped when a number of free parameters reached 10400 or20800.Table 1 shows the experimental results where %reduction denotesthe reduction of phoneme error rates defined by%error of ML-SSS  %error of FD-SSS%error of ML-SSSWe can see that FD-HMnet created by FD-SSS achieved aphoneme error reduction rate of about 10%. It is considered thatacoustic feature vector sequences are modeled efficiently by FD-SSS based on FD-AC.4. CONCLUSIONIn this paper, we have proposed a new notion of feature-dependent allophone clustering (FD-AC) that clusters the allo-phones for each individual feature separately. As an implemen-tation of FD-AC, we have developed Feature-Dependent Succes-sive State Splitting (FD-SSS), in which SSS-like state splittingoccurs separately for each individual feature to create a feature-dependent hidden Markov network (FD-HMnet).In the continuous phoneme recognition experiments, the pro-posed FD-HMnet successfully reduced the error rates by about10% compared with the conventional HMnet created by ML-SSS.It is considered that FD-AC is an effective technique for improv-ing the speech recognition performance of the acoustic models.Future works will include the evaluation of a speaker-independentmodel and the application to continuous speech recognition sys-tem.REFERENCES1. J. Takami and S. Sagayama, “A Successive State Splitting Al-gorithm for Efficient Allophone Modeling,” in Proc. ICASSP,vol. I, pp. 573–576, 1992.2. M. Ostendorf and H. Singer, “HMM Topology Design UsingMaximum Likelihood Successive State Splitting,” ComputerSpeech and Language, vol. 11, no. 1, pp. 17–41, 1997.3. X. D. Huang, K. F. Lee, H. W. Hon, and M. Y. Hwang, “Im-proved Acoustic Modeling with the SPHINX Speech Recog-nition System,” in Proc. ICASSP, vol. 1, pp. 345–248, 1991.4. J. Bellegarda and D. Nahamoo, “Tied mixture continuous pa-rameter models for large vocabulary isolated speech recogni-tion,” in Proc. ICASSP, vol. 1, pp. 13–16, 1989.5. S. Takahashi and S. Sagayama, “Four-Level Tied-Structurefor Efficient Representation of Acoustic Modeling,” in Proc.ICASSP, vol. 1, pp. 520–523, 1995.6. S. Matsuda, S. Sagayama, M. Nakai, and H. Shimodaira,“Asynchronous Transition HMM,” in Proc. ICASSP, vol. 2,pp. 1005–1008, 2000.7. S. Sagayama, S. Matsuda, M. Nakai, and H. Shimodaira,“Asynchronous Transition HMM for Acoustic Modeling,” inProc. IEEE Workshop on Automatic Speech Recognition andUnderstanding, 1999.

Feature-Dependent Allophone Clustering

https://era.ed.ac.uk/bitstream/1842/985/1/Matsuda2000ICSLP10.pdf

Feature-Dependent Allophone Clustering

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Archive