In this work we do a comparative evaluation between Artificial Neural Networks (RNA's) and Continuous Hidden Markov Models (CDHMM), in the framework of the recognition of isolated words, under the constrain of using a small number of features extracted from each voice signal. In order to accomplish such comparison we used two models of neural networks: the Multilayer Perceptron (MLP) and a variant of the Radial Basis (RBF), and some HMM models. We evaluated the performance of all models using two different test set and observed that the neural models presented the best results in both cases. Seeking to improve the HMM performance we developed a hybrid system, HMM/MLP, that improved the results previously obtained with all HMMs, and even those obtained with the neural networks for the all previous HMM, and even the neural nets for the hardest test set case

Diniz, Suelaine dos Santos

Santos, Sidney Cerqueira Bispo dos

Silva, Dirceu Gonzaga da

Thomé, Antonio Carlos Gay

English

Pantheon

Automatic Speech Recognition: A Comparative Evaluation between NeuralNetworks and Hidden Markov ModelsAntonio Carlos Gay Thomé -PhD Sidney Cerqueira Bispo dos Santos -DCthome@nce.ufij.br sidney@aquarius.ime.eb.brTel: 5521598 3163 Tel/Fax: (021) 5420999Dep. Ciência da Computação -IM-UFRJ Dep. Engenharia Elétrica -IME -DE/3Ilha do Fundão, Cidade Universitária Pça Gen Tibúrcio, 80 -Praia Verme1ha20001-970 Rio de Janeiro -RJ -Br, 1998 22290- 000 Rio de Janeiro -RJ -Br, 1998Suelaine dos Santos Diniz -MC Dirceu Gonzaga da Silvasuelaine@fund.cepel. br dirceu@aquarius .ime .e b .brTel: 5521 5982348 Tel/Fax: (021) 54209 99Dep. Ciência da Computação -IM-UFRJ Dep. Engenharia Elétrica -IME -DE/3Ilha do Fundão, Cidade Universitária Pça Gen Tibúrcio, 80 -Praia Vermelha20001-970 Rio de Janeiro -RJ -Br, 1998 22290- 000 Rio de Janeiro -RJ -Br, 1998Abstract -In this work we do a comparative On the other hand, although they do noteva1uation between Artificial Neural Networks accomplish the temporal modeling, feedforward neural(RNA 's) and Continuous Hidden Markov Models nets do present some other important characteristics(CDHMM), in the framework of the recognition of such as: ability to learn complex functions,isolated words, under the constrain of using a small generalization, parallelism and tolerance to noise, thatnumber of features extracted from each voice signal. return them an important option for the area of voiceIn order to accomplish such comparison we recognition.used two models of neural networks: the Multilayer The main goal for this work was to comparePerceptron (MLP) and a variant of the Radial Basis the voice recognition performance among the two(RBF), and some HMM models. We evaluated the chosen neural models, the MLP and the Radial Basisperformance of all models using two different test sets nets, and some Continuous Density Hidden Markovand observed that the neural models presented the best Models (CDHMM). A reduced Portuguese vocabularyresults in both cases. Seeking to improve the HMM was chosen as used in [5], which is composed by theperformance we developed a hybrid system, following words: liga (turn on), pare (stop), graveHMM/MLP, that improved the results previously (record), pausa (pause), avance (move), siga (proceed),obtained with all HMMs, and even those obtained with volte (return), desliga (turn off), ejete (eject) andthe neural networks for the all previous HMM, and apague ( erase ). The amount of 1697 repetitions ofeven the neural nets for the hardest test set case. these words where recorded from a group of 113different male speakers, averaging of 25 years old.According to [5,9], we also limited the1. Introduction member of voice signal features to be used as input tothe models, to maximum of 5. This constrain wasThe project of any voice recognition system adopted in order to reduce processing time andrequires special attention with two major figures: the memory space requirements, main1y for the neuralacoustic and the temporal components of the signal. It models.is known that the Markov models (HMMs) are able totackle with both phenomena and that the same does nothappen in the case of feedforward neural network 2. The Modelstypes.In spite of doing the acoustic modeling 2.1 HMMthrough the states mixtures and the temporal modelingthrough the transitions among states and the duration of We only used CDHMM -Continuous Densitythe states, Markov models suffer from some peculiar Hidden Markov Models for this work. The chosenproblerns. The independence hypothesis, for example, structure for the models was the Baki's model, aignores any context information, and the flfSt order particular case of the left-right, and for the trainingMarkov hypothesis may causes unsuitable modeling of procedure, we adopted the Segmental-Kmeans methodthe coarticulations and the duration ofthe states. as described in [I].For the experiments we chose 3 basic models The training procedure is accornplished inas described below in Table 1. three distinct phases. The flrst phase is off line and assaid before, it consists of an unsupervised clustering.over the data patterns. The second phase deals with thelearning of the flrst set of MLP networks and the thirdaussian rnixtures phase ends the system learning through with the secondaussian mixtures MLP network. AlI MLPs were trained with theaussian mixtures backpropagation algorithm.For the test procedure it is only used the firststage cornpounded by the gaussian and first MLP layer,2.2 Neural Network and the second stage (second MLP). In order to get amore detailed description ofthe model, see [5].Two different models were used, the first onewas a the traditional MLP with two hidden 1ayers andlearning ru1e irnplemented through the 2.3 Hybrid System : HMM-MLPbackpropagation a1gorithm [5]. The net networktopology was composed as 481x69x20xI0, i.e., 481 In the CDHMMs, the durations of the eventsnodes in the input layer, 69 neurons in the flrst hidden are modeled through the probabilities ofthe transitions,layer, 20 in the second and 10 in the output layer. We which gives to an exponential distribution.adopted an adaptive learning rate, a momentum of 0.75 Another way of getting information about theand the sigmoid function for alI neurons. With this duration of the states is using the Viterbi's algorithm,configuration the MLP network reached to an error of to perform a direct measure of the training data2.7662 after 36710 epochs. segments as purposed in [1].The other neural model used was a variant of Using Viterbi's algorithm, the procedure forthe radial basis as developed in [5]. Its structure is the recognition becomes the following: flrst the bestbasically' cornposed by 4 layers divided into 2 stages, s~~entatio~ is found and the correspondingas illustrated in Figure 1. 1ikelihood lS computed; after that the number ofsegments that were captured by each state is taken and--: the corresponding probability function is then.estimated. The system likelihood, as can be seen in EqCl 1, is then computed by the addition of the likelihoodprovided by the Viterbi's algorithm and the estimatedcJ probability function.f ' NC3 l°gP(q,OIÀ)=fJlogP(q,OIÀ)+adLlog[pj(dj)] (1)j=1o stage where a and 13 are scaling multiplier, and dj is thec,. computed duration of the state j .k-means.1 ..-I, 10 stage I d . th b b .l . d .--~ n or er to estimate e pro a llty ensltyfunction as wel1 as the value of the two scaling factors,Figure 1 -Variant ofthe radial basis network and a1so to discriminate the input patterns among theHMMs, we projected the usage of a MLP network, asThe flrst layer is only used for the training can be seen in the Figure 2. The network, receives aspurpose, it is off line and performs an unsupervised input from each of the 10 HMM, the duration of statesclustering on the data set through the usage of the K- and the likelihood cornputed by each HMM. For themeans algorithm In our case, each pattem was hybrid system we used the "8s5m" HMM (table1) andcomposed by 4 sets of 120 points each corresponding a "90x20xI0" MLP network.each one, to one of the 4 voice features used as input For the network training we adopted anand extracted over the 120 windows ofthe signal. Each adaptive leaming rate, starting it at the value of 0.002,of these data sets was divided into 10 clusters, giving a momentum factor of 0.7, and a sigmoid function foreach one the parameters, mean and deviation, to the the neurons. The convergence occurred very fast,construction ofthe second layer, that is composed by a within 1500 epochs, to an error level ofO,001.set of gaussian functions. For the third and fourth layerof the model we used two 4 layer MLP networks, thefirst with a topology of 1 Ox40x20x1 0 and the secondof40x55x20xI0.used only the 120j76s, because of the static input sizeQ Q Q Q -., restriction. For the HMM we tested alI configurations~..~ and alI model~ described in 2.1. The hybrid system was.tested only Wlth the HMM 8s5m model and the xj50s~..~ ., configuration. ...As sald before, we restrlcted our experiments: to the set offeatures selected in [5], that is:Q Q Q Q --" .1° Cepstrurn coefficient;~..~ .1°, 2° and 3° mel-cepstrurn coefficient; and.zero crossing rate relation between thefIrst and the second half of the voicesignal.Figura 2: Hybrid System ModelTable 3 shows the results obtained with theHMM. It can be seen that for the fIrst group, the best3. Simulations and Results result was provided by the models 8s10m and 15s10mboth with the xj76s configuration, and for the secondThe data set used for the experiments was test group: by the model 15s10m with the 120j76scomposed by 1697 patterns recorded from 113 confi~atlon. Th~se results a1low us to make thespeakers. As can be seen in Table 2, we divided the ~O1loWIng concluslons: larger number of states rnaydata set into three groups, one for training with 1000 1mprove p~rforrnan~e; larger superposition areapatterns (100 from each word) and two for test, one between .adJacen~ WIndows improve co~elation andwith 110 patterns and another with 697. In the fIrst test ~ontext inforrnati?~; free number of WIndows tnayset, the words were spoken by 11 different speakers, 1mprove illv:IM ablhty to mod~l temp~ral variation, andwere thl'ee of them also provided tokens for the fma1ly, the mcrease of the WIndow s1Ze seems to turntraining set. The second test set was a superset of the the model more robust.fIrst, including 587 new patterns recorded from 59 newspeakers. T ABLE 3: The tests results accomplished with HMMThe patterns were recorded through a 16 bits ...Creative Labs sound blaster board, in a room withoutany special acoustic protection. Repetitions from one IIspeaker were done in different days and 288 patterns of 2the second test set were recorded with a different type 2of microphone. 3TABLE 2- Training and test groups used in the Table 4 shows the results obtained with theex eriments MLP and Radial Basis networks. As can be seen, bothnetworks provided similar order of accuracy. VVhatdiffers from one model to the other is the time spent fortraining, that was much srna1ler in the case of theRadial Basis. Two important aspects to consider are:the first stage of the Radial Basis can also be used toConsidering the correlation between adjacent e~aluate the relevance of each selected feature of voicewindows and considering also that the inforrnation slgnal, and secondly, the training of the fIrst stage canbrought by each window is directed related to the size be done in para1lel [5].of the voice signal frames and their superposition, wedecided to test three distinct strategies of windowing TABLE 4: The t~sts results acc~~lished with RNAsand superpositioning: m % of reco tlon1 -A f1Xed number of 120 windows, with MLP RBFaverage size of 35 ms, and a f1Xed rate of 76% of Gro I 95,0 98,0superposition, as used in [5]. From now we wi1l report Gro I 79,0 80,5to it as the 120j76s configuration;2 -A free number of windows, of fixed size, Table 5 shows a comparison among the twoand a fixed rate of 50% of superposition -xj50s; RNAs, one of the HMMs and the hybrid model. As can3 -A free number of windows, of fixed size, be seen the neural nets surpassed the HMM, what rnayand a fixed rate of76% of superposition -xj76s. be due to the context inforrnation that they possiblyFor the fixed size cases, it was used windows were able to extract better than the HMM. On the otherwith a duration of 20 ms. For the neural networks we hand, we noticed that as the test set became larger andharder all mode1s 1ost performance: around 16% the Publishing Company, Nova Iorque, 1993.MLP, 18% the Radial Basis, 14% the HMM, and 11% [3] LEE, K. F. ..Automatic Speech Recognitin -the hybrid system, which proved to be the most robust The Development of the SPHINX System",mode1. This suggests that f!1e combination of HMM Kluwer Academic Publisher, Boston, 1989.and neural models may result in better systems, [4] PARANAGUÁ, E.D.S. ..Reconhecimento depossibi1ity because they may be able to take advantage Locutores Utilizando Modelos de Markovofwhat is more effective in each its components. Escondidos Contínuos", Tese de Mestrado,IME, 1997TABLE 5: The tests results accomplished with RNAs, [5] DINIZ, S.S. ..Uso de Técnicas Neurais para oHMM and H. tem in % ofreco Reconhecimento de Comandos à Voz", TeseTe M F HMM st. de Mestrado, IME, 1997Gro I 95,0 98,0 87,3 , [6] HAYKIN, S. ..Neura1 Networks AGrou I 79,0 80,5 73,5 83,1 Comprehensive Foundation", Macmillan 1994[7] LEE,C.H; SOONG F.K.; PALIWAL K.K...Automatic Speech and Speaker Recognition -4. Conclusions Advanced Topics", Kluwer AcademicPublisher, Boston, 1996.In spite of Markov Hidden Models had not [8] COLE, R. A. ..Survey of the State of the Artshowed the same leveI of accuracy of the neural in Human Language Technology", Center Formodels, the experiments were valid to show that flrst Spoken Languagem Understanding, Oregonones are more robust and faster trained then the Graduate Institute, Publicações Técnicas, Nov.second. 1995The experiments also permitted us to [9] DINIZ, S. S. , THoMÉ, A. C. G. ..Uso deconc1ude about the potentiality of hybrid models, Técnica Neural para o Reconhecimento deespecially those with the type of combination Comandos à Voz", IV Simpósio Brasileiro dedeveloped here, where the HMM takes care of the Redes Neurais, pp 23-26, dez 1997temporal modeling and the neura1 net takes cases ofthe [10] DUDA, O.R., HART, E., P ., ..Patternacoustic modeling and context extraction. The state Classification and Scene Ana1ysis", Wiley-duration modeling provided by the neural nets was Intercience, 1973, pp. 114-118more efficient than that given by the exponential [11] RENALS, S., MORGAN, N., ..Connectionistdistribution of the standard HMM algorithm Probabilit Estimation in HMM SpeechAnother factor observed was the importance Recogation", International Computer Scienceofthe se1ection ofa set ofre1evant features to represent Institute, Berkeley, Dec 1992the voice signal. It was possible to verify that the [12] CHO, S-B, .'Neural- Network Classifiers forfeatures used and chosen by [5] for the neural net Recognizing Totally Unconstrainedsystems, were not the most adequate ones for the case Handwritten Numera1s" , IEEE Transactionsof the HMM systems. A larger set of features was not on Neura1 Networks, vol. 8, n.1 , January 1997.considered in our study because this would imply in asevere increase in of the complexity and time spent fortraining the neural models.Our goal is to proceed the research evaluatingdifferent schemes of hybrid systems connecting HMMand neural nets, and also testing them over morecomplex data sets.References[1] RABINER, L. R. & JUANG, B.H...Fundamenta1s of Speech Recognition"Prentice Hall, Inc., Eng1ewood Clifs, NovaJersey, 1993.[2] DELLER Jr., J. R. et alii. ..Discrete-TimeProcessing of Speech Signals", Macmillian

Automatic speech recognition: a comparative evaluation between neural networks and hidden markov models

http://pantheon.ufrj.br/bitstream/11422/2568/1/14_99_000611273.pdf

Automatic speech recognition: a comparative evaluation between neural networks and hidden markov models

Abstract

Similar works

Full text

Available Versions

Pantheon