In this paper we propose and study low complexity algorithms for on-line estimation of hidden Markov model (HMM) parameters. The estimates approach the true model parameters as the measurement noise approaches zero, but otherwise give improved estimates, albeit with bias. On a nite data set in the high noise case, the bias may not be signi cantly more severe than for a higher complexity asymptotically optimal scheme. Our algorithms require O(N3) calculations per time instant, where N is the number of states. Previous algorithms based on earlier hidden Markov model signal processing methods, including the expectation-maximumisation (EM) algorithm require O(N4) calculations per time instant

Ford, Jason J.

Moore, John B.

Queensland University of Technology ePrints Archive

Reduced Complexity On-line Estimation of Hidden Markov ModelParameters.J.B. Moore1and J.J. Ford11Department of Systems Engineering, Research School of Information Sciences andEngineering, Australian National University, Canberra, ACT 0200, Australia.AbstractIn this paper we propose and study low complexity algorithms for on-line estimation of hidden Markov model (HMM) parameters. The estimatesapproach the true model parameters as the measurement noise approacheszero, but otherwise give improved estimates, albeit with bias. On a nitedata set in the high noise case, the bias may not be signicantly more severethan for a higher complexity asymptotically optimal scheme. Our algorithmsrequire O(N3) calculations per time instant, where N is the number of states.Previous algorithms based on earlier hidden Markov model signal processingmethods, including the expectation-maximumisation (EM) algorithm requireO(N4) calculations per time instant.1 IntroductionA discrete time homogeneous Markov chain, with a nite state space having N elements, issupposed observed in noise. Such a situation is termed a hidden Markov model or HMM.A recent treatment can be found in Elliott[3].In addition to estimating the state of the chain given the observations, it is oftenof importance to estimate the state values, transition probability matrix and the noisecharacteristics, see Collings[2], Elliott[3], Ford[4, 5] and Moore[12, 13]. To estimate thetransition matrix of the Markov chain using the well established classical Baum-Welchalgorithm and related expectation-maximisation (EM) algorithms based on data up untiltime T and initial model parameter estimates, it is usual to estimate the number ofjumps JijTof the chain from state i to state j, for 1  i; j  N , up to the time T andthe occupation times OiTis state i for 1  i  N . Here N is the number of states inthe model. Then JijT(OiT) 1gives an improved estimate in a likelihood sense for theprobability of the state switching from i to j, ie. aij. To estimate other model parametersinvolved in the measurement equation, related transitions TiTfrom states i to outputs arerequired. Using the improved model parameters the process is repeated until convergenceto a (local) maximum of the likelihood function. The EM algorithm re-estimates modelparameters using forward and backward passes through the data, and so is not really anon-line scheme. Also, the theory gives only local convergence to a local maximum of thelikelihood function.In Moore[12], almost-sure consistent convergent parameter estimators are proposed forestimating hidden Markov model parameters online. The almost-sure asymptotic conver-gence results are obtained via standard martingale convergence results, refer to Meyer[11]and Neveu[14], and the Kronecker lemma, refer to Loeve[10] and Neveu[14], stability prop-erties for HMM lters are used, see Dey[1], Shue[17], and ordinary dierential equation the-ory for stochastic approximation, see Ljung[9], Kushner[7], Gerencser[6] and LeGland[8].The optimal parameter estimates converges almost surely to the true parameter values Ofcourse, there must be persistence of excitation in the models, (and estimators), to achievethis property. The results presented in Moore[12] contrast the Baum-Welch re-estimationresults which are only guaranteed to converge to a local maximum of the likelihood func-tion.Here, we propose reduced complexity estimation schemes based on the consistentschemes in Moore[12] with the view to reducing (additional) computational eort at eachstep from O(N4) to O(N3) The schemes become consistent as the noise level approacheszero, but otherwise giving improved estimates, albeit with bias.In Section 2, the discrete-time HMM signal model is dened, on-line estimators areintroduced, and associated convergence properties reviewed, including the almost sure con-vergence result from Moore[12]. In Section 3, reduced complexity algorithms are proposed.Simulations are given in Section 4, and conclusions in Section 5.2 Dynamics, Measure Change, and Martingale Properties2.1 DynamicsOur time parameter set is the non-negative integers Z+= f0; 1; 2; : : :g. On a probabil-ity space (;F ; P ) we suppose we have a nite state, time homogeneous Markov chainfX`g; ` 2 Z+. As pointed out in Elliott[3], without loss of generality we can take the statespace of the Markov chain to be the set S = fe1; e2; : : : ; eng of unit vectors in IRN. Hereei= (0; 0; : : : ; 1; : : : ; 0)02 IRN.Consequently, at each `;X`2 S. Consider the state space modelXk= AXk 1+ Vk(1)yk= (C0Xk 1) + (D0Xk 1)wk: (2)Here A 2 IRNNis the stochastic matrix with 10A = 1 := [1; 1; : : : ; 1]0elements aij> 0,1  i; j  N , and satisfying E[Xk] = AE[xk 1], aij= P (X`jX` 1= ej). Also Vkis amartingale increment satisfyingE [Vk+1jXk] = 0 2 IRN:Here, wkis a sequence of i.i.d. N(0; 1) random variables dened on (;F ; P ). Thatthe measurements yk2 IR are linear in Xk 1is not really a restriction since nonlinear op-erations f(X) are linear in X as with f=(f(e1); f(e2); : : : ; f(eN)) ; f(X) = hf;Xi. Denote,C = (c1; : : : ; cN)0; D = (d1; : : : ; dN)0.Write Gk:= the sigma eld generated by fX0; X1; : : : ; Xk; y1; : : : ; ykg and Yk:= thesigma eld generated by fy1; : : : ; ykg. We shall writeM for the model determined by theseparameters (aji; ci; di) ; 1  i; j; N . We assume throughout that the model order N isknown.2.2 Measure ChangeFrom Elliott[3] recall that a probabilityP is introduced such that underP , X is stilla Markov chain with transition matrix A, but the random variables ykare themselvesindependent and normally distributed as N(0; 1).`(X` 1) = ((y`  C0X` 1)=(D0X` 1))(D0X` 1)(y`)where (x) is, for example1p2e x2=2.With 0= 1 and k= k`=1`(X` 1), a probability measureP can be dened on(;G1) by puttingdPdPjGk=k.One can then show (see Elliott[3]), that under P , fwkg is a sequence of independentN(0; 1) random variables, where wk:= (yk  (C0Xk 1))=(D0Xk 1).Furthermore, under P the state X remains a Markov chain with transition matrix A.Let us denote the model (1) (2) asM =M(A;C;D; 0)where 0= E[X0].2.3 Transitions and Occupation TimesFor 1  r; s  N write Jrsk=Pk`=1hX` 1; eri hX`; esi, then Jrskis the number of jumps offXkg from state r to state s up to time k. The Markov chain Xkis not observed directly,but only through the observations j. The occupation times for being in state r up untiltime k are Ork=Pns=1Jrsk.Let us denoteJk:= (Jrsk); 1  r; s  N 2 IRNN(3)Ok:= Jk1 2 IRN(4)Thus in obvious notationJk=kX`=1X` 1X0`; (5)(Ok)diag=kX`=1X` 1X0` 1=kX`=1(X` 1)diag(6)Post multiplication of (1) by X0k 1and summing yieldsk 1J0k= A(k 1Ok)diag+ k 1kX`=1V`X0` 1(7)Given knowledge of fXkg, it makes sense to estimate A asAk= J0k(Ok) 1diag(8)at least when the inverse exists (as when all states are excited). We say that the statesare persistently exciting when there is an nite integer M > 0, such that Oikis k > Mfor i = 1; : : : ; N and a constant B < 1 such that for all K > M and 1  i  N thenk(Oik) 1< B. This is equivalent to the condition thatlimk!1supi;j;k(1kOk 1) 1diag<1 (9)In a parallel manner to the above, one can dene transitions Trk(f) =Pk`=1hX` 1; eri f(y`)where f(y) is either y, or y2, depending on application. So dene the row vector withelements TrkasT0k(f) =kX`=1f(y`)X0` 1: (10)Now (2) leads tok 1T0k(y) = C0k 1(Ok)diag+D0k 1kX`=1(X0` 1)diagw`(11)and thus estimates can be dened asCk= T0k(y)(Ok) 1diag: (12)Likewise, squaring (2) and post-multiplying by X0k 1we have:1kT0k(y2) = [c21; c22; : : :]1k(Ok)diag+ (d21; d22; : : :)1kkX`=1(X` 1)diagw2`+2(c1d1; c2d2; : : :)1kkX`=1(X` 1)diagw`: (13)EstimatesD can be constructed from(d21;d22; : : : ;d2N) = T0k(y2)(Ok)diag  (c21; c21; : : : ; c2N): (14)2.4 Parameter EstimationConsider model \estimates", possibly time varying, denotedcMk=cM1;cM2; : : : ;cMk. Letus denote associated conditional mean estimates based on the correct model and \incor-rect" model asbXkjk= E [Xkj Yk;M ] ;bXkjk;cMk 1= EhXkj Yk;cMk 1ibJkjk= E [Jkj Yk;M ] ;bJkjk;cMk 1= EhJkj Yk;cMk 1ibOkjk= E [Okj Yk;M ] ;bOkjk;cMk 1= EhOkj Yk;cMk 1ibTkjk(f) = E [Tk(f) j Yk;M ] ;bTkjk;cMk 1(f) = EhTk(f) j Yk;cMk 1i(15)Similar notation will denote one step ahead predictionsbXkjk 1etc., or smoothed estimatesbXkjk+detc.Now, in seeking on-line estimates of Jk, such as conditional mean ltered, it turnsout, perhaps surprisingly, to be essential to work rst with on-line estimates of JkgivenXk= eifor each i and then derive the desired estimates from these. Thus deneJXk:= Xkrow vec Jk2 IRNN2(16)row vec Jk= 10JXk2 IR1N2: (17)Let us dene conditional mean estimates and associated unnormalized forms underP forJkas followsbJXkjk:= E [Jkj Yk;M ] ; (JXkjk) :=EhkJXkj Yk;Mi: (18)WithbTXkjk,bOXkjk, (TXkjk) and (OXkjk) similarly dened. The following lemma is a convenientre-packaging of the following results of Chapter 3 of Elliot[3] to a matrix form.Lemma 1 Consider the HMM signal model (1) (2) denoted by M . The conditional meanltered estimates for JXk;Jk; TXk; Tk, underP , are obtained from:(JXkjk) = AB(yk)(JXk 1jk 1) + ((Ae1)diag; (Ae2)diag; : : : ; (AeN)diag) (B(yk)(k 1)diag I)row vec (Jkjk) = 10(JXkjk) (19)(TXkjk) = AB(yk)(TXk 1jk 1) + AB(yk)(k 1)diagf(y)row vec (Tkjk) = 10(TXkjk): (20)Proof: See Moore[12]Remarks:1. Exponential stablilty or initial-condition forgetting of the lters (19) and (20) followsby appealing to the generalised Perron-Frobenius result, see Seneta[16], in the sameway as in Dey[1], LeGland[8], Shue[17]2. The computational diculty is that the (JXkjk) calculation requires N4multiplica-tions for each up-date. See Elliott[3] for alternative form of the lters which can beeasier to implement in practice.3. Note Ok= Jk1, so that (Okjk) = (Jkjk)1. (Note also that kcan be derived from(JXkjk) by summing operations).Now consider parameter estimatesAkjk;cMk 1=bJ0kjk;cMk 1bOkjk;cMk 1 1diagCkjk;cMk 1=bT0kjk;cMk 1(f)bOkjk;cMk 1 1diag(21)and likewise forDkjk;cMk 1via (14) and then introduce the persistently excitation conditionassociated with the model M and its \estimate"cMkaslimk!1sup1kbOkjk;cMk 1 1diag<1 (22)The case studied in Moore[12] is wherecMk 1is given adaptively from estimatesAkjk;cMk 1,Ckjk;cMk 1andDkjk;cMk 1.2.5 Convergence ResultsBefore proceeding to propose reduced-complexity algorithms for estimating HMM param-eters we repeat here the almost sure convergence results for estimating the transitionprobability matrix A, stated in Moore[12].Theorem 1 Consider the HMM of (1) (2) and a particular observation sequence andstate sequence outcome, fykg and fXkg, of the HMM with all states in fXkg persistentlyexciting in that (9) holds. Consider the (somewhat articial) case where the conditionalmean estimates based on the true model M are available. Thenlimk!1Akjk;M;Ckjk;M;Dkjk;M= A;C;D; a:s: (23)The almost sure convergence rate guaranteed is (k)12which is like1pk(lnk(lnlnk))12, fork > 4 and for any  > 1, and the mean square rate is (k ) 12where (k ) 12is arbitrarilyslower than k 12.Proof: See Moore [12]. The proof is based on martingale convergence results and theKronecker Lemma.Theorem 2 Consider the HMM of (1) (2) and a particular observation sequence andstate sequence outcome, fykg and fXkg, of the HMM with all states in fXkg persistentlyexciting in that (9) holds. Consider also an assumed model setcMkwherecMkis adaptivelyupdated usingAkjk;cMk 1which we suppose is persistently exciting, along with M , in that(22) holds and C and D are known. Thenlimk!1Akjk;cMk 1= A a:s; (24)Proof: See Moore[12]. The proof is based on the ordinary dierential equation approach.Remark: See Moore[12] for the case whenCkjk;cMk 1andDkjk;cMk 1are also to beestimated.3 Reduced Complexity AlgorithmsOne way to reduce the computational requirements of the overall estimation algorithm (21)is by reducing the computational eort used to produce estimates of the conditional mean.We note that there is a subset of models fMg for which calculation of the conditional meanestimates is computational easier than for other models. Hence, the key idea of this paperis to calculate the conditional mean estimates of Jkand Okcorresponding to one of thesespecial modelsM (or a sequence of these models) rather than generic model \estimates"such as the adaptive Mkabove. This will reduce the computational eort required toimplement the overall estimation algorithm (21) .An example of one suchM is the i.i.d. sequence model which is a subset of the validHMMs. For the purpose of generating the conditional mean estimates,bJkjk;cMk 1andbOkjk;cMk 1, the state sequence can be modelled as an i.i.d. sequence, leading to a modelestimate withcMk=Miidfor all k asMiid= fAiid; C;D; 0g; Aiid=1N2641    1...   ...1    1375: (25)The lters now denotedbJkjk;MiidandbOkjk;Miidrequire O(N3) calculations per time instant.This is a reduction from the O(N4) calculations per time instant required to implementthe the general ltersbJkjk;cMk 1andbOkjk;cMk 1using estimatesAkjk;cMk 1.The lters for the model set M= fAiid+ INN; C;D; 0g for scalar ;  wheref;  : 10A = 1g also requires only O(N3) calculations per time instant. This set canapproximate a larger class of models and results in reduced estimation error.Remark1. Even though A = I may appear a likely candidate model of fMg for reducingcomplexity the persistently excitation condition (22) is not satised for this modeland the convergence no longer holds.2. Correct estimation occurs in low noise becausebXkjk;bA Xk. and invariant ofbA.4 Simulations Studies4.1 Reduced Complexity EstimationA 3000 point, 2-state HMM was generatedwith parameters: aii= 0:8; i = 1; 2 and aij= 0:2 8i; j i 6= j, C = [2; 4]0andD = 0:1 1.Estimation was performed in two ways: using the simplied model approximation (25),and usingcMkcorresponding to the best available parameter estimates. At this noise levellittle bias is introduced by the approximation.4.2 ThresholdTo examine the bias introduced into the parameter estimation a 10000 point, 2-state HMMgenerated with parameters: aii= 0:8; i = 1; 2 and aij= 0:2 8i; j i 6= j, C = [2; 4]0wassimulated at various SNRs. Estimation of A is performed assuming M = Miid. Figure1 shows estimation error verses SNR. From this gure it appears that for SNR > 12 orD < 0:2 the bias introduced is insignicant.4.3 More Complicated ApproximationsA 10000 point, 2-state HMM was generated with parameters: aii= 0:8; i = 1; 2 andaij= 0:2 8i; j i 6= j, C = [2; 4]0and D = 0:7 or SNR=6:3. Estimation of A was performedusing three methods: using the model set M, using the model set Miid, and using theadaptive scheme of Moore [12]. Figure 2 shows the convergence of parameters estimatesover time At this noise level it appears using models Mreduces the estimation error.5 ConclusionsIn this paper we present reduced complexity algorithms for on-line estimation of hiddenMarkov model parameters. The presented algorithm requires only O(N3) calculations pertime instant compared to the O(N4) calculations required to implement the estimation al-gorithm presented in Elliott[3]. The various estimates are not guaranteed to be consistent,but simulation studies indicate their usefulness.References[1] R.K. Boel, J.B. Moore and S. Dey, \Geometric Convergence of Filters for HiddenMarkov Models", Proceedings of the 34th Conference on Decision and Control, NewOrleans, Dec, 1995.[2] I.B. Collings, V. Krishnamurthy and J.B. Moore, \Online Identication of HiddenMarkov Models via Recursive Prediction Error Techniques", IEEE Transactions onSignal Processing, Dec, 1994.[3] R.J. Elliott, L. Aggoun and J.B. Moore, Hidden Markov Models, Estimation andControl, Springer-Verlag, 1995.[4] J.J. Ford and J.B. Moore, \On Adaptive HMM state Estimation", To appear in IEEETransactions on Signal Processing.[5] J.J. Ford and J.B. Moore, \Adaptive Estimation of HMM Transition Probabilities",To appear in IEEE Transactions on Signal Processing.[6] L. Gerencser, \Rate of Convergence of recursive estimators", SIAM J. Control andOptimization, Vol. 30, No. 5, pp. 120-1227, Sept. 1992.[7] H.J. Kushner and D.S. Clark, Stochastic Approximation Methods for Constrained andUnconstrained Optimization, Springer-Verlag, Berlin, New York, 1978.[8] F. LeGland and L. Mevel, \Geonmetric ergodicity in hidden Markovmodels", InternalPublication 1024, July 1996. ftp://ftp.irisa,fr/techreports/1996/PI-1028.ps.gz.[9] L. Ljung, \Analysis of recursive stochastic algorithms", IEEE Trans. Automatic Con-trol, vol. AC-22, pp. 551-575, Aug 1977.[10] M. Loeve, Probability Theory, 2nd Ed., Van Nostrand, Princeton, NJ, 1960.[11] P. Meyer, Martingales and Stochastic Integrals - I, Lecture Notes in MathematicsSeries, No. 284, Springer-Vertag, New York, 1972.[12] J.B. Moore, J.J. Ford & R.J. Elliott, \On-line consistent estimation of hidden Markovmodel parameters", submitted to IEEE Transactions on Signal Processing.[13] J.B. Moore, V. Krishnamurthy, \On-line Estimation of Hidden Markov Model Pa-rameters Based on the Kullback-Leibler Information Measure", IEEE Transactionson Signal Processing, Vol. 41, No. 3, pp 2557-2573, Aug 1993.[14] J. Neveu, Discrete Parameter Martingales, Amsterdam, Holland: North-Holland,1975.[15] L.R. Rabiner, \A tutorial on Hidden Markov Models and selected applications inspeech recognition", Proc. IEEE, Vol.77, No.2, pp 257-285, 1989.[16] E. Seneta, Non-negative matrices and Markov Chains, 2nd ed., Chapters 3-4,Springer-Verlag, 1981.[17] L. Shue, B.D.O. Anderson and S. Dey, \Exponential stability of lters and smoothersfor hidden Markov models" , Proceedings of the 4th European Control Conference,1997.10 15 20 25 30 35 40 45 50 55 6000.020.040.060.080.10.120.140.160.180.2SNR, dBBIASFigure 1: The empirical calculated of bias after 10000 points.0 1000 2000 3000 4000 5000 6000 7000 8000 9000 1000010−210−1100Time, kEstimation ErrorReduced 2iid      Adaptive Figure 2: An empircal comparsion

Reduced complexity on-line estimation of hidden Markov model parameters

http://eprints.qut.edu.au/78152/1/078.PDF

Reduced complexity on-line estimation of hidden Markov model parameters

Abstract

Similar works

Full text

Available Versions

Queensland University of Technology ePrints Archive