We present and analyse three online algorithms for learning in discrete
Hidden Markov Models (HMMs) and compare them with the Baldi-Chauvin Algorithm.
Using the Kullback-Leibler divergence as a measure of generalisation error we
draw learning curves in simplified situations. The performance for learning
drifting concepts of one of the presented algorithms is analysed and compared
with the Baldi-Chauvin algorithm in the same situations. A brief discussion
about learning and symmetry breaking based on our results is also presented.Comment: 8 pages, 6 figure

Alamino, Roberto C.

Caticha, Nestor

English

arXiv

Roberto Alamino

Nestor Caticha

Crossref

Online Learning in Discrete Hidden Markov Models

We present and analyze three different online algorithms for learning in discrete Hidden Markov Models (HMMs) and compare their performance with the Baldi-Chauvin Algorithm. Using the Kullback-Leibler divergence as a measure of the generalization error we draw learning curves in simplified situations and compare the results. The performance for learning drifting concepts of one of the presented algorithms is analyzed and compared with the Baldi-Chauvin algorithm in the same situations. A brief discussion about learning and symmetry breaking based on our results is also presented

Aston Publications Explorer

arXiv:0708.2377v1  [stat.ML]  17 Aug 2007Online Learning in Discrete Hidden MarkovModelsRoberto Alamino∗ and Nestor Caticha†∗Neural Computing Research Group,Aston UniversityAston Triangle, Birmingham, B4 7ET, United Kingdom†Instituto de Física,Universidade de São Paulo,CP 66318 São Paulo, SP, CEP 05389-970 BrazilAbstract. We present and analyse three online algorithms for learning in discrete Hidden MarkovModels (HMMs) and compare them with the Baldi-Chauvin Algorithm. Using the Kullback-Leiblerdivergence as a measure of generalisation error we draw learning curves in simplified situations.The performance for learning drifting concepts of one of the presented algorithms is analysedand compared with the Baldi-Chauvin algorithm in the same situations. A brief discussion aboutlearning and symmetry breaking based on our results is also presented.Key Words: HMMs, Online Algorithm, Generalisation Error, Bayesian Algorithm.INTRODUCTIONHidden Markov Models (HMMs) [1, 2] are extensively studied machine learning modelsfor time series with several applications in fields like speech recognition [2], bioinfor-matics [3, 4] and LDPC codes [5]. They consist of a Markov chain of non-observablehidden states qt ∈ S, t = 1, ...,T , S = {s1, s2, ..., sn}, with initial probability vectorpii = P(q1 = si) and transition matrix Aij(t) = P(qt+1 = sj |qt = si), i, j = 1, ..,n. Atdiscrete times t, each qt emits an observed state yt ∈ O, O = {o1, ...,om}, with emis-sion probability matrix Biα(t) = P(yt = oα|qt = si), i= 1, ...,n, α = 1, ...,m, which arethe actual observations of the time series represented, from time t = 1 to t = T , by theobserved sequence yT1 = {y1,y2, ...,yT }. The qt’s form the so called hidden sequenceqT1 = {q1, q2, ..., qT }. The probability of observing a sequence yT1 given ω ≡ (pi,A,B) isP(yT1 |ω) =∑qT1P(y1)P(y1|q1)T∏t=2P(qt+1|qt)P(yt|qt). (1)In the learning process, the HMM is fed with a series and adapts its parameters toproduce similar ones. Data feeding can range from offline (all data is fed and parameterscalculated all at once) to online (data is fed by parts and partial calculations are made).We study a scenario with data generated by a HMM of unknown parameters, an ex-tension of the student-teacher scenario from neural networks. The performance, as afunction of the number of observations, is given by how far, measured by a suitable cri-terion, is the student from the teacher. Here we use the naturally arising Kullback-Leibler(KL) divergence that, although not accessible in practice since it needs knowledge of theteacher, is an extension of the idea of generalisation error being very informative.We propose three algorithms and compare them with the Baldi-Chauvin Algorithm(BC) [6]: the Baum-Welch Online Algorithm (BWO), an adaptation of the offline Baum-Welch Reestimation Formulas (BW) [1] and, starting from a Bayesian formulation, anapproximation named Bayesian Online Algorithm (BOnA), that can be simplified againwithout noticeable lost of performance to a Mean Posterior Algorithm (MPA). BOnAand MPA, inspired by Amari [7] and Opper [8], are essentially mean field methods [9]in which a manifold of prior tractable distributions is introduced and the new datumleads, through Bayes theorem, to a non-tractable posterior. The key step is to take as thenew prior, not the posterior, but the closest distribution (in some sense) in the manifold.The paper is organised as follows: first, BWO is introduced and analysed. Next, wederive BOnA for HMMs and, from it, MPA. We compare MPA and BC for drifting con-cepts. Then, we discuss learning and symmetry breaking and end with our conclusions.BAUM-WELCH ONLINE ALGORITHMThe Baum-Welch Online Algorithm (BWO) is an online adaptation of BW where ineach iteration of BW, y becomes yp, the p-th observed sequence. Multiplying the BWincrement by a learning rate ηBW we get the update equations for ωωˆp+1 = ωˆp+ηBW ∆ˆωp, (2)with ∆ˆωp the BW variations for yp. The complexity of BWO is polynomial in n and T .In figure 1, the HMM learns sequences generated by a teacher with n= 2, m= 3 andT = 2 for different ηBW . Initial students have matrices with all entries set to the samevalue, what we call a symmetric initial student. We took averages over 500 randomteachers and distances are given by the KL-divergence between two HMMs ω1 and ω2dKL(ω1,ω2)≡∑yT1P(yT1 |ω1) ln[P(yT1 |ω1)P(yT1 |ω2)]. (3)We see that after a certain number of sequences the HMM stops learning, which isparticular to the symmetric initial student and disappears for a non-symmetric one.Denoting the variation of the parameters in BC by ∆, in BW by ∆ˆ, in BWO by ∆˜,and with γt(i)≡ P(qt = si|yp,ωp), we have to first order in λ∆pii =ληBCn∆ˆpii =λnηBCηBW∆˜pii, (4)∆Aij =ληBCn[T−1∑t=1γt(i)]∆ˆAij =λnηBCηBW[T−1∑t=1γt(i)]∆˜Aij ,∆Biα =ληBCn[T∑t=1γt(i)]∆ˆBiα =λnηBCηBW[T∑t=1γt(i)]∆˜Biα.10 100 1000p0.1d (Kullback-Leibler)0.0010.010.1FIGURE 1. Log-log curves of BWO for three different ηBW indicated next to the curves.For ηBW ≈ ληBC/n and small λ, variations in BC are proportional to those in BWO,but with different effective learning rates for each matrix depending on yp. Simulationsshow that actual values are of the same order of approximated ones.THE BAYESIAN ONLINE ALGORITHMThe Bayesian Online Algorithm (BOnA) [8] uses Bayesian inference to adjust ω in theHMM using a data set DP = {y1, ...,yP}. For each data, the prior distribution is updatedby Bayes’ theorem. This update takes a prior from a parametric family and transforms itin a posterior which in general has no longer the same parametric form. The strategy usedby BOnA is then to project the posterior back into the initial parametric family. In orderto achieve this, we minimise the KL-divergence between the posterior and a distributionin the parametric family. This minimisation will enable us to find the parameters of theclosest parametric distribution by which we will approximate our posterior. The studentHMM ω parameters in each step of the learning process are estimated as the means ofthe each projected distribution.For a parametric family that has the form P (x)∝ e−Piλifi(x), which can be obtainedby the MaxEnt principle where we constrain the averages over P (x) of arbitrary func-tions fi(x), minimising the KL-divergence turns out to be equivalent to equating theaverages < fi(x) > over P (x) to the average of these functions over the unprojectedposterior (our posterior distribution just after the Bayesian update for the next data).For HMMs, the vector pi and each i-th row Ai of A and Bi of B are different discretedistributions which we assume independent in order to write the factorized distributionP(ω|u)≡P(pi|ρ)n∏i=1P(Ai|ai)P(Bi|bi), (5)where u= (ρ,a,b) represents the parameters of the distributions.As each factor is a distribution over probabilities, the natural choice are the Dirichletdistributions, which for a N-dimensional variable x isD(x|u) =Γ(u0)∏Ni=1Γ(ui)N∏i=1xui−1i , (6)where u0 =∑iui and Γ is the analytical continuation of the factorial to real numbers.These can be obtained from MaxEnt with fi(x) = lnxi [13]:∫dµD(x) lnxi = αi, dµ≡ δ(∑ixi−1)∏iθ(xi)dxi. (7)The function to be extremized isL=∫dµD lnD+λ(∫dµD−1)+∑iλi(∫dµD lnxi−αi), (8)and with δL/δD = 0 we get the Dirichlet with normalisation eλ+1 and ui = 1−λi.Each factor distribution is separately projected by equating the average of the loga-rithms in the original posterior Q and in the projected distributionsψ(ρi)−ψ(∑jρj)= 〈lnpii〉Q ≡ µi(ρ), (9)ψ(aij)−ψ(∑kaik)= 〈lnAij〉Q ≡ µij(a),ψ(biα)−ψ(∑βbiβ)= 〈lnBiα〉Q ≡ µiα(b),where ψ(x) = d lnΓ(x)/dx is the digamma function. We call a set of N equationsψ(xi)−ψ(∑jxj)= µi, (10)with i= 1, ...N a digamma system in the variables xi with coefficients µi.Let us call P p(ω) the projected distribution after observation of yp, and Qp+1(ω) theposterior distribution (not projected yet) after yp+1. By Bayes’ theorem,Qp+1(ω)∝ P p(ω)∑qp+1P(yp+1, qp+1|ω). (11)The calculation of µ’s in (9) leads to averages over Dirichlets of the form [10]µi =〈[∏jxrjj]lnxi〉=Γ(u0)∏j Γ(uj)∏j Γ(uj+ rj)Γ(u0+ r0)[ψ(ui+ ri)−ψ(u0+ r0)]. (12)To solve (10), we solve for xi, sum over i with x0 ≡∑ixi and find numerically, byiterating from an arbitrary initial point, the fixed points of the one-dimensional mapxn+10 =∑iψ−1[µi+ψ(xn0 )], (13)1 10 100p0.1d (Kullback-Leibler)FIGURE 2. Comparison in log-log scale of MPA (dashed line) and BOnA (circles).where we found a unique solution except for µi ≈ 0, which is rare in most applications.BOnA has a common problem of Bayesian algorithms: the sum over hidden vari-ables makes the complexity scales exponentially in T . Also, the calculation of severaldigamma functions is very time consuming. In the following, we develop an approxi-mation that runs faster, although still with exponential complexity in T . This is not aproblem for we can make T constant and the algorithm will scale polynomially in n.MEAN POSTERIOR APPROXIMATIONThe Mean Posterior Approximation (MPA) is a simplification of BOnA inspired inits results for Gaussians, where we match first and second moments of posterior andprojected distributions. Noting it, instead of minimising dKL we match the mean andone of the variances of posterior and projected distributions as an approximation, whichgives, with hatted variables for reestimated values [10]ρˆi = 〈pii〉Q〈pi1〉Q−〈pi21〉Q〈pi21〉Q−〈pi1〉2Q, (14)aˆij = 〈aij〉Q〈ai1〉Q−〈a2i1〉Q〈a2i1〉Q−〈ai1〉2Q,bˆiα = 〈biα〉Q〈bi1〉Q−〈b2i1〉Q〈b2i1〉Q−〈bi1〉2Q,with complexity again of order nT , but with heavily reduced real computational timemaking it better for practical applications.Figure 2 compares MPA and BOnA. The initial difference decreases in time and bothcome closer relatively fast. We used n = 2, m = 3 and T = 2 and averaged over 150random teachers with symmetric initial students. The computational time for BOnA was340min, and for MPA, 5s in a 1GHz processor. Figure 3a compares MPA to BC andfigure 3b to BWO. In both cases MPA has better generalisation. We used n = 2, m= 3,T = 2, symmetric initial students and averaged over 500 random teachers.10 100 1000 10000p0.1d (Kullback-Leibler)0.0010.010.110 100 1000 10000p0.1d (Kullback-Leibler)0.0050.00010.0005a) b)FIGURE 3. a) Comparison between MPA (dashed) and BC (continuous). Values of λ are indicated nextto the curves. ηBC = 0.5. b) Comparison between MPA (dashed) and BWO (continuous). Values of ηBWare indicated next to the curves. Both scales are log-log.0 500 1000 1500 2000 2500 3000p00.20.40.60.8d (Kullback-Leibler)0 500 1000 1500 2000 2500 3000p0.20.40.60.81d (Kullback-Leibler)a) b)FIGURE 4. Drifting concepts. Continuous lines correspond to MPA and dashed lines to BC. a) Abruptchanges at 500 sequences interval. b) Small random changes at each new sequence.LEARNING DRIFTING CONCEPTSWe tested BC and MPA for changing teachers. In figure 4a, it changes at random aftereach 500 sequences (λ = 0.01, ηBC = 10.0). In figure 4b, each time a sequence isobserved, a small random quantity is added to the teacher. Both have n = 2, m = 3and are averaged over 200 runs.Figure 4b shows that BC adapts better, but is not fully adaptive and we do not knowhow to modify it. MPA instead derives from Bayesian principles and we can guess theproblem by analogy with similar Bayesian algorithms [12]: variances decrease in theprocess as in the perceptron, where they are the learning rates, explaining the memoryeffect difficulting the learning after changes. Although not proved yet, we expect thesame relationship in MPA, which can be used to improve performance.LEARNING AND SYMMETRY BREAKINGLearning from symmetric initial students requires that the parameters separate from eachother in some point, which depends on the algorithm and is an important feature in online0 1000 2000 3000p00.511.522.53d (Kullback-Leibler)0 1000 2000 3000p00.20.40.60.81Initial Probabilities0 1000 2000 3000p00.20.40.60.81Transition Matrix0 1000 2000 3000p00.20.40.60.81Emission Matrix0 1000 2000 3000p00.511.522.53d (Kullback-Leibler)0 1000 2000 3000p00.20.40.60.81Initial probabilities0 1000 2000 3000p00.20.40.60.81Transition Matrix0 1000 2000 3000p00.20.40.60.81Emission Matrixa) b)FIGURE 5. KL-divergence and student’s parameters for a) BC and b) MPA.algorithms [11], breaking the symmetry with a sharp decrease in the generalisation error.Instead of taking averages to smooth abrupt changes, here we draw curves for onlyone teacher, rendering them visible. Flat lines before a symmetry breaking are calledplateaux and occur when it is difficult to break the symmetry.Figure 5a shows BC (λ= 0.01, ηBC = 1.0) with two abrupt changes: in the beginningand after 1000 sequences. pi and A only break the symmetry in the second point, and Bin both. Figure 5b shows that in MPA the second change is stronger and the symmetrybreaking affects both B and A. Figure 6 shows BWO with ηBW = 0.01 where only B isaffected. The more symmetries are broken, the best the generalisation of the algorithm.In all simulations we set n= 2, m= 3 and T = 2 with a teacher HMM given bypi =(10), A =(0 11 0), B =(1 0 00 0 1). (15)CONCLUSIONSWe proposed and analysed three learning algorithms for HMMs: Baum-Welch On-line (BWO), Bayesian Online Algorithm (BOnA) and Mean Posterior Approximation(MPA). We showed the superior performance of MPA for static teachers, but the Baldi-Chauvin (BC) algorithm is better for drifting concepts, although the Bayesian nature ofMPA suggests how to fix it. The results seem to be confirmed by initial tests on real data.The importance of symmetry breaking in learning processes is presented here in abrief discussion where the phenomenon is shown to occur in our models.ACKNOWLEDGEMENTSWe would like to thank Evaldo Oliveira, Manfred Opper and Lehel Csato for usefuldiscussions. This work was made part in the University of São Paulo with financialsupport of FAPESP and part in the Aston University with support of Evergrow Project.0 1000 2000 3000p11.522.53d (Kullback-Leibler)0 1000 2000 3000p0.20.40.60.81Initial probabilities0 1000 2000 3000p0.20.40.60.81Transition Matrix0 1000 2000 3000p00.10.20.30.40.5Emission MatrixFIGURE 6. KL-divergence and student’s parameters for BWO.REFERENCES1. Y. Ephraim, N. Merhav, Hidden Markov Processes. IEEE Trans. Inf. Theory 48, 1518-1569 (2002).2. L. R. Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recogni-tion. Proc. IEEE 77, 257-286 (1989).3. P. Baldi, S. Brunak, Bioinformatics: The Machine Learning Approach. MIT Press (2001).4. R. Durbin, S. Eddy, A. Krogh, G. Mitchison, Biological sequence analysis: Probabilistic models ofproteins and nucleic acids. Cambridge University Press, Cambridge (1998).5. J. Garcia-Frias, Decoding of Low-Density Parity-Check Codes Over Finite-State Binary MarkovChannels. IEEE Trans. Comm. 52, 1840-1843 (2004).6. P. Baldi, Y. Chauvin, Smooth On-Line Learning Algorithms for Hidden Markov Models. NeuralComputation 6, 307-318 (1994).7. S. Amari, Neural learning in structured parameter spaces - Natural Riemannian gradient. NIPS’96 9,MIT Press (1996).8. M. Opper, A Bayesian Approach to On-line Learning. On-line learning in Neural Networks, editedby D. Saad, Publications of the Newton Institute, Cambridge Press, Cambridge (1998).9. M. Opper, D. Saad, Advanced Mean Field Methods: Theory and Practice. MIT Press (2001).10. R. Alamino, N. Caticha, Bayesian Online Algorithms for Learning in Discrete Hidden MarkovModels. Submitted to Discrete and Continuous Dynamical Systems.11. T. Heskes, W. Wiegerinck, W., On-line Learning with Time-Correlated Examples. On-line Learningin Neural Networks, 251-278, edited by David Saad, Cambridge University Press, Cambridge (1998).12. R. Vicente, O. Kinouchi, N. Caticha. Statistical Mechanics of Online Learning of Drifting Concepts:A Variational Approach. Machine Learning 32, 179-201 (1998).13. M. O. Vlad, M. Tsuchiya, P. Oefner, J. Ross. Bayesian analysis of systems with random chemicalcomposition: Renormalization-group approach to Dirichlet distributions and the statistical theory ofdilution. Phys. Rev. E 65, 011112(1)-01112(8) (2001).

Online learning in discrete hidden Markov models

https://publications.aston.ac.uk/id/eprint/39029/1/0708.2377.pdf

Online Learning in Discrete Hidden Markov Models

Abstract

Similar works

Full text

Available Versions

Crossref

Crossref

Aston Publications Explorer