Hidden Markov Models (HMMs) are learning methods for pattern recognition. The
probabilistic HMMs have been one of the most used techniques based on the
Bayesian model. First-order probabilistic HMMs were adapted to the theory of
belief functions such that Bayesian probabilities were replaced with mass
functions. In this paper, we present a second-order Hidden Markov Model using
belief functions. Previous works in belief HMMs have been focused on the
first-order HMMs. We extend them to the second-order model

A. Aregui

D. Dubois

E. Ramasso

F. Fayad

G. Zhou

H. Soubaras

J. Kupiec

L.M. Lee

L.R. Rabiner

M.E.Y. Boudaren

P. Lanchantin

P. Smets

R.R. Yager

S.M. Thede

T. Brants

W. Pieczynski

English

arXiv

International audienceHidden Markov Models (HMMs) are learning methods for pattern recognition. The probabilistic HMMs have been one of the most used techniques based on the Bayesian model. First-order probabilistic HMMs were adapted to the theory of belief functions such that Bayesian probabilities were replaced with mass functions. In this paper, we present a second-order Hidden Markov Model using belief functions. Previous works in belief HMMs have been focused on the first-order HMMs. We extend them to the second-order model

Park, Jungyeul

Chebbah, Mouna

Jendoubi, Siwar

Martin, Arnaud

HAL-Rennes 1

Second-Order Belief Hidden Markov ModelsJungyeul Park, Mouna Chebbah, Siwar Jendoubi, Arnaud MartinTo cite this version:Jungyeul Park, Mouna Chebbah, Siwar Jendoubi, Arnaud Martin. Second-Order Belief Hid-den Markov Models. Belief 2014, Sep 2014, Oxford, United Kingdom. pp.284 - 293, 2014,<10.1007/978-3-319-11191-9 31>. <hal-01108238>HAL Id: hal-01108238https://hal.archives-ouvertes.fr/hal-01108238Submitted on 22 Jan 2015HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Second-order Belief Hidden Markov ModelsJungyeul Park1, Mouna Chebbah1,2 and Siwar Jendoubi1,2, Arnaud Martin11 UMR 6074 IRISA, Université de Rennes1, Lannion, France2 LARODEC Laboratory, University of Tunis, ISG Tunis, Tunisia{jungyeul.park,mouna.chebbah,arnaud.martin}@univ-rennes1.frjendoubi.siwar@etudiant.univ-rennes1.frAbstract. Hidden Markov Models (HMMs) are learning methods for patternrecognition. The probabilistic HMMs have been one of the most used techniquesbased on the Bayesian model. First-order probabilistic HMMs were adapted tothe theory of belief functions such that Bayesian probabilities were replaced withmass functions. In this paper, we present a second-order Hidden Markov Modelusing belief functions. Previous works in belief HMMs have been focused on thefirst-order HMMs. We extend them to the second-order model.Keywords: Belief functions, Dempster-Shafer theory, first-order belief HMM,second-order belief HMM, probabilistic HMM1 IntroductionA Hidden Markov Model (HMM) is one of the most important statistical models in ma-chine learning [1]. A HMM is a classifier or labeler that can assign label or class to eachunit in a sequence [2]. It has been successfully utilized over several decades in manyapplications for processing text and speech such as Part-of-Speech (POS) tagging [3],named entity recognition [4] and speech recognition [5]. However, such works in theearly part of the period are mainly based on first-order HMMs. As a matter of fact, theassumption in the first-order HMM, where the state transition and output observationdepend only on one previous state, does not exactly match with the real applications[6]. Therefore, they require a number of sophistications. For example, even though thefirst-order HMM for POS tagging in early 1990s performs reasonably well, it capturesa more limited amount of the contextual information than is available [7]. As conse-quence, most modern statistical POS taggers use a second-order model [8].Uncertainty theories can be integrated in statistical models such as HMMs: Theprobability theory has been used to classify units in a sequence with the Bayesianmodel. Then, the theory of belief functions is employed to this statistical model becausethe fusion proposed in this theory simplifies computations of a posteriori distributionsof hidden data in Markov models. This theory can provide rules to combine evidencesfrom different sources to reach a certain level of belief [9–13]. Belief HMMs intro-duced in [14–22], use combination rules proposed in the framework of the theory ofbelief functions. This paper is an extension of previous ideas for second-order beliefHMMs. For the current work, we focus on explaining a second-order model. However,the proposed method can be easily extended to higher-order models.2 Park et al.This paper is organized as follows: In Sections 2 and 3, we detail probabilisticHMMs for the problem of POS tagging where HMMs have been widely used. Then,we describe the first-order belief HMM in Section 4. Finally, before concluding, wepropose the second-order belief HMM.2 First-order probabilistic HMMsPOS tagging is a task of finding the most probable estimated sequence of n tags giventhe observation sequence of v words. According to [1], a first-order probabilistic HMMcan be characterized as follows:N The number of states in a model St = {st1,st2, · · ·stN} at the time t.M The number of distinct observation symbols. V = {v1,v2, · · · ,vM}.A = {ai j} The set of N transition probability distributions.B = {b j(ot)} The observation probability distributions in state j.pi = {pii} The initial probability distribution.Figure 1 illustrates the first-order probabilistic HMM allowing to estimate the prob-ability of the sequence st−1i and stj where ai j is the transition probability from st−1i tostj and b j(ot) is the observation probability on the state stj. Regarding POS tagging, thenumber of possible POS tags that are hidden states St of the HMM is N. The numberof words in the lexicons V is M. The transition probability ai j is the probability that themodel moves from one tag st−1i to another tag stj. This probability can be estimated us-ing a training data set in supervised learning for the HMM. The probability of a currentPOS tag appearing in the first-order HMM depends only on the previous tag. In general,first-order probabilistic HMMs should be characterized by three fundamental problemsas follows [1]:– Likelihood: Given a set of transition probability distributions A, an observation se-quence O = o1,o2, · · · ,oT and its observation probability distribution B, how do wedetermine the likelihood P(O|A,B)? The first-order model relies on only one obser-vation where b j(ot) = P(o j|stj) and the transition probability based on one previoustag where ai j = P(stj|st−1i ). Using the forward path probability, the likelihood αt( j)of a given state stj can be computed by using the likelihood αt−1(i) of the previousstate st−1i as described below:αt( j) = ∑iαt−1(i)ai jb j(ot) (1)– Decoding: Given a set of transition probability distributions A, an observation se-quence O = o1,o2, · · · ,oT and its observation probability distribution B, how dowe discover the best hidden state sequence? The Viterbi algorithm is widely usedfor calculating the most likely tag sequence for the decoding problem. The Viterbialgorithm can calculate the most probable path δt( j) which contains the sequenceof ψt( j). It can select the path that maximizes the likelihood of the sequence asdescribed below:Second-order Belief HMMs 3δt( j) =maxδt−1(i)ai jb j(ot)ψt( j) = argmaxψt−1(i)ai j(2)– Learning: Given an observation sequence O = o1,o2, · · · ,oT and a set of states S ={st1,st2, · · · ,stN}, how do we learn the HMM parameters for A and B? The parameterlearning task usually uses the Baum-Welch algorithm which is a special case of theExpectation-Maximization (EM) algorithm.In this paper, we focus on the likelihood and decoding problems by assuming asupervised learning paradigm where labeled training data are already available.3 Second-order probabilistic HMMsNow, we explain the extension of the first-order model to a trigram3 in the second-ordermodel. Figure 2 illustrates the second-order probabilistic HMM allowing to estimate theprobability of the sequence of three states st−2i , st−1j and stk where ai jk is the transitionprobability from st−2i and st−1j to stk, and bk(ot) is the observation probability on thestate stk. Therefore, the second-order probabilistic HMM is characterized by three fun-damental problems as follows:– Likelihood: The second-order model relies on one observation bk(ot). Unlike thefirst-order model, the transition probability is based on two previous tags where ai jk= P(stk|st−2i , st−1j ) as described below:αt(k) = ∑jαt−1( j)ai jkbk(ot) (3)However, it will be more difficult to find a sequence of three tags than a sequenceof two tags. Any particular sequence of tags st−2i , st−1j , stk that occurs in the testset may simply never have occurred in the training set because of data sparsity[2]. Therefore, a method for estimating P(stk|st−2i ,st−1j ), even if the sequence st−2i ,st−1j , stk never occurs, is required. The simplest method to solve this problem is tocombine the trigram Pˆ(stk|st−2i ,st−1j ), the bigram Pˆ(stk|st−1j ), and even the unigramPˆ(stk) probabilities [8]:P(stk|st−2i ,st−1j ) = λ1Pˆ(stk|st−2i ,st−1j )+λ2Pˆ(stk|st−1j )+λ3Pˆ(stk) (4)Note that Pˆ is the maximum likelihood probabilities which are derived from therelative frequencies of the sequence of tags. Values of λ are such that λ1+λ2+λ3 =1 and they can be estimated by the deleted interpolation algorithm [8]. Otherwise,[7] describes a different method for values of λ as below:λ1 = k3λ2 = (1− k3) · k2λ3 = (1− k3) · (1− k2)(5)3 The trigram is the sequence of three elements, i.e. three states in our case.4 Park et al.where k2 =log(C(st−1j ,stk)+1)+1log(C(st−1j ,stk)+1)+2, k3 =log(C(st−2i ,st−1j ,stk)+1)+1log(C(st−2i ,st−1j ,stk)+1)+2, andC(st−2i ,st−1j ,stk) is thefrequency of a sequence st−2i ,st−1j ,stk in the training data. Note that λ1+λ2+λ3 isnot always equal to one in [7]. The likelihood of the observation probability for thesecond-order model uses B where bk(ot) = P(ok|stk,st−1j ).– Decoding: For second-order model we require a different Viterbi algorithm. For agiven state s at the time t, it would be redefined as follows [7]:δt(k) =maxδt−1( j)ai jkbk(ot)where δt( j) =maxP(s1,s2, · · · ,st−1 = si,st = s j,o1,o2, · · · ,ot)ψt(k) = argmaxψt−1( j)ai jkwhere ψt(k) = argmaxP(s1,s2, · · · ,st−1 = si,st = s j,o1,o2, · · · ,ot)(6)– Learning: The problem of learning would be similar to the first-order model exceptthat parameters A and B are different.With respect to performance measures, different transition probability distributionsin [8] and [7] obtain 97.0% and 97.09% tagging accuracy for known words, respec-tively for the same data (the Penn Treebank corpus). Even though probabilistic HMMsperform reasonably well, belief HMMs can learn better under certain conditions onobservations [21].4 First-order Belief HMMsIn probabilistic HMMs, A and B are probabilities estimated from the training data. How-ever, A and B in belief HMMs are mass functions (bbas) [17, 21].According to previousworks on belief HMMs, a first-order HMM using belief functions can be characterizedas follows4:N The number of states in a model Ωt = {St1,St2, · · · ,StN}.M The number of distinct observation symbols V .A = {mΩta [St−1i ](Stj)} The set of conditional bbas to all possible subsets of states.B = {mΩtb [ot ](Stj)} The set of bbas according to all possible observations Ot .pi = {mΩ1pi (SΩ1i )} The bba defined for the the initial state.Difference between the first-order probabilistic and belief HMMs is presented inFigure 1, the transition and observation probabilities in belief HMMs are described asmass functions. Therefore, we can replace ai j by mΩta [St−1i ](Stj) and b j(ot) by mΩtb [ot ](Stj).The set Ωt has been used to denote states for HMMs using belief functions [17, 21].Note that sti is the single state for probabilistic HMMs and Sti is the multi-valued statefor belief HMMs. First-order belief HMMs should also be characterized by three fun-damental problems as follows:– Likelihood: The likelihood problem in belief HMMs is not solved by likelihood,but by using the combination. The first-order belief model relies on (i) only oneobservation mΩtb [ot ](Stj) and (ii) a transition conditional mass function based on4 In the model Ωt , St are focal elementsSecond-order Belief HMMs 5one previous tag mΩta [St−1i ](Stj). Mass functions of sets A and B are combined usingthe Disjunctive Rule of Combination (DRC) for the forward propagation and theGeneralized Bayesian Theorem (GBT) for the backward propagation [23]. Usingthe forward path propagation, the mass function of a given state Stj can be com-puted as the combination of mass functions on the observation and the transition asdescribed below:qΩtα (Stj) = ∑mΩt−1α (St−1i ) ·qΩta [St−1i ](Stj) ·qΩtb (Stj) (7)Note that the mass function of the given state Stj is derived from the commonalityfunction qΩtα .– Decoding: Several solutions have been proposed to extend the Viterbi algorithm tothe theory of belief functions [17, 24, 25]. Such solutions maximize the plausibil-ity of the state sequence. In fact, the credal Viterbi algorithm starts from the firstobservation and estimates the commonality distribution of each observation untilreaching the last state. For each state Stj, the estimated commonality distribution(qΩtδ(Stj)) is converted back to a mass function that is conditioned on the previousstate. Then, we apply the pignistic transform to make a decision about the currentstate (ψt(stj)):qΩtδ(Stj) = ∑St−1i ⊆At−1mΩt−1δ(St−1i ) ·qΩta [St−1i ](Stj) ·qΩtb (Stj)ψt(stj) = argmaxSt−1i ∈Ωt−1(1−mΩtδ[St−1i ]( /0)) ·Pt [St−1i ](Stj)(8)where At = ∪St−1j ∈Ωtψt(Stj) [17].– Learning: Instead of the traditional EM algorithm, we can use the E2M algorithmfor the belief HMM [22].To build belief functions from what we learned using probabilities in the previoussection, we can employ the least commitment principle by using the inverse pignistictransform [26, 27].5 Second-order Belief HMMsLike the first-order belief HMM, N, M, B and pi are similarly defined in the second-orderHMM. The set A is quite different and is defined as follows:A = {mΩta [St−2i ,St−1j ](Stk)} (9)where A is the set of conditional bbas to all possible subsets of states based on thetwo previous states. Second-order belief HMMs should also be characterized by threefundamental problems as follows:– Likelihood: The second-order belief model relies on one observation mΩtb [ot ](Stk)in a state Sk at time t and the transition conditional mass function based on two pre-vious states St−2i and St−1j , defined by mΩta [St−2i ,St−1j ](Stk). Using the forward path6 Park et al.propagation, the mass function of a given state Stk can be computed as the disjunc-tive combination (DRC) of mass functions on the transition mΩta [St−2i ,St−1j ](Stk) andthe observation mΩtb (Stk) as described below:qΩtα (Stk) = ∑mΩt−1α (St−1j ) ·qΩta [St−2i ,St−1j ](Stk) ·qΩtb (Stk) (10)where qΩta [St−2i ,St−1j ](Stk) is the commonality function derived from the conjunctivecombination of mass functions of two previous transitions. The conjunctive combi-nation is used to have the conjunction of observations on previous two states St−2iand St−1j .The combined mass function mΩta [St−2i ,St−1j ](Stk) of two transitions mΩt−1a [St−2i ](St−1j )and mΩta [St−1j ](Stk) is defined as follows:mΩta [St−2i ,St−1j ](Stk) = mΩt−1a [St−2i ](St−1j ) ∪© mΩta [St−1j ](Stk) (11)The conjunctive combination is required to obtain the conjunction of both transi-tions. Note that the mass function of the given state Stk is derived from the com-monality function qΩtα . We use DRC with commonality functions like in [17]. Notethat the observation only on one previous state is taken into account in the first-order belief HMM, but the conjunction of observations on two previous states isconsidered in the second-order belief HMM.– Decoding: We accept our assumption of the first-order belief HMM for the second-order model. Similarly to the first-order belief HMM, we propose a solution thatmaximizes the plausibility of the state sequence. The credal Viterbi algorithm es-timates the commonality distribution of each observation from the first observa-tion till the final state. For each state Stk, the estimated commonality distribution(qΩtδ(Stk)) is converted back to a mass function that is conditioned on a mass func-tion of the two previous states. This mass function is the conjunctive combinationof mass functions of the two previous states. Then, we apply the pignistic transformto make a decision about the current state (ψt(stj)) as before:qΩtδ(Stk) = ∑St−1j ⊆At−1mΩt−1δ(St−1j ) ·qΩta [St−2i ,St−1j ](Stk) ·qΩtb (Stk)ψt(stk) = argmaxSt−1j ∈Ωt−1(1−mΩtδ[St−1j ]( /0)) ·Pt [St−2i ,St−1j ](Stk)(12)– Learning: Like the first-order belief model, we can still use the E2M algorithm forthe belief HMM [22].Since the combination of mass functions in the belief HMM is required where theprevious observation is already considered in the set of conditional bbas mΩta [St−2i ,St−1j ],we do not need to refine the observation probability for the second-order model as inthe second-order probabilistic model.6 Conclusion and future perspectivesThe problem of POS tagging has been considered as one of the most important tasksfor natural language processing systems. We described such a problem based on HMMsSecond-order Belief HMMs 7st−1i stjbj(ot)aijSt−1i StjmΩtb [ot]mΩta [St−1i ]Fig. 1. First-order probabilistic and belief HMMsst−2i st−1j stkbk(ot)aij ajkaijkSt−2i St−1j StkmΩtb [ot]mΩta [St−1j ]mΩt−1a [St−2i ]mΩta [St−2i , St−1j ]Fig. 2. Second-order probabilistic and belief HMMsand tried to apply our idea to the theory of belief functions. We extended previous workson belief HMMs to the second-order model. Using the proposed method, we will be ableto easily extend the higher-order model for belief HMMs. Some technical aspects stillremain to be considered. Robust implementation for belief HMMs are required wherein general we can find over one million observations in the training data to deal with theproblem of POS tagging. As described before, the choice of inverse pignistic transformswould be empirically verified.5 We are planning to implement these technical aspectsin near future.The current work is described to rely on a supervised learning paradigm from la-beled training data. Actually, the forward-backward algorithm in HMMs can do com-pletely unsupervised learning. However, it is well known that EM performs poorly inunsupervised induction of linguistic structure because it tends to assign relatively equalnumbers of tokens to each hidden state [29].6 Therefore, the initial conditions can bevery important. Since the theory of belief functions can take into consideration of uncer-tainty and imprecision, especially for the lack of data, we might obtain a better modelusing belief functions on an unsupervised learning paradigm.References5 For example, [28] used the inverse pignistic transform in [26] to calculate belief functionsfrom Bayesian probability functions. As matter of fact, the problem of POS tagging can benormalized and inverse pignistic transforms in [26] did not propose the case for m( /0).6 The actual distribution of POS tags would be highly skewed as in heavy-tail distributions.

Second-Order Belief Hidden Markov Models

Abstract

Similar works

Full text

Available Versions

HAL-Rennes 1

Crossref

HAL-CentraleSupelec

HAL: Hyper Article en Ligne