The purpose of this note is to study the assumption of mutual information independence", which is used by Zhou (2005) for deriving an output-dependent hidden Markov model, the so-called discriminative HMM (D-HMM), in the context of determining a stochastic optimal sequence of hidden states. The assumption is extended to derive its generative counterpart, the G-HMM. In addition, state-dependent representations for two output-dependent HMMs, namely HMMSDO (Li, 2005) and D-HMM, are presented

D. Michael Titterington

Jing-Hao Xue

Zhou

English

Xue, J.H.

Titterington, D.M.

Enlighten: Publications

 
 
 
 
 
 
 
Xue, J.H. and Titterington, D.M. (2008) Short note on two output-
dependent hidden Markov models. Pattern Recognition Letters, 29 (9) pp. 
1424-1426. ISSN 0167-8655 
 
 
http://eprints.gla.ac.uk/6835/ 
 
Deposited on: 13 August 2009 
 
 
Enlighten – Research publications by members of the University of Glasgow 
http://eprints.gla.ac.uk 
Short Note on Two Output-dependent Hidden1
Markov Models2
Jing-Hao Xue a,∗, D. Michael Titterington a3
aDepartment of Statistics, University of Glasgow, Glasgow G12 8QQ, UK4
Abstract5
The purpose of this note is to study the assumption of “mutual information inde-6
pendence”, which is used by Zhou (2005) for deriving an output-dependent hidden7
Markov model, the so-called discriminative HMM (D-HMM), in the context of deter-8
mining a stochastic optimal sequence of hidden states. The assumption is extended9
to derive its generative counterpart, the G-HMM. In addition, state-dependent rep-10
resentations for two output-dependent HMMs, namely HMMSDO (Li, 2005) and11
D-HMM, are presented.12
Key words: Discriminative models; Generative models; Mutual information13
independence; Output-dependent hidden Markov model14
∗ Corresponding author. Tel.: +44 141 330 2474; fax: +44 141 330 4814.
Email addresses: jinghao@stats.gla.ac.uk (Jing-Hao Xue),
mike@stats.gla.ac.uk (D. Michael Titterington).
Preprint submitted to Pattern Recognition Letters 19th July 2007
1 Introduction1
Generative models like normal-based discriminant analysis and discriminative2
models like logistic regression are comprehensively investigated and compared3
in the machine learning literature (Rubinstein and Hastie, 1997; Ng and Jor-4
dan, 2001). Amongst the latent (hidden) variable models for structured data5
such as time series, hidden Markov models (HMMs) for discrete-valued hidden6
states and state-space models (SSMs) for continuous-valued hidden states are7
widely used.8
Traditionally, an HMM is generative because it models a distribution P (On1 |Sn1 ),9
the data generation process (DGP) of the observed output sequence, On1 =10
o1, . . . , on, given the hidden state sequence, S
n
1 = s1, . . . , sn, and thus P (O
n
1 |Sn1 ),11
a state-dependent term, is included in the criterion for determining a sto-12
chastic optimal sequence of hidden states. Recently, Zhou (2005) proposes13
a discriminative hidden Markov model (D-HMM), which includes output-14
dependent terms P (st|On1 ), t = 1, · · · , n, in the criterion, based on an assump-15
tion of “mutual information independence”. Meanwhile, Li (2005) presents16
the so-called “hidden Markov models with states depending on observations”17
(HMMSDO), which assumes that the current state st depends not only on18
the last state st−1 but also on the last output ot−1, so that output-dependent19
terms P (st|st−1, ot−1) are included in the criterion.20
Both the D-HMM and HMMSDO show superior performance in determining21
the optimal state sequence for certain applications. Zhou (2005) shows that the22
D-HMM outperforms the corresponding generative hidden Markov model (G-23
HMM) for part-of-speech tagging and phrase chunking; Li (2005) shows that24
2
HMMSDO outperforms the standard HMM for prediction of protein secondary1
structures when the training set is large enough.2
In this note, we shall study the assumption of “mutual information indepen-3
dence” that is used for deriving the D-HMM (Zhou, 2005) in the context of4
determining an optimal state sequence, and then extend it to derive its gener-5
ative counterpart, the G-HMM. In addition, state-dependent representations6
for these two output-dependent HMMs will be presented.7
2 Generative HMM8
Following the notation used by Zhou (2005), the definition of the optimal9
hidden state sequence Sn1 based on the observed output sequence O
n
1 is that10
of the maximum a posteriori (MAP) estimator S∗ of Sn1 :11
S∗ = argmax
Sn1
{logP (Sn1 |On1 )} . (1)
The G-HMM rewrites the criterion (1) through applying Bayes’ theorem and12
ignoring the item determined purely by On1 as13
S∗ = argmax
Sn1
{logP (Sn1 ) + logP (On1 |Sn1 )} ,
which is further factorised as14
S∗ = argmax
Sn1
{
logP (Sn1 ) + log
(
P (o1|Sn1 )
n∏
k=2
P (ok|Ok−11 , Sn1 )
)}
.
In order to make this formulation tractable, an assumption that On1 is condi-15
tionally independent given Sn1 is in general introduced as, for all k ∈ {2, . . . , n},16
P (ok|Ok−11 , Sn1 ) = P (ok|Sn1 ) , (2)
3
and thus based on such a conditional independence assumption, the MAP1
estimator for the G-HMM is simplified to2
S∗ = argmax
Sn1
{
logP (Sn1 ) +
n∑
i=1
logP (oi|Sn1 )
}
. (3)
The G-HMM is regarded as being generative because it directly models the3
DGP P (oi|Sn1 ) of the observed oi from the hidden Sn1 .4
In practice, as for the standard HMM, the assumption (2) is further simplified5
to6
P (ok|Ok−11 , Sn1 ) = P (ok|Sn1 ) = P (ok|sk) , (4)
and thus the MAP estimator of the standard HMM is7
S∗ = argmax
Sn1
{
logP (Sn1 ) +
n∑
i=1
logP (oi|si)
}
. (5)
3 Discriminative HMM from Mutual Information Independence8
The D-HMM rewrites the criterion (1) through applying Bayes’ theorem, but9
not ignoring the item determined purely by On1 , as10
S∗ = argmax
Sn1
{
logP (Sn1 ) + log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
}
.
To make this formulation tractable, an assumption that the mutual informa-11
tion (MI (Sn1 , O
n
1 ) = log
P (Sn1 ,O
n
1 )
P (Sn1 )P (O
n
1 )
) between Sn1 and O
n
1 is independent with12
respect to each hidden si was introduced by Zhou (2005) as13
MI(Sn1 , O
n
1 ) =
n∑
i=1
MI(si, O
n
1 ) , (6)
or, in more detail,14
log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
=
n∑
i=1
log
P (si, O
n
1 )
P (si)P (On1 )
=
n∑
i=1
log
P (si|On1 )
P (si)
. (7)
4
Based on such a representation, the MAP estimator for the D-HMM is sim-1
plified as (Zhou, 2005)2
S∗ = argmax
Sn1
{
logP (Sn1 ) +
n∑
i=1
logP (si|On1 )−
n∑
i=1
logP (si)
}
. (8)
The D-HMM is regarded as being discriminative because the criterion (8)3
includes directly the discriminative process P (si|On1 ), representing an output-4
dependence of a hidden state si on all the observed outputs O
n
1 .5
We shall make four observations about the D-HMM.6
First, it is noted that the criterion (8) is simultaneously to maximise the max-7
imum posterior marginal (MPM) estimator
∑n
i=1 logP (si|On1 ) of logP (Sn1 |On1 )8
and to maximise the distance between the state transition model logP (Sn1 )9
and its independent-based counterpart
∑n
i=1 logP (si).10
Second, in order to satisfy the assumption (7) underlying the D-HMM, it is11
required that12
n∏
k=2
P (sk|Sk−11 , On1 )
P (sk|Sk−11 )
=
n∏
k=2
P (sk|On1 )
P (sk)
.
Since this is valid for any value of sk, it follows that, for all k ∈ {2, . . . , n},13
P (sk|Sk−11 , On1 )
P (sk|Sk−11 )
=
P (sk|On1 )
P (sk)
. (9)
Third, the assumption (7) can be rewritten as14
log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
=
n∑
i=1
log
P (si, O
n
1 )
P (si)P (On1 )
=
n∑
i=1
log
P (On1 |si)
P (On1 )
. (10)
Based on such a representation, the MAP estimator (8) for the D-HMM can15
be rewritten, with the term
∑n
i=1 logP (O
n
1 ) determined purely by O
n
1 being16
ignored, as17
S∗ = argmax
Sn1
{
logP (Sn1 ) +
n∑
i=1
logP (On1 |si)
}
. (11)
5
Therefore, the D-HMM can also be represented as being generative because1
the criterion (11) includes a generative-like process P (On1 |si), representing a2
state-dependence of all the observed outputs On1 on a hidden state si.3
Fourth, it can be seen that, when the assumption (6) of mutual information4
independence develops from independence between pairs (si, O
n
1 ) into that be-5
tween local pairs (si, oi) such that MI(S
n
1 , O
n
1 ) =
∑n
i=1MI(si, oi), the criteria6
(11) and (8) degenerate into the criterion (5), indicating that the D-HMM7
degenerates into the standard HMM.8
4 Generative HMM from Mutual Information Independence9
Furthermore, similarly to the assumption (6) proposed by Zhou (2005), an10
assumption that mutual information between Sn1 and O
n
1 is independent with11
respect to each observed oi can be introduced here as12
MI(Sn1 , O
n
1 ) =
n∑
i=1
MI(Sn1 , oi) , (12)
or, in more detail,13
log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
=
n∑
i=1
log
P (Sn1 , oi)
P (Sn1 )P (oi)
=
n∑
i=1
log
P (oi|Sn1 )
P (oi)
. (13)
Based on such a representation, we can obtain another generative model and14
its MAP estimator, with the term
∑n
i=1 logP (oi) determined purely by O
n
115
being ignored, as16
S∗ = argmax
Sn1
{
logP (Sn1 ) +
n∑
i=1
logP (oi|Sn1 )
}
. (14)
This estimator is in fact the estimator (3) of the G-HMM, i.e., the G-HMM17
can be derived under the assumption (12), a type of mutual information in-18
6
dependence.1
Similarly, we shall make three observations about this G-HMM, which is de-2
rived from mutual information independence.3
First, in order to satisfy the assumption (13) of the G-HMM, it is required4
that, for all k ∈ {2, . . . , n},5
P (ok|Ok−11 , Sn1 )
P (ok|Ok−11 )
=
P (ok|Sn1 )
P (ok)
. (15)
Therefore, under the MAP criterion (1), the conditions (15) and (2) have the6
same effect on determining the optimal hidden Sn1 .7
Second, the assumption (13) can be rewritten as8
log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
=
n∑
i=1
log
P (Sn1 , oi)
P (Sn1 )P (oi)
=
n∑
i=1
log
P (Sn1 |oi)
P (Sn1 )
. (16)
Based on such a representation, the MAP estimator (14) for the G-HMM can9
be rewritten, with the terms related to logP (Sn1 ) being combined, as10
S∗ = argmax
Sn1
{
(1− n) logP (Sn1 ) +
n∑
i=1
logP (Sn1 |oi)
}
. (17)
Therefore, in this sense, the G-HMM can also be represented as being dis-11
criminative because the criterion (17) includes a discriminative-like process12
P (Sn1 |oi), representing an output-dependence of all the hidden states Sn1 on13
an observed output oi.14
Third, it can be seen that, when the assumption (12) of mutual information15
independence develops from independence between pairs (Sn1 , oi) into that be-16
tween local pairs (si, oi) such that MI(S
n
1 , O
n
1 ) =
∑n
i=1MI(si, oi), the criteria17
(17) and (14) degenerate into the criterion (5), indicating that the G-HMM18
degenerates into the standard HMM.19
7
5 Equivalence between G-HMM and D-HMM1
Once we assume a fully independent mutual information between any state-2
output combination (si, oj) as3
MI(Sn1 , O
n
1 ) =
n∑
i=1
n∑
j=1
MI(si, oj) , (18)
or, in more detail,4
log
P (Sn1 , O
n
1 )
P (Sn1 )P (O
n
1 )
=
n∑
i=1
n∑
j=1
log
P (si, oj)
P (si)P (oj)
=
n∑
i=1
n∑
j=1
log
P (oj|si)
P (oj)
=
n∑
i=1
n∑
j=1
log
P (si|oj)
P (si)
,
(19)
this assumption results in two criteria, one generative and the other discrimi-5
native, with the MAP estimators as6
S∗ = argmax
Sn1
{logP (Sn1 ) +
n∑
i=1
n∑
j=1
logP (oj|si)} , (20)
7
S∗ = argmax
Sn1
logP (Sn1 ) +
n∑
i=1
n∑
j=1
logP (si|oj)−
n∑
i=1
{n logP (si)}
 , (21)
respectively. These two criteria are equivalent.8
In the context of determining an optimal sequence of hidden states, apart9
from the equivalence above, up to now, we find two occurrences of equivalence10
between a discriminative representation of the MAP criterion and its genera-11
tive counterpart: one is for the D-HMM between the criteria (8) and (11), the12
other is for the G-HMM between the criteria (17) and (14).13
We shall further illustrate such equivalence with two simple but related HMMs:14
one is a generative-like state-dependent model, which assumes that the current15
output ot depends not only on the current state st but also on the last state16
st−1; the other is a discriminative-like output-dependent model, the so-called17
8
HMMSDO (Li, 2005), which assumes that the current state st depends not1
only on the last state st−1 but also on the last output ot−1.2
The joint distribution of the first generative-like state-dependent model is3
P (Sn1 , O
n
1 ) = P (s1)P (o1|s1)
n∏
i=2
P (si|si−1)P (oi|si, si−1) . (22)
This distribution can be rewritten as4
P (Sn1 , O
n
1 ) = P (o1, s1)
n∏
i=2
P (si, oi|si−1)
= P (o1)P (s1|o1)
n∏
i=2
P (oi|si−1)P (si|si−1, oi) ,
(23)
which leads to a discriminative-like output-dependent part P (si|si−1, oi) in the5
distribution. In fact, the difference between the probabilistic directed acyclic6
graphs (DAGs) corresponding to the joint distributions (22) and (23) is only7
in that directions of edges from si to oi are reversed.8
Similarly, the joint distribution of the discriminative-like output-dependent9
HMMSDO, with P (si|si−1, oi−1) included, is (Li, 2005)10
P (Sn1 , O
n
1 ) = P (s1)P (o1|s1)
n∏
i=2
P (si|si−1, oi−1)P (oi|si) . (24)
This distribution can be rewritten as11
P (Sn1 , O
n
1 ) = P (s1)P (on|sn)
n∏
i=2
P (si, oi−1|si−1)
= P (s1)P (on|sn)
n∏
i=2
P (si|si−1)P (oi−1|si, si−1) ,
(25)
which leads to a no longer discriminative-like output-dependence in the dis-12
tribution. In fact, the difference between the DAGs corresponding to the joint13
distributions (24) and (25) is only in that directions of edges from si to oi−114
are reversed. In practice, whether or not P (oi−1|si, si−1) is reasonable needs15
to be justified, because it means that the current output depends on the next16
9
state.1
6 Summary2
This note has suggested that the mutual information assumption (12) resulted3
in the G-HMM, while another mutual information assumption (6) resulted in4
the D-HMM. However, in practice, whether or not the assumptions are reason-5
able and how the corresponding HMMs perform can be data-dependent; re-6
search efforts to explore an adaptive switching between or combination of these7
two models may be worthwhile. Meanwhile, this note has suggested that the8
so-called output-dependent HMMs could be represented in a state-dependent9
manner, and vice versa, essentially by application of Bayes’ theorem.10
References11
Li, Y., 2005. Hidden Markov models with states depending on observations.12
Pattern Recognition Letters 26 (7), 977–984.13
Ng, A. Y., Jordan, M. I., 2001. On discriminative vs. generative classifiers: a14
comparison of logistic regression and na¨ıve bayes. In: NIPS. pp. 841–848.15
Rubinstein, Y. D., Hastie, T., 1997. Discriminative vs. informative learning.16
In: KDD. pp. 49–53.17
Zhou, G. D., 2005. Direct modelling of output context dependence in discrim-18
inative hidden Markov model. Pattern Recognition Letters 26 (5), 545–553.19
10


Short note on two output-dependent hidden Markov models

Crossref

Enlighten

       Xue, J.H. and Titterington, D.M. (2008) Short note on two output-dependent hidden Markov models. Pattern Recognition Letters, 29 (9) pp. 1424-1426. ISSN 0167-8655   http://eprints.gla.ac.uk/6835/  Deposited on: 13 August 2009   Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Short Note on Two Output-dependent Hidden1Markov Models2Jing-Hao Xue a,∗, D. Michael Titterington a3aDepartment of Statistics, University of Glasgow, Glasgow G12 8QQ, UK4Abstract5The purpose of this note is to study the assumption of “mutual information inde-6pendence”, which is used by Zhou (2005) for deriving an output-dependent hidden7Markov model, the so-called discriminative HMM (D-HMM), in the context of deter-8mining a stochastic optimal sequence of hidden states. The assumption is extended9to derive its generative counterpart, the G-HMM. In addition, state-dependent rep-10resentations for two output-dependent HMMs, namely HMMSDO (Li, 2005) and11D-HMM, are presented.12Key words: Discriminative models; Generative models; Mutual information13independence; Output-dependent hidden Markov model14∗ Corresponding author. Tel.: +44 141 330 2474; fax: +44 141 330 4814.Email addresses: jinghao@stats.gla.ac.uk (Jing-Hao Xue),mike@stats.gla.ac.uk (D. Michael Titterington).Preprint submitted to Pattern Recognition Letters 19th July 20071 Introduction1Generative models like normal-based discriminant analysis and discriminative2models like logistic regression are comprehensively investigated and compared3in the machine learning literature (Rubinstein and Hastie, 1997; Ng and Jor-4dan, 2001). Amongst the latent (hidden) variable models for structured data5such as time series, hidden Markov models (HMMs) for discrete-valued hidden6states and state-space models (SSMs) for continuous-valued hidden states are7widely used.8Traditionally, an HMM is generative because it models a distribution P (On1 |Sn1 ),9the data generation process (DGP) of the observed output sequence, On1 =10o1, . . . , on, given the hidden state sequence, Sn1 = s1, . . . , sn, and thus P (On1 |Sn1 ),11a state-dependent term, is included in the criterion for determining a sto-12chastic optimal sequence of hidden states. Recently, Zhou (2005) proposes13a discriminative hidden Markov model (D-HMM), which includes output-14dependent terms P (st|On1 ), t = 1, · · · , n, in the criterion, based on an assump-15tion of “mutual information independence”. Meanwhile, Li (2005) presents16the so-called “hidden Markov models with states depending on observations”17(HMMSDO), which assumes that the current state st depends not only on18the last state st−1 but also on the last output ot−1, so that output-dependent19terms P (st|st−1, ot−1) are included in the criterion.20Both the D-HMM and HMMSDO show superior performance in determining21the optimal state sequence for certain applications. Zhou (2005) shows that the22D-HMM outperforms the corresponding generative hidden Markov model (G-23HMM) for part-of-speech tagging and phrase chunking; Li (2005) shows that242HMMSDO outperforms the standard HMM for prediction of protein secondary1structures when the training set is large enough.2In this note, we shall study the assumption of “mutual information indepen-3dence” that is used for deriving the D-HMM (Zhou, 2005) in the context of4determining an optimal state sequence, and then extend it to derive its gener-5ative counterpart, the G-HMM. In addition, state-dependent representations6for these two output-dependent HMMs will be presented.72 Generative HMM8Following the notation used by Zhou (2005), the definition of the optimal9hidden state sequence Sn1 based on the observed output sequence On1 is that10of the maximum a posteriori (MAP) estimator S∗ of Sn1 :11S∗ = argmaxSn1{logP (Sn1 |On1 )} . (1)The G-HMM rewrites the criterion (1) through applying Bayes’ theorem and12ignoring the item determined purely by On1 as13S∗ = argmaxSn1{logP (Sn1 ) + logP (On1 |Sn1 )} ,which is further factorised as14S∗ = argmaxSn1{logP (Sn1 ) + log(P (o1|Sn1 )n∏k=2P (ok|Ok−11 , Sn1 ))}.In order to make this formulation tractable, an assumption that On1 is condi-15tionally independent given Sn1 is in general introduced as, for all k ∈ {2, . . . , n},16P (ok|Ok−11 , Sn1 ) = P (ok|Sn1 ) , (2)3and thus based on such a conditional independence assumption, the MAP1estimator for the G-HMM is simplified to2S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1logP (oi|Sn1 )}. (3)The G-HMM is regarded as being generative because it directly models the3DGP P (oi|Sn1 ) of the observed oi from the hidden Sn1 .4In practice, as for the standard HMM, the assumption (2) is further simplified5to6P (ok|Ok−11 , Sn1 ) = P (ok|Sn1 ) = P (ok|sk) , (4)and thus the MAP estimator of the standard HMM is7S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1logP (oi|si)}. (5)3 Discriminative HMM from Mutual Information Independence8The D-HMM rewrites the criterion (1) through applying Bayes’ theorem, but9not ignoring the item determined purely by On1 , as10S∗ = argmaxSn1{logP (Sn1 ) + logP (Sn1 , On1 )P (Sn1 )P (On1 )}.To make this formulation tractable, an assumption that the mutual informa-11tion (MI (Sn1 , On1 ) = logP (Sn1 ,On1 )P (Sn1 )P (On1 )) between Sn1 and On1 is independent with12respect to each hidden si was introduced by Zhou (2005) as13MI(Sn1 , On1 ) =n∑i=1MI(si, On1 ) , (6)or, in more detail,14logP (Sn1 , On1 )P (Sn1 )P (On1 )=n∑i=1logP (si, On1 )P (si)P (On1 )=n∑i=1logP (si|On1 )P (si). (7)4Based on such a representation, the MAP estimator for the D-HMM is sim-1plified as (Zhou, 2005)2S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1logP (si|On1 )−n∑i=1logP (si)}. (8)The D-HMM is regarded as being discriminative because the criterion (8)3includes directly the discriminative process P (si|On1 ), representing an output-4dependence of a hidden state si on all the observed outputs On1 .5We shall make four observations about the D-HMM.6First, it is noted that the criterion (8) is simultaneously to maximise the max-7imum posterior marginal (MPM) estimator∑ni=1 logP (si|On1 ) of logP (Sn1 |On1 )8and to maximise the distance between the state transition model logP (Sn1 )9and its independent-based counterpart∑ni=1 logP (si).10Second, in order to satisfy the assumption (7) underlying the D-HMM, it is11required that12n∏k=2P (sk|Sk−11 , On1 )P (sk|Sk−11 )=n∏k=2P (sk|On1 )P (sk).Since this is valid for any value of sk, it follows that, for all k ∈ {2, . . . , n},13P (sk|Sk−11 , On1 )P (sk|Sk−11 )=P (sk|On1 )P (sk). (9)Third, the assumption (7) can be rewritten as14logP (Sn1 , On1 )P (Sn1 )P (On1 )=n∑i=1logP (si, On1 )P (si)P (On1 )=n∑i=1logP (On1 |si)P (On1 ). (10)Based on such a representation, the MAP estimator (8) for the D-HMM can15be rewritten, with the term∑ni=1 logP (On1 ) determined purely by On1 being16ignored, as17S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1logP (On1 |si)}. (11)5Therefore, the D-HMM can also be represented as being generative because1the criterion (11) includes a generative-like process P (On1 |si), representing a2state-dependence of all the observed outputs On1 on a hidden state si.3Fourth, it can be seen that, when the assumption (6) of mutual information4independence develops from independence between pairs (si, On1 ) into that be-5tween local pairs (si, oi) such that MI(Sn1 , On1 ) =∑ni=1MI(si, oi), the criteria6(11) and (8) degenerate into the criterion (5), indicating that the D-HMM7degenerates into the standard HMM.84 Generative HMM from Mutual Information Independence9Furthermore, similarly to the assumption (6) proposed by Zhou (2005), an10assumption that mutual information between Sn1 and On1 is independent with11respect to each observed oi can be introduced here as12MI(Sn1 , On1 ) =n∑i=1MI(Sn1 , oi) , (12)or, in more detail,13logP (Sn1 , On1 )P (Sn1 )P (On1 )=n∑i=1logP (Sn1 , oi)P (Sn1 )P (oi)=n∑i=1logP (oi|Sn1 )P (oi). (13)Based on such a representation, we can obtain another generative model and14its MAP estimator, with the term∑ni=1 logP (oi) determined purely by On115being ignored, as16S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1logP (oi|Sn1 )}. (14)This estimator is in fact the estimator (3) of the G-HMM, i.e., the G-HMM17can be derived under the assumption (12), a type of mutual information in-186dependence.1Similarly, we shall make three observations about this G-HMM, which is de-2rived from mutual information independence.3First, in order to satisfy the assumption (13) of the G-HMM, it is required4that, for all k ∈ {2, . . . , n},5P (ok|Ok−11 , Sn1 )P (ok|Ok−11 )=P (ok|Sn1 )P (ok). (15)Therefore, under the MAP criterion (1), the conditions (15) and (2) have the6same effect on determining the optimal hidden Sn1 .7Second, the assumption (13) can be rewritten as8logP (Sn1 , On1 )P (Sn1 )P (On1 )=n∑i=1logP (Sn1 , oi)P (Sn1 )P (oi)=n∑i=1logP (Sn1 |oi)P (Sn1 ). (16)Based on such a representation, the MAP estimator (14) for the G-HMM can9be rewritten, with the terms related to logP (Sn1 ) being combined, as10S∗ = argmaxSn1{(1− n) logP (Sn1 ) +n∑i=1logP (Sn1 |oi)}. (17)Therefore, in this sense, the G-HMM can also be represented as being dis-11criminative because the criterion (17) includes a discriminative-like process12P (Sn1 |oi), representing an output-dependence of all the hidden states Sn1 on13an observed output oi.14Third, it can be seen that, when the assumption (12) of mutual information15independence develops from independence between pairs (Sn1 , oi) into that be-16tween local pairs (si, oi) such that MI(Sn1 , On1 ) =∑ni=1MI(si, oi), the criteria17(17) and (14) degenerate into the criterion (5), indicating that the G-HMM18degenerates into the standard HMM.1975 Equivalence between G-HMM and D-HMM1Once we assume a fully independent mutual information between any state-2output combination (si, oj) as3MI(Sn1 , On1 ) =n∑i=1n∑j=1MI(si, oj) , (18)or, in more detail,4logP (Sn1 , On1 )P (Sn1 )P (On1 )=n∑i=1n∑j=1logP (si, oj)P (si)P (oj)=n∑i=1n∑j=1logP (oj|si)P (oj)=n∑i=1n∑j=1logP (si|oj)P (si),(19)this assumption results in two criteria, one generative and the other discrimi-5native, with the MAP estimators as6S∗ = argmaxSn1{logP (Sn1 ) +n∑i=1n∑j=1logP (oj|si)} , (20)7S∗ = argmaxSn1logP (Sn1 ) +n∑i=1n∑j=1logP (si|oj)−n∑i=1{n logP (si)} , (21)respectively. These two criteria are equivalent.8In the context of determining an optimal sequence of hidden states, apart9from the equivalence above, up to now, we find two occurrences of equivalence10between a discriminative representation of the MAP criterion and its genera-11tive counterpart: one is for the D-HMM between the criteria (8) and (11), the12other is for the G-HMM between the criteria (17) and (14).13We shall further illustrate such equivalence with two simple but related HMMs:14one is a generative-like state-dependent model, which assumes that the current15output ot depends not only on the current state st but also on the last state16st−1; the other is a discriminative-like output-dependent model, the so-called178HMMSDO (Li, 2005), which assumes that the current state st depends not1only on the last state st−1 but also on the last output ot−1.2The joint distribution of the first generative-like state-dependent model is3P (Sn1 , On1 ) = P (s1)P (o1|s1)n∏i=2P (si|si−1)P (oi|si, si−1) . (22)This distribution can be rewritten as4P (Sn1 , On1 ) = P (o1, s1)n∏i=2P (si, oi|si−1)= P (o1)P (s1|o1)n∏i=2P (oi|si−1)P (si|si−1, oi) ,(23)which leads to a discriminative-like output-dependent part P (si|si−1, oi) in the5distribution. In fact, the difference between the probabilistic directed acyclic6graphs (DAGs) corresponding to the joint distributions (22) and (23) is only7in that directions of edges from si to oi are reversed.8Similarly, the joint distribution of the discriminative-like output-dependent9HMMSDO, with P (si|si−1, oi−1) included, is (Li, 2005)10P (Sn1 , On1 ) = P (s1)P (o1|s1)n∏i=2P (si|si−1, oi−1)P (oi|si) . (24)This distribution can be rewritten as11P (Sn1 , On1 ) = P (s1)P (on|sn)n∏i=2P (si, oi−1|si−1)= P (s1)P (on|sn)n∏i=2P (si|si−1)P (oi−1|si, si−1) ,(25)which leads to a no longer discriminative-like output-dependence in the dis-12tribution. In fact, the difference between the DAGs corresponding to the joint13distributions (24) and (25) is only in that directions of edges from si to oi−114are reversed. In practice, whether or not P (oi−1|si, si−1) is reasonable needs15to be justified, because it means that the current output depends on the next169state.16 Summary2This note has suggested that the mutual information assumption (12) resulted3in the G-HMM, while another mutual information assumption (6) resulted in4the D-HMM. However, in practice, whether or not the assumptions are reason-5able and how the corresponding HMMs perform can be data-dependent; re-6search efforts to explore an adaptive switching between or combination of these7two models may be worthwhile. Meanwhile, this note has suggested that the8so-called output-dependent HMMs could be represented in a state-dependent9manner, and vice versa, essentially by application of Bayes’ theorem.10References11Li, Y., 2005. Hidden Markov models with states depending on observations.12Pattern Recognition Letters 26 (7), 977–984.13Ng, A. Y., Jordan, M. I., 2001. On discriminative vs. generative classifiers: a14comparison of logistic regression and na¨ıve bayes. In: NIPS. pp. 841–848.15Rubinstein, Y. D., Hastie, T., 1997. Discriminative vs. informative learning.16In: KDD. pp. 49–53.17Zhou, G. D., 2005. Direct modelling of output context dependence in discrim-18inative hidden Markov model. Pattern Recognition Letters 26 (5), 545–553.1910

Short note on two output-dependent hidden Markov models

Abstract

Similar works

Full text

Available Versions

Enlighten: Publications

Crossref

Enlighten