New results and insights concerning a previously published iterative procedure for obtaining maximum-likelihood estimates of the parameters for a mixture of normal distributions were discussed. It was shown that the procedure converges locally to the consistent maximum likelihood estimate as long as a specified parameter is bounded between two limits. Bound values were given to yield optimal local convergence

Peters, B. C., Jr.

Walker, H. F.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19760013826 2020-03-22T15:18:48+00:00Z
^p OF,y
i C^co DEPARTMENT OF MATHEMATICS 	 NASA CR•
z	 o	 i 7 //UNIVERSITY OF HOUSTON	 HOUSTON, TEXAS	 ^	 :
(NASA •CR-147521) AN ITERATIVE PROCEDURE FOR	 $76-20914
OBTAINING MAXIMUM-LIKELIHOOD ESTIMATES OF
THE PARAMETERS FOR A MIXTURE OF NORMAL 	
Uncles
11 P
STR HC O$3.50 t ADDENDUM (Houston Uni
CSCL 12A G3/65 21495
i
t07pr
Ins
p	 ^	 itECE1V E'^	 ^'
INPUS B(ZAMCH
EARTH OF SFRVAT If;'
	 Iv P5I 
Ctl"T 0r.T
	
-11777
3801 CULLEN BLVD.
HOUSTON, TEXAS 77004
Addendum to "An I.tma.tive Procedure 601L
Obtaining Maximum-likelihood Estimates o6
the Parameters bon a Mixture o6 Notma.E D.i.sthibuti.onb"
by
H. ChakteA Peters, Jr.
NASA/NWkiia2 Research Councc.8 Research Associate.
Ea&tA Observations D.w.caa.on, Johnson Space Center
and
HomeA F. WatkeA
Department o6 Mathematlrs, UnivWW4 o6 Kouston
Houston, Texas
Report 047
SeptembeA, 1975
NAS 9-12777
i	 i	 I
Addendum to "An Iterative Procedure. for
r
Obtaining Maximum-Likelihood Estimatea of
f	 ,A
rr
	
the Parameters for a Mixture of Normal Distributions'
r4,
k'
by
B. Charles Peters, Jr.
NASA/National Research Council Researc& Associate
Earth. Observations Division, Johnson Space Center
and
Homer P. Walker
Department of Mathematics, University of Houston
Houston, Texas
1. Introduction.
In this report, me. discuss new results. and insights concerning an iterative
procedure introduced in .jlj for obtaining maximum likelihood e4timates of the
parameters for a mixture of normal distributions. For any questions concerning
notation, definitions, etc., the reader is referred to that report.
The. iterative. procedure in question is the follouSmg; Beginning tritii some
iicll
starting value	 ^1	 in the space Q(fB W ®,8 introduced in t 1], define
( 
E^A
2successive iterates inductively by the relationship
OL
lY	 ^
+l)
given in (17. It is shown in [1] that, with. probability approaching 1 as
the sample size N approaches infinity, this procedure converges locally to
the consistent maximum-likelihood estimate whenever a is sufficiently small.
+l)(n+2 guarantees the local convergence of this pro-(In particular, e < m
cedure in probability.)
rn this report, we prove that, in probability, the procedure G*I converges
locally to the consistent maximum-likelihood estimate whenever 0 < E < 2. We
also show that the E whic& yields optimal local convergence rates lies between
1 and 2. In fact, the optimal E is near 1 0 if the component populations
are widely separated,and near 2 if the component populations have nearly
identical means and covariance matrices.
1. Local Convergence.
	As in jl), we say that le is locally contractive (in a norm	 on
'	 a
ad) MZ ® ,g } near — E• or@ bX © ,$ if there is a number A t 0 < A ^ 1
JU
such-that
3a	 a'	 a
^ECa''.u,El- U sa u' - u
a'	 a
whenever u' lies sufficiently near	 y	 Our result is the following.
E'	 E
Theorem. With. probability approaching 1 as' N approaches infinity, 3e iss
a locally contractive operator (in a norm to be defined on 0(0 M (9j ) near
the consistent maximum-likelihood estimate whenever a '4 e < 2.
Corollary. With probability approaching 1 as N approaches infinity, the
iterative procedure (*) converges locally to the consistent maximum-likelihood
estimate whenever 0 < e < 2..
Proof. As observed in [1), the theorem will be proved if it can be shown that,
for 0 < e < 2, E(0 ^e(&°,u°,E °))qq has operator norm less than 1 with respect
to some vector norm on a®	 ® p . (Throughout this note, the superscript
" 0 " indicates that the superscripted parameters are the true parameters of the
mixture density.) For	 let < , >1 and < , >"i 	the inner pro-
ducts on in and the apace of real, symmetric nxn matrices introduced in (1),
i.e., let
-1•
<v ,w>i - v (aiEi )w	 for v,w a Rn.
<A,B>i - tr{A (-2Ei	 T})B	 forreal, symmetric nxn A and B.
I
and
PI
pl
P
P
pm
P
Pm
P
i
4
4
These inner products, together with scalar multiplication on R1 , induce an
`` a
	 inner product < , > on	 Now E(V f ,(a°,µ°,£°))	 I	 e QR,
where
(diag ai)	 0	 0
Q m	 0	 I	 0
0	 0	 (diag Ei)
R
n
Pp(x-Ni)
pp(x-um)
Pp 
LEl 1 (x-ul) (x-1 11-I ]
pm[E, -1 (x IQ (x-P )T-I]p m	 m	 m
PI(x-u1)
Lp(x-um)
p1 IEi 1(x-N0V(x-Vl)T-I1
P ( Em 1 (x, Um) (x-V°m)T-I]
s >P dx
One sees that the theorem will be proved if it can be shown that with respect
to some vector norm on L'7(SM ®,8 , the operator norm of QR is no greater
than 1. Since QR is positive definite and symmetric with respect to the
inner product < Q-1  >, it follows that the theorem will be proved if it can
be shown that <V,Q 1 (QR]V> a
 <V,RV> s <V,Q 1V> for V e OCI@ M ®,g .
is
^u
s
i
For
yl
ym
vl
V-
°m
Al
Am
e Q®PT Gj ,
one has
<V,RV> - ( T pi + E v (O'l.1i )Pi(x-u ) +i li p ili t. i	 p	 1
Rn
iLltr(Ai(21 Ei 1)pi[E°i 1(x-Pi)(x-ui)T - IlT} ) Zp dx
l[^ily liv1Ei 1(x-}li)+ tr(Aj'i i l ) [ Ei 1 (x-ui) (x-ui) `-I]T} laPp(i i) 2p dx
Rn
Y
S I(iEl [ai lyi4v1E °il (x- Pi )
+ tr{Ai(?Li l)[Ei 1 (x-Ui)(x-1Pi) T-I )) ] 2 appi)P dx
Rn
by Schwarz ' s inequality. If the squared expressions in the last sum above are
written out in full, one sees that the integrals of the cross terms in these
expressions vanish. Consequently,
<V,RV> s j ( 1E l[aiZYi+(vi
£i 1(x-ui))Z+(tr{Ai(ZEiV-1 9-11(x-Ui)(x-ui)T-I]T})]Zaipi)d,
Rn
^ ...._iw
r6
Now
(1)1 ai lYipidx ai lYi
Rn	 ,
(2) (viEi 1 (x- 1a tae dx f viEi 1(x-ui)( -)I TE,1 lviaipidxf
Rn
	
in
,
<vi'vi>i(3)f (tr{Ai(2Ei 1)[Ei 1(x-ui)(x-ui)T I)T )) 2aipidx - <Ai' Ei 1Ai >i
Rn
(A proof of (3) follows below.) From (1), (2), and (3), one concludes that
'	 1	 to
<V,RV> s 1E1ai 
1yi + <v
i ,v >i + <Ai ,Ei Ai>i w <V,Q 1v>.
This completes the proof of the theorem.
Proof of (3): Setting y - E°i1/2 (x-Ui) and
I	 J (tr{Ai(21"i 1) [E0-1(X-1ti) (K-POP 	C,." idx,
Rn
one obtains
I - 4i J (tr{Ai [Ei
e-1/2yyTEi 1/2 _ Ei1)T))2Po Y,
Rn
where po k N(0,I).
 
Denoting Ei 1/2A1E1	 b1/2 - B - ( jk
c A.,
7is one then derives
a
0 
r
4 J ( tr {B IYYT-I M 2 P y
`^'	 Rn
m 
a4 J 
[(tr {ByyT}) 2 - 2tr{B}tr{ByyT} + (tr{B
}) 2 } po y
p	 Rn
° 4j.K.P.g5JOpq f ykyjygyppo y - 2 tr{B)JEOjk s YkyjPody + ( tr{B})2}
in	Rn
°
4i{k pkSkkSPP + k ^ks3 k^jk + k Jfk s^kski 
+ 3ks - 2(tr{B} ) 2 + (tr{B})2}
a1 tr{B2} 4 ai tr{En -1/2A E° -lA E°-1/2} tr {A (a1E 1) (E°-lAi) T}2	 2	 1	 i i i i	 1 2 1	 i' 
n
<Ai' Ei '-Ai>i
3. The optimal e.
From the proof of the theorem, on.. sees that, asymptotically as N approaches
infinity, the value of a which yields optimal local convergence rates is that
which minimizes the spectral radius of MT. (a`,,E°)). (Indeed,
°,u°,E°))	 I - e QR is symmetric with respect to the inner productE(V fC(a
< Q-1  >; hence, its operator norm with respect to this inner product is equal
to its spectral radius.) Letting p and T denote, respectively, the largest
and smallest eigenvalues of QR, one verifies that the spectral radius of
o 
—0- 	 2E(0 j E (a ,µ ,E )) is minimized when 1 - e T ° Ep - 1, i.e., when e ° P+T*
Now p = 1 always, for it follows from the proof of the theorem that p is
iF
never greatar than 1, and
8
°
al
a°m 
e a^^e ^,^0
0
is always an eigenvector of QR with eigenvalue 1. Thus optimal convergence
rates are obtained when e - 14C2 . where T lies between 0 and 1. In
particular, the best choice of a lies between 1 and 2.
Suppose that the component populations in the mixture are "widely separated"
in the sense that each pair ( ui ,V) differs greatly from every other such
pair. Thep.
^
aipi(x)1 (a'ipi(x)1
p(x) I	 p(x) 1	 6i3 for x e Rn and i,j = 1,...,m,
and one verifies that QR z I. Consequently, optimal convergence rates are
obtained for an a near 1 and, for the optimal e,
E(0 Tea °°,E°)) . I - e QR z 0. Thus for mixtures whose component populations
are "widely separated", optimal convergence rates are obtained for an a near
1, and rapid first-order convergence can be expected for this e.
Now suppose that the component populations in the mixture are such that
each pair (pj,Ei) differs little from every other such pair. Then
1	 I ^	 !	 ^
9
p(x) z pi (x) and Pi(x)
p(x) z 
1 for x e Rn and i 1,...,m, and one verifies
that the smallest eigenvalue of QR is near zero. It follows that optimal
u
convergence rates are obtained for an a near 2. In this case, the spectral
radius of E(41E (3°9u°.E°)) is near 1, even for the optimal value of e;
hence, slow first-order convergence is to be expected.
We conclude by observing that the major practical implication of this note
is that the iterative procedure under consideration converges whenever the
step-size a lies in an interval which is completely independent of the particular
mixture problem at hand. It is readily ascertained that this cannot be ::aid for
the regular steepest descent procedure
a(q+l) OL	 + e[ 1 	 pi(xk.) _ 1	 G p-j-- 1N k-1 p(xk)	 mN j-1 k-1 p(N)
Uiq+1) 
uiq' + e [N k l _F
	
E(4)-1(xk-uiq)))
E (q+l) E (q) + E [1 CCN aiq)pi(xk) [-E'q)-i + E sq)-1 ( _u(4) )( u(q))TE(0 -111i	 1	 2N k l p(xk)	 1	 1.	 xk i	 xk i	 i
Thus the procedure considered here offers considerable practical advantages over
the steepest descent procedure, even though it is itself a generalized steepest
descent (deflected gradient) procedure.
REFERENCE
1. B.C. Peters and H.F. Walker, "An iterative procedure for obtaining
maximum-likelihood estimates of the parameters for a mixture of normal
distributions, "Report 1143, NASA Contract WAS-9-12777, University of
Houston, Department of Mathematics.


An iterative procedure for obtaining maximum-likelihood estimates of the parameters for a mixture of normal distributions, Addendum

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server