General theorems concerning the strong consistency of the MLE of exponential mixture parameters are proved. These theorems imply the strong consistency of the MLE of normal mixture parameters when the data is organized into fields each of which is a random sample from one of the component normal distributions

Peters, C.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19800002591 2020-03-21T20:19:10+00:00Z
-2- . 
" _ 
. ' OH THE CONSISTE?<CY OF 
THE MAXIHUM LIKE1:HOOD ESTIMATE' 
OF NORHAL MIXTURE PARAMETERS 
FOs A SAMFLE WITH FZELD STRUCTURE 
BY 
CHBRLES PETERS 
( i~ s&&~s- (  60352) 01 T!iE COESISTEBCI OF 281 
MmEBH &IKELYBOOD. EST33AT.B OP ~ O R ~ A L  
w q q ~ ~  paaaa~esas ~ e i  A SIIPLE U I F B  PIBLD 
slrat#@oa~ (Houston Univ- 1- 18 P 
a c - - r a 2 ~ r  a01 CSCL 12A 63/60 45993 
t -  
PFi'EF'AI$Efi FOR 
EARTH OBSERVATION DIVISION9 ,.jSC 
- .  - 
HOUSTON, TEXAS 77004 
Parameters for  a Sample with Field Structure. 
Charles Peters 
Department o f  Mathematics 
Uni versi t y  o f  Houston 
Houston, Texas 
Report #74 
August, 1979 
, . 

i i - I. I n W U k t i o n  / 
. 
- 
.-' 
.- 
' r; 
e 
~ * 
* I '" 
c .  
In C 5 I 8 s ta t i s t i ca l  moddl for L&!DSAT apr lcul tura l~data based on nonu? 
mixtures was lntroduccd whichadwdts a $peciflr'kind o f  dependeke among the *. j 
i \* r -  -. 
obsekations. nkrly thejr associathn in to  f ie lds  $,each representhg a single 
, - * I  
rgr icu l  b r a 1  class. bcessary conditioni were derived fo r  ; maxinn 1 i kel i - 
b 
> 
hwd estlmte of the parameters of the mdel  and a numerical procedure for  
solution of the l ikelihood equations was su~gested. The question of the 
-- 1 
consistency o f  the waxinnnn 1ikel ihoob~est i lp te i s  complicated by the fact I 
I 
that it i s  no longer possible to  reduce the sample t o  a set o f  independent I 1 I 
ident ical ly distributed variables. The purpose o f  th i s  note i s  t o  establish 
a general theorem on the existence o f  a consistent ~xinwan 1 i kelihood estimate 
when the observations are not ident ical ly distributed and t o  show i t s  applica- 
b i l f  ty t o  the s ta t is t ica l  model described i n  deta i l  below. 
We d s s w  that each pixel i s  ident i f ied by a pai r  ( j ,k )  of positive 
integers, where the f i r s t  index j, 1 s 2 j  s p, ident i f ies the f i e l d  containing , 
the pixel and the second index k, 1 s k s K j .  distinguishes i t  from other i I 
pixels i n  the same f ield. We suppose that the f i e l d  structure i s  predetermined, I 
1 
perhaps as part o f  a spatial clustering algorithm such as AMOEBA. Le t  
xJk c R~ be the random vector o f  spectral measurements from pixel (j,k) m d  
l e t  0 c l ; , m  be an unobserved random variable indicating i t s  class 
index. We assum that the class indices Bjl, Bj2, ' * *  O ' N ~  from the j t h  
f i e l d  a n  a l l  the same and denote thei r  connon value by Q We further j 
assume that, condi tioned on Q = 8 ,  the neasuremnts Xjl, j x J ~ j  are 
indeprndently distr ibuted as Nn(*, u;, $1, the n-variate normal wi th  
unknown man p i  and unknown covariance E: . Let xj = 
our f inal  a s s ~ p t f o n s  a m  that  (xi, el), , (X , O ) are independent 
P P 
Under these assumptions, the j o i n t  density o f  a11 the observations i s  
4 
, xjNj) 4 ~~~j . This j o i n t  density i s  parametrized by 
m ( a ,  pL, 1 . * *  , ml where oL > 0 ;  C a t z  1; u a s  R ~ ;  and Ze i s  
a=i 
a 1 nxn posi t ive defini;e symetr ic  mi3trix. For c~avenience, we l e t  
@ = {ag, ug, zL)1t = 1, ... , m) denote an arb i t rary  member o f  the parameter 
space and $O the true value of the parameter. Thus the l ikel ihood function 
CorFesponding t o  the sample xi , * 

2. The General Theorem 
-
... Let 0 be an open subset o f  i2 and l e t  J,' uo 0 . Suppose X1.x2. , 
i s  a  sequence of independent 9 random vectors wi th  xr having Nr-vari a te  density 
function qr(*  I@) wi th  respect t o  some f ixed 0 - f i n i t e  measure A, on ~ ~ r .  
Suppose the dens i t i i es  q  1 )  are defined f o r  each J, c . Given a 
pos i t i ve  integer p , def ine a  maximum l i ke l ihood  estimate o f  fl t o  be an 
P 
element $ c B which l o c a l l y  maximizes L (I)) = Z log qr(xr(@) . The equation 
P r = l  
D,,,Lp($) = 0 w i l l  be ca l led the l i ke l ihood  equation, where the symbol D 
J, 
denotes the Frechet der ivat ive  w i th  respect t o  9 . 
A number o f  theorems deal i ng  wi th  the consistency of maximum 1 i kel ihood 
estimates, under the addi t ional  assumption tha t  the xr's are i den t i ca l l y  
d ist r ibuted,  have been presented i n  the 1  i te ra tu re  (see f o r  instance Chanda 1 ? 3 . > 
Cramer I 1 4  1, and Wald [ H 1. ) Extending any o f  these resu l t s  t o  the case of , 
nonident ical ly  d is t r ibuted observations i s  p r imar j l y  a  matter of f i nd ing  a  
convenient set o f  conditions which insures tha t  a  law of large numbers can be 
invoked a t  several points i n  the proofs. The fo l lowing theorem i s  such an 
I outgrowth of the p r w f  o f  strong consistency contained i n  1.6 I. 
Theorem 1: Suppose there i s  a neighborhoad 0 o f  and a ar nul l  sets 
gr such that  for  a l l  r . n: w N r 2 ; 1- - - - -  - 
. 'i ana --, 
a$i a$, a$ ah. aa . ~ I I , .  ex is t  and 
satisfy: 
-* .- * 
-arm--- r r r ~ p - ~ p -  . ... -='
- - 
-7- -- ' - -  --- .--=-* - 
. - --. 
- - -  
- = .lr* I s J s K  = 1, ,Z, r r (the 
aqr(xI*) 
~ o ~ i t i v e  integers) -: . - - A  a310g q,o(l$) 
1 -7J -'-k 
I r (x) r a*, I 
( t i )  (XI 
( i i i )  f;;t,..ix) 
2 ('v) 'lfj kr(xr)2~ = / f i j k r  (XI ( r ( * / ~ ) d ~ r ( i )  r M 
- -  HN 
a l l  r where M i s  a constant. C t ~ n n ~ c a  -la- A L - A  JI 
(4 El[ :;:xr~#O) 
and 
O r  1 1 ' *  ' 2 a d  r r . Ff rial l y  suppose tha t  3 .. 0 such tha t  
('if) J,($O) = '['*log qr(xr 1 $O)v*log qr(xr r I v i v  
I 
for a1 1 r r . where the ordering i s  the usual one on vxv symmetric matrices. 
Then, i t i s  almost surely t rue that, given a su f f i c ien t l y  small neighborhood 
of (O; for a r e  p there i s  a unique so lu t ion of the l i ke l ihood  equation 
D,,,Lp(r) = 0 i n  tha t  neighborhood. Furthemore, tha t  so lu t ion i s  a maximum 
I 
Hkel ihood estimate. 
Remark: I n  the proof we make repeated use of the f o l  lowing simple version 
of the strong law of large nymbers (see Chung [i I): Let zl, 4, - * *  be 
uncorrelated random variables and suppose the sequence o f  variances t tar(Z ) 
1 i-1 
1 i s  bounded. Then - C (Z1 - E(Zi)) + 0 a.s. as n -. = . 
i i = l  
Pmof of the theo-en: Let p($) = C P D,,,log qr'(Xr19) . By assumptfon (i) 
P r = l  
E ( ~ ~ ( $ J ~ ) )  = 0 and by assumption (v) and the strong law, sp( rO)  4 0 a.s. as 
p + =. Now consider the vxv matr ix D 2 ($') whose 1 j t h  element i s  
9 P 
I I 
1 
t 
BY assmption (ii) the expected value o f  the f i r s t  term on the r i g h t  i s  zero. < .  , 
- -I_- -r 
---- ---- > - - A= - 
u 
1 P Hence, by assuptfons (v) and (vi'! D s (@) + - C J,($') - 0 a.s. as 
'4 P p r=l 
d 
p + rn. It follows that wf t h  pmbab i l i t y  1. f o r  each n i n  0 < g< - 
2 
t h e n  i s  a p, c so that f o r  p 2 p, 
. LJithout loss of generality we can assume .Q i s  conve*~. 
Thus, fo r  b c n , 
, .t h  probabi 1 i t y  1, fo r  large p 
A Y 
- 
P 
C f... ( X  \ < 1 + Y FrC ( X  ).I e 1 + i 5  . 
- . 'J&r'-.r ' - . p r.1 " " ' i j k r  r p r=l 
by assumption ( d v ) .  
I t  follows that f o r  any part icular norms on R' and on the vxv svnnetrlc 
--- - - - A - &-, 
I l o p  - at,(& i l s 111 111 - I 
Thus, there I s  a convex neighborhood no o f  6' such that  
Oqtp(9) s -01 
for a l l  $ E 0' , p 2 pl . I t  now follows as i n  C o I that fo r  p 2 p, 
a t  @ of small radius 6 contains the sphere Q b (9)) a t  c,($') of 
n6 P k' 
radius r 6 . Since 0 i s  eventually i n  Q (x (I$)), there i s  a unique 
n6 Y 
solution of ~ ~ ( 9 )  = 0 i n  f i6(+O).  Since D,fp($) i s  negative def ini te,  
th is  solution i s  a ma~imum l ikel ihood estimate. This concludes the proof. 
Theorem 1 shows that by res t r i c t ing  at tent ion t o  a f ixed neighborhood 
of a0 i t  i s  possible t o  speak unambiguously o f  the unique consistent 
solution of the l ikel ihood equations w, equivalently, of the unique 
consistent MLE o f  yo This tenninologywi l l  beadopted i n  thenext  theorem. 
3. Application t o  Exponential Uixtures 
I n  t h i s  secLion we apply Theorem 1 t o  a class o f  mixturae models which 
contains the normal mixture model o f  section 1. Referryng t o  the notation of 
that section. we assume that conditioned on Oj = L, the random n-vectors 
f!xl~,) = i('rL) exp <rLJF(x)> 
with respect to  a dominating o - f i n i t e  measure A where the parameter rL 
i s  an arb1 t rary  member o f  an open subset U o f  a f i n i t e  dimensional vector 
splce V with Inner pmduct <*I*>. b ass- also that C 4% o y  t o  on 
and that the support o f  the measure induced on U by F and A contalns 
m open set. These conditions imply that the parameter rl i s  Ident i f iable 
t 1 I ,  and any paramtr l tat ion o f  the form (1) satisfying them w i l l  be called 
a canonlca!, representation o f  the given family of distributions a 
The j o i n t  density, 
exponential type; i . e. , 
given O j = ~ . o f  xj=(xjl. .... ) I s a l s o o f  
, X j ~ j  
for XI = ( x ~ ~ , " *  *.x,jHJ) 
where 
and the representation (2) i s  also canonical. 
Sprne useful facts about exponential fwil les are collected i n  the 
following l a ~ w .  For proofs see Barndorff-Nielsen ! 1 I. 
L m  1: Let (1) be a canonlcal representation o f  an exponential family. 
For each r c U l e t  K(T) = - I n  C(r) = I n  j exp <TJF(x)> dA(x). Then 
nrn 
( 4 )  for each r E U. F(x) has moments o f  a l l  orders with respect 
f (~1T);  
(11) r(r) has derivatives of a l l  orders with respect to  T, which 
my be obtained by di f ferent iat ing under the integral sign. In- 
k d d  DT r(r) can be represented as r synnetric k-linear form 
on V which i s  a polynmlal i n  the f i r s t  k moments of F. I n  
part1 cular ,.

(7) DTtOqj(xjI$) = <Gj - u- ( t  )I@> . t = l,.;.m - l J a 
(8) DTmDa?j(xjI$) =-pj(xjI?,,,) aj - ~ j (~ , , , ) l~>  , t =  l , * - = , m -  1 
= 1,. ..,IN .
Instead of ver i fy ing conditions ( i )  and ( i i )  o f  Theorem 1, it i s  easier 
t o  recal l  that they were needed only i n  order t o  conclude that the integrals 
of the f i r s t  a d  second order derivatives o f  q.(x-10) are z e k  a t  $ = $O. J J 
This i s  obvious form (5) - ( 9 ) .  Similarly, using Lema 1 and the boundedness 
of the ver i f icat ion of conditions ( i i i )  - ( v i )  presents no problem 
more serious than tedium. It remains t o  ve r i f y  condition ( v i i ) .  We may 
wri te J,($) i n  matrix form as 
!L,k=l,. .. ,In-1 
---- 
- LA -. -*I>- 
P 
* - - 0 and A1 ... - A = 0. ~t i s  ~ s i l y  saen that i f  m 
Jr(#) f a i l s  t o b c  m l t i v e  def in i te  then there i s  a nontr iv ia l  l inear  
for each r. Condition ( v i i )  w i l l  be established once i t  i s  shawn that 
the smallest eigenvalue of Jr($) i s  bounded away from zero as N, + a. 
Let u(A) denote the smallest eigenvalue of a posi t ive def in i te  
operator A. Clearly, 
I 
. . 
If Xr I s  a sample from f ( x l r k ) .  then the expression i n  square brackets 
4 
$ 1 
i 
I 
i 
i 
1 
- - -  - d i 
which i s  > 0 by the s t r i c t  convexity o f  K. Hence, 
Given that  Xr i s  from f (x l rk ) ,  N ; ~  (6, - ur(rk)) converges i n  distr ibut ion 
Let A be any element of V and considcr 
7 
After expanding and taking expectation with respect to  rt ,  i t  w i l l  be 
~f tm arc Nr + (r) = 0 (6. T ~ S  
i s  bounded as Nr * -. I t  f o l l a rs  froa a standard theorem an m a c e  
of  moments C 3 .p. 951 that  
Thus E@(B,) + 0. Simi!ar reasoning shows tha t  
Ey(Cr) + COY (F)) 
=k 
as Nr + Therefore o(J,!$)) i s  bounded away fm 0 and t h i s  concludes 
the proof. 
4. Concluding Remarks. 
Clearly the assumption i n  Theoren 2 that N r  i s  bounded can be 
weakened. Infact,Theor- 1 could be modified i n  such a way as t o  show that 
the MLE of exponential mixture parameters i s  strongly consistent when 
Redner 1 7 I has shown that when each Nr = 1, a certain numerical 
procedure for obtaining the MLE o f  exponential mixture parameters i s  con- 
vergent. The generalization t o  bounded { should not be d i f f i cu l t .  and 
w i l l  be addressed i n  a future report. 
References 
i 1 
1. 0. Barndorff - Wielsen, Inforinatfon and Exponential Families i n  Sta t is t i ca l  
---
I 
Theory, Yiley and Sons, New t w k  (1978). I i 
2. K. C. Chanda, A note on the consistencv and maxima of  the roots o f  the - - - - -
l ikel ihood equations, Biometrika, 41 (1954). pp. 56-61. 
3. I(. L. Chung, A Course i n  Probabil i ty Theorfi Second Edition. Academic 
-
Press, New York (1974). 
i 
i 
4. H. Cramer, Mathematical Methods o f  Stat ist ics,  Princeton University 
-
Press, Princeton, 1946. 
5. C. Peters, A modification of the l ike l ihood equations f o r  normal classes 
wi th  f i e l d  structure (preliminary report). Report 173, Department 
o f  ib thwat ics,  ~ n ' i v e r s i t ~  o f Houston, June 1979. 
6. and H. F. Walker, An i te ra t ive  procedure f o r  obtaining maximum 
lfkel ihood estimates o f  the ~arameters f o r  a mixture o f  normal dis- 
tr ibutions. S I N  J. Appl. Math. B., 35, 2, Sept. 1978. 
7. R. A. Redner, An i t e ra t i ve  procedure f o r  obtaining maximum l ike1 ihood 
estimates i n  a mixture model ( t o  appear). 
8. A. Wald, Note on the consistency o f  the maximum l ikel ihood estimate, 
Ann. Math. Stat. 20 (1949), p. 595. 
- ---


On the consistency of the maximum likelyhood estimate of normal mixture parameters for a sample with field structure

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server