Maximum-likelihood estimates are discussed which are based on an unlabeled sample of observations, of unknown parameters in a mixture of normal distributions. Several successive approximation procedures for obtaining such maximum-likelihood estimates are described. These procedures, which are theoretically justified by the local contractibility of certain maps, are designed to take advantage of good initial estimates of the unknown parameters. They can be applied to the signature extension problem, in which good initial estimates of the unknown parameters are obtained from segments which are geographically near the segments from which the unlabeled samples are taken. Additional problems to which these methods are applicable include: estimation of proportions and adaptive classification (estimation of mean signatures and covariances)

Walker, H. F.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19750018668 2020-03-22T21:51:21+00:00Z
fDEPARTMENT OF MATFIEMAI WS 	 NASA CR.
^'	 UNIVERSITY 01= HOUSTON	 HOUSTON, TEXAS0
(NASA -CR-141882)	 MAXIMUM LIKELIHOOD
SIGNATURE ESTIHATIOu (Houston Univ.) 	 10 p
HC $3.25	 CSCL 12A
N75-2b740
Unc , as
G3/64 27270
MA X I MUM L I K E L I F1rTrlF S I G NI A TURF.FSTIMAT Ir,N
RY lW h 'F-R F. 4ALKFR
APR IL 1975
k	 PP 'ARED FnR
EARTH 08SFPVATI0N 01VISION 9
 JSL
! INUEN
CHNVRACT NAS-14-17177
3801 CULLEN BLVD.
HOUSTON, TEXAS 77004
timation
j
by
Homer F. Walker
Mathematics Department
University of Houston
Houston, Texas
Report 042
NASA Contract NAS-9-12777
tism" -..
Maximum Likelihood Signature Estimation
Abstract
In this outl.ne, we discuss maximum-likelihood estimates, based on an
,s
	
unlabeied sammple of observations, of unknown parameters in a mixture of normal
{	
distributions. Several "successive approximation" procedures for obtaining
such maximum-likelihood estimates are described. These procedures, which are
theoretically justified by the local contractibility of certain maps, are designed
k,
to take advantage of good initial estimates of the unknown parameters. It is
anticipated that they can be profitably applied to the signature extension
problem, in which good initial estimate of the unknown parameters are obtained
from segments which are geographically near the segments from which the unlabeled
samples are taken. Additional problems to which these methods are applicable
include: estimation of proportions and adaptive classification (estimation of
mean signatures and covariances)
1. Introduction
Let{ }	 c	 n be an unlabeled sample of observations from a
xlc k-1 , ... , N —
•	 mixture of m populations,•where each population is normally distributed, and
let some (possible empty) subset of the signature parameters fai'uiPEi}i^l,...,m
be known. (Here, a 	 is the arp iori probability.that a sample observation comes
from the ith population; 
P  
and E 	 are, respectively, the mean vector and
covariance matrix for observation from the i th population in /)e n .) A maximum-
likelihood estimate of the remaining parameters is a choice of those parameters
which maximizes the log-likelihood function
4
I
i
r
-7
TPP_
2
N
L ' kEllog P(xk)•
In this expression, p denotes the mixture aensity function, i.e., for
x E p n,
n-
m
P(x) - 140'ipi(x).
where
	
1	 1 -	 T -1(x-u )
	
Pi(x)^(2n)n/2 
I 
	 I112 e
i
Techniques ?or obtaining maximum-likelihood estimates of this type have
been studied by many authors and are considered by a number of them to be
superior in general to other methods of estimating the parameters of a mixture
of normal distributions. ( See, for example, [ 2] and [6].) Clearly, L is
a differentiable function of the signature parameters to be estimated, and there
are many approaches to obtaining a maximum of such a function. We diacuss
several such approaches each involving "successive approximation" iterative
procedures suggested by the particular form of L.
The iterative procedures to be described in the following are based upon
manipulating the gradient of L, with respect to the unknown parameters, and
incorporating the resulting Pxpressions in fixed-point equations for the
unknown parameters. Some of these iterative schemes have been studied by other
authors; others are new. In recent preliminary results of Peter9 and Walker,
ft.
the iterates has been established for initial estimates within a
call neighborhood of the maximum likelihood estimate. This is
accomplished by establishing that the appropriate maps are locally contractive
at their fixed points. Consequently, these procedures are well-suited for
application to the signature extension problem, whenever to 	 initial
signature estimates (i.e. 9 those satisfying the contractability condition) can
be obtained from segments which are geographically near the segments from which
the unlabeled samples are taken. We discuss the application of these schemes
to the signature extension problem and other problems.
MW
lm^	 I .	
I	
- - ___ L
4
2. The likelihood equations.
The procedures require (for a given unlabeled sample) the calculation of
the partial derivatives of the log-likelihood function with respect to the signature
paramet-ra. Equating the resulting partial derivatives to zero (the necessary
extramum condition), a straight-forward calculation yields the likelihood equations-
C1 i N Pi(xk)
N k11 P(xk)
(l.b)	 Ui	 { N k^lxlc PP() }	 { N kE l pp() }k	 xk'
I' Pi(xk) / 1 N pi(xk)
(1. c)	 Ei	 { N kw l (xk_P i )(xk-u i )	 P(xk ) } ^ { N kEl p(xk)}
1 M 1....'m
In the following, we will assume that a solution of any subset of the
likelihood equations is a maximum likelihood estimate of the corresponding
signature parameters. For example, if a set of mean vectors and covariance
matrices is given, then a maximurt-likelihood estimate of the a priori
probabilities is a solution {ai}.,m of the equations (l.a).
1.f
3. The natural iterative procedure.
The likelihood equations, as given. suggest the following iterative
^y
^r^Fd	 	 '	 •^	 •	 ^	 I	 /	 i ^/
0
S
procedure: Beginning with some initial estimate, obtain successive approximations
of the unknown parameters by inserting the preceding approximations in the
expressions on the right-hand sides of the appropriate equations (l.a), (1A),
(l.c). Such a scheme for obtaining maximum-likelihood estimates has been
investigated by several authors.
Empirical studiee in (2), (3], and (4] suggest that this scheme is
convergent, even if all the parameters are unknown, and that convergence appears
to be particularly fast when the populations are "widely separated".
Unfortunately, the likelihood equations may have several solutiLau,
and the iterates may converge to a solution which is not a maximum likelihood
estimate ifcare is not taken in the choice of an initial estimate. No
theoretical evidence of convergence is given in (2], (3], or (4].
Coberly and Peters (1] have proved that, if the unknown parameters are
the a prior i probabilities, then the scheme is locally convergent, i.e.,
convergent for an initial estimate which is sufficiently near a maximum-likelihood
estimate. They also report on numerical studies in which the computational
feasibility of this procedure is demonstrated. 	 Recent results of Walker
state that the scheme is locally convergent when the unknown parameters are
the means, whenever there are only twc populations (1. e., m - 2)
or whenever the populations are "widely separated". The local
convergence results are all achieved by showing that the expressions on the
right-hand sides of the appropriate likelihood equations are locally con-
tractive functions of the unknown parameters (in some vector norm) near a
maximum-likelihood estimate.
i
64. A modified iterative procedure.
We will now describe a :codification of the .iterative procedure just given
for which more extensive local convergence results have been obtained. Becaus,
these results are not yet sufficiently complete to allow the covariance matrices
to be unknown parameters, ve v Ul give the fixed-point equations for the a
priori probabilities and the mien vectors only. These eq uations are
ai N Pi(xk)(2. a)(1-E)a i + c
	 kEl p( )
xic
i ! 1, ..., m
1 N	 Pi(xk)1 N Pi(xk)
(2.b)
	 ui ' ( 1 -E)P i + c N kE 1 k P(xk )	 N k1 1 p(xk)
where the scalar parameter c is to be determined , Clearly, these equations
are satisfied if and only if the equations (l.a) and (l.b) are satisfied.
If any or all the a priori probabilities and the mean vectors are
unknown, the the equations (2.a) and (2.b) suggest an iterative scheme
analogous to that associated with equations (l.a), (l.b), and (l.c). Recent
preliminary results of Peters and Walker state that this scheme is locally
convergent when E 5=+1. If only the means are unknown, then the scheme is
locally convergent for E s m. As before, these local convergence results are
obtained by showing that the appropriate maps in the equations (2.a) and
(2.b) satisfy local contractibility conditions near a maxtmum-likelihood
estimate.
This iterative scheme appears to be new, and we feel that it holds con-
sideraol , promise. It is as easy to implement in practice as the scheme des-
cribed in the preceding section, and it should converge just as rapidly.
^	 I	
T-
7
Unless m end n are small, it will be a more practical metnod of obtaining
maximum likelihood estimates than Newton's method or the method of scoring, as
described by Kale [5]. Actually, Newton's method and the method of scoring
should require fewer iterations for convergence than this scheme. However,
the computational effort in these methods may be considerable because the inverse
of an (n+l)m x (a+?)m matrix must :,e :alculated at each iteration. The
modified versions of Newton's method and the method of scoring given in [5]
will require the same number of iterations as this method. However, there
is additional computation involved at each iteration for these modified methods.
5. Applications to signature extension and other problems.
The iterative procedures described in the preceding sections appear well-
suited for application to the signature extension problem. This problem has
been characterized as that of developing a computationally useful method of
"extending signatures" from one sample segment to geographically nearby sample
segments. In this context, "extending signatures" means modifying a given
set of signature estimate. Its order to obtain a set which is more useful for
the purposes at hand, e.g., classification or estimation of proportions.
Although incomplete, the results given here are encouraging. The numerical
studies reported in [1], [2], (3), and [4) demonstrate the computational
feasibility of the procedure described in Section 3. The procedure discussed
•	 in Section 4 appears to be no more difficult to implement. All results, both
empirfcu.& and theoretical, obtained so far lead one to believe that the
iterative schemes of Sections 3 and 4 will converge to a maximum - likelihood
r---T
1
1
estimate of the signatures w!^Qnever " reasonably goc' initial signature
estimates are provided. ( Initial signature estimates which lie within a "radius
of local contractibility" of a maximum- likelihood estimate, as suggested by
the results of Sections 3 and 4, can certainly be considered "reasonably good.')
It is our hope that, in practice, initial signature estimates obtained from
Of
	
nearly sample segments" will prove to be "reasonably good" in
this sense.
In addition, we anticipate that these iterative ?rocedures will be pro-
fitably applied to other problems of remote sensing. The iterative scheme for
the equations ( l.a) is shown ii [1) to be a viable approach to the problem
of estimation of proportions. Even more reliable proportion estimates should
result when the remaining equations (l.b) and ( l.c) are also utilized to
provide maximum-likelihood estimates of all the signature parameters. (The
equations (2.a), (2 . b), and their analogues for the covariance matrices can,
of cot :rse, be used to the name end.) Also, an effective solution to the signature
extension problem would appear to be applicable to the adaptive classification
problem. Indeed, this problem, that of continually updating population statistics
on the basis of incoming samples, is clearly seen to be closely related to the
signature extension problem from both a mathematical and a statistical point of
view.
6. Future areas of work.
Despite the encouraging results obtained so far concerning the iterative
procedures described in the preceding sections, considerable research remains
to be done. Generally speaking, the major theoretical problem is to determine
the precise circumstances under which these iterative procedures can be expected
0
z
ii
to converge to maximusl-likelihood estimates. More specifically, the local
convergence results given here must be extended, hopefully to allow a^y subset
(including all) of the signature parameters to be unknown. In addition, it
in necessary to determine quantitatively how near an initial signature estimate
must lie to a maximum-likelihood estimate in order for the iterates ti converge
to a maximum likelihood estimate.
In the absence of more extensive theoretical results, it will be necessary
to run many numerical trials, varying the unkn-wn parameter sets, the true
population signatures, the starting values and other factors, in order to
determine empi:lcally when the iterates can be expected to converge to a
maximum-likelihood estimate. Whether or not further theoretical results are
obtained, numerical procedures need to be studied with an eye toward optimizing
computational efficiency. For example, allowing the covariance matrices to
vary arbitrarily in these procedures will require the calculation of their
determinants and inverses at each iteration. Hence, one might study iterative
schemes in which the covariance matrices are assumed to vary in a particularly
simple way, e.g., by multiplication on the left and right by diagonal matrices.
BIBLI(Y;k"HY
1. W.A. Coberly and B.C. Peters, "The numerical evaluation of the maximum
likelihood estimate of mixture proportauns," to appear.
2. N.E. Day, "Estimating the components of a mixture of nc,rmal distributions,"
Biometrika 56 (1969), pp. 463-474.
•	 3. R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis, John
Wiley and Sons, 1973.
4. V. hasselbl y d, "Estimation of parameters for a mixture of normal distributions,"
Technometrics 8 (1966), pp. 431-446.
5. B.K. Kale, "On the solution of the likelihood equations by iteration processes.
The multiparametric case," Biometrika 49 (1962), pp. 479-486.
6. J.H. Wolfe, "Pattern clustering by multivariate mixture analysis," Multivariate
Behavorial Research, 5 (1970), pp. 329-350.
r
yr
a


Maximum likelihood signature estimation

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server