Classifying large quantities of multidimensional remotely sensed agricultural data requires efficient and effective classification techniques and the construction of certain transformations of a dimension reducing, information preserving nature. The construction of transformations that minimally degrade information (i.e., class separability) is described. Linear dimension reducing transformations for multivariate normal populations are presented. Information content is measured by divergence

Decell, H. P., Jr.

Mayekar, S. M.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19770007852 2020-03-22T12:29:20+00:00Z
t^^lv of yo
^^l^^aff^G^ DEPARTMENT OF MATHEMATICS 	 ^IV	 ^31
2 UNIVERS!TY OF HOUSTON	 HOUSTON, TEXAS
v ^deci \A
(NASA — CR-151131)	 FEATUidh COMBiNATIONS AND	 N77-14"95
THE DIVERGENCE CRITERION (houston Univ.)
13 p HC A02/MF A01	 CSCL 12A Unclas
G3/64 56938
FcATU"F CCMBINAT IONS AND THE
UIVERGEN'; E CRITERION
P.Y tip. ijECFLL AND S- F . "AYEK4R
REPORT #56 JUNE 1976
1:54 Dee
<<^ NASR^^V
6
^^^1«0^61
PPCPARED FOP,
EARTH PBSFPVATION. DIV1cI'iN, J^C
UNGEa,
CONTR^CT NAS-9-15000
! i
HOUSTON, TEXAS 77004
.mil
	
-iii
	 _ II 	 _
FEATURE COMBINATIONS AND THE
DIVERGENCE CRITERION
by
Henry P. Dece.U, Jn, and Shaituh M. Mgyekah
Depantmen-t o6 Math emati.a
UnivM ty o6 Hcua.ton
Houz ton, Tex"
Report 056
NASA Contact NAS- 9-15000
r• ^
FEATURE COMBINATIONS AND THE
DIVERGENCE CRITERION
Henry P. Decell, Jr. and ihailesh M. Mayekar
Department of Mathematics
University of Houston
ABSTRACT
Classifying large quantities of multidimensional data (e.g.,
remotely sensed agricultural data)(Remote, 1968) requires effi-
cient and effective classification techniques and the construction
of certain transformations of a dimension-reducing, information-
preserving nature. This paper will deal with the construction of
transformations that minimally degrade information (i.e., class
separability). We will only consider the construction of linear
dimension-reducing transformations for multivariate normal popu-
lations and information content will be measured by divergence
(Kullback, 1968).
1. INTRODUCTION
For n-dimensional normal classes N(m i ,V i ) i = 1,...,m, the
divergence between class i and j (Kullback, 1968) is given by
Dij 2t r[(Vi— V
1 
)(V
1
— + ?tr[(Vil+Vjl)(mi—mj)(mi-mj)T]
Letdij = mi-mj . Then
Dij s 2 r[(Vi-Vj )(Vj l-Vil ) j + Ztr[(V il+ Vjl)(aij)(dij)T]
t2 r[V il (Vj + 6 ij 6ij )] + 2 r[V j l (V i + 6 ij 6ij )j - n.
The interclass divergence (Decell and Quirein, Oct. 1973) for m
populations is given by
-1
D = ^ ^ Dij
1 ^
	
. "	 i#.j
and it follows that
	
D = 2tr[
	 Vi l (	 (Vj + 6 ij 6i j ))] - m(
2-1) n
i#j
	
r[	
Vil S1] - m(2-1) n
1
where
	
Si =	 (Vj + 'iij dij) .	 '.
i#j
If B is a k x n rank k matrix, the B-interclass diver-
gence (Decell and Quirein, Oct. 1973) is given by
	
DB	 DB(i,j)
i#j
t	 ^
DB = 2 trj^ (BV iBT ) `1 (BS iBT) l - 
m(m- 1)
 k.
As in the case of average interclass divergence, the B-interclass
divergence is a measure of the "separation" in the classes
r	
T(Bmi,BViB ) i = 1,...,m, and is a useful tool for constructing
rank k linear transformations that preserve "class separability".
It has been shown (Decell and Quirein, Oct. 1973) that whenever
D = DB , the probability of misclassification (Anderson, 1958) for
the classes N(Emi ,BV iBT), i = 1,...,m is the same as the probabili-
ty of misclassificaticn for the classes N(m i ,Vi ), i = 1,...,m.
2. THEORETICAL PRELIMINARIES
We will assume that k is an integer (k < n) and develop a
procedure for selecting a k x n rank k matrix B such that D  is
maximum. The procedure will be based upon the following theorem
(Decell and Smiley, to appear). We will let C = {u e Rn: 11u11=1}
and T(H) = 1 H = I-2uuT : u -, C'1 denote the set of Householder
transformations defined on Rn
 (Householder, 1968).
Theorem. For each positive integer i let H, a T(H) be inductive-
ly chosen such that
D(Ik IZ)HiHi-1
...H
1	 HeT(H)[D(IkIZ)HH1-1 " .Hll
where
D(IkIZ)H1	
1,u.b. D(IkfZ)H.
The following hold:
(1) D(Ik IZ)HiHi-l
.,,H
l L D(IkIZ)Hi+1Hi,,.gl•
(2) D (Ik IZ)Hi Hi-1 ••• H H L	 kD(I (Z)Hi+1 Hi 
••• H 1' for every H c T(H).
1
(3) D(1k IZ)HH i1; i-l
...H
1 L D(Ik IZ)Hi+1Hi . 'Hl' for every H c T(H).
(4) D(IkIZ)HiHi-l...Hi-(p-1)HHi-(p+1)...111 L D(IkIZ)Hi+l...Hl,
fcr every H e T(H), ,, = 0,1,...,1-2.
(S, The monotone sequence
ID  }co = {D (IIZ)H •••H 100	 is bounded above,i i= 1	 k	 1	 1 i=1
and hence
	
imD(Ik^Z)H1	 H1 = l.i.b. {D(IkIZ)'Hi...Hl}.
We would, of course, be pleased if it were the case that
l.i.b. {D(IkIZ) 
Hi­ 
Hl} = D. This, unfortunately, is not always
the case for some choice of k < n and is not possible, in general,
for any k < n. We do know that there is some k X n rank k
matrix B fcr which D  is maximum and, in general, that D  L D
(Decell and Quirein, Oct. 1973). It follows, moreover, that since
the matrices of the form (I k 'Z)Hi ••• H1 have rank k,
D(I k' Z)H  i •	 1H !L- D 
	 D for every integer J.
	
We will call the sequence {D(Ik.1Z)H, „ *H1}0
	
suboptimal
whenever
l.u.b. {D (I IZ)H ...H } < D 
i	 k	 i	 1
(and optimal in the case of equality).
There are several open theoretical questions that deal with
the conjecture that the sequence is, in general, optimal and co-
	 E
finally constant beyond the index i = win{k,n-k} (Decell and
Smiley, to appear). In what follows we will develop a procedure
for constructing the subject sequence and demonstrate its
application to agricultural data.
.a-. 1
J
v................
3. THE GRADIENT OF DB
It has been shown (Quirein, Nov. 1972) that the differential
dDB of D 	 (regarded as a function of the k x n matrix S) can
be expressed in the fo rn dDB = F + G, where, when the indicated
inverses exist,
1	 T -1	 T	 T
_F = 2tr[	 (BV iB ) (d}i S iB + BS dB) ]
§w•
fir[	 (dB SiBT)(BViBT) -1]
+ 2 r[	 (BSi dBT ) (BV i BT ) 1)
3
3
M
= tr[;q (dB SiBT)(BV,BT)-1]
and
	
-G= - 2tr[	 (BV iBT ) -1 (dB ViBT + BVidBT;(BVi BT ) -I (BS iBT) ]
	
- -
21tr[	 (dB V i BT)(BViBT) -1
 (BS iBT)(BViBT)-1]
- -12-tr[^ (BV iBT) -1 (BS iBT)(BViBT ) (BV idBT)]
tr[^. (dB ViBT)(BViBT ) -I (BS iBT)(BViBT) -1 ].
dDB = tr[^ dBCS i BT - ViBT(BViBT)-1(BSiBT)}(F
Thus,
= tr	 dB Q 
where
Qi = FS iBT - ViBT(BViBT)-1(BSiBT)}(BViBT)-1].
We are, of course, interested in extremizing D 	 over the
particular subclass of k x n rank is matrices of the form
(Ik 'Z)H where H e T(H) (e.g., for i = 1 we find H 1 that maxi-
mizes D (IkIZ)H ). Actually, one need only consider what is re-
quired to compute H 1 . The computation of H2
 is accomplished by
the same procedure as that for H 1 . It is s_mply a matter of,
after selecting H l , redefining the m classes to be
N(111mi9 H 1ViH1 ), i = 1,...,m and proceeding as in the selection of
Hl.
With these facts in mind we will simply calculate the gra-
dient of D  where B is restricted to having the form
B = (Ik iZ)H, H E T(H). The restrictions H E T(H) can be accom-
plished by considering those k x n rank k matrices of the form
T
B = (Ik IZ)(I - 2 w1 ), w c Rn (w # A)
W w
It follows that
T
dB = d[(Ik 1Z) (I - 2 wT )J = -2(I.IZ) d(wwT/wTw)
w w
-2(L
 IZ) [wTwd(wwT) - wwTd(wTw))	 (wTw)2
Where Mi = wwT0 i (I k JZ) and Ni = Qi(IkIZ)WWT.
dDB
 = T2 2 tr[	
{wT Mi dw - wT
 Ni dw + Ni w
(w W)	 1
=
w
TW 2 tr(^{dw ` Mi w - dwT Ni w + Ni w
(	 )
2(I Iz)
T 2 [wTw(dw w T + wdwT) - ww
T (wTdw + dwT w)J
(w W)
_ - 2(1kI Z2 [dw w T w wT + w wTw dwT - w wTdw wT - w dwTw wT]
(W W)
i	 I2I
l T Z[(dw wT - wdwT)wwT - w^.4T (dw wT - wdwT)]
(w w)
Substituting the latter in the expression for dDB,
^I
	
dDS 6 tr
	 [ -	 T. 22 {(dw wT - wdwT)wwT - wwT (dw wT - wdwT)}Qi]
w w
	
tr	 [ - 2QiTIk2Z) {(dw wT - wdwT)wwT - wwT (dw wT - wdwT)}]
3=1	 w w
tr	 T2 2 [WwT Q  (Ik 1Z)(dw wT - wdwT )	 .i=1 w w
- Ql (Ik ^Z)wwT (dw wT - wdwT)]
= T2 2 tr	 [Midw wT - MiwdwT - Nidw wT + NiwdwT ]	 I
w w	 i=1
dD8	 T2 2 tr[{Mi wd wT - N. w dw T + Ni w dwT - M  w dwT}]
(w w)	 ;_^
T2 2 tr[	
{(Mi - Ni)T - (Mi - Ni ))w dwT).
(w w)	 i-
-
The necessary condition that w he extremal is then,
G(w) = - ,2r
 2 tUMi -Ni )TNi)}
w = 8 (the zero vector).
(w w) 
We note that G(w) is the gradient of D (I IZ)(I 	 2'TT) and
k	
w 
use a steepest descent procedure For finding the extremal w. The
process is repeated for each sequential index until corresponding
values of divergence "-tabilize." Test results are presented in
the following tables. The C-1 flight line data is twelve channel
data for nine agricultural classes: soybeans, corn, oats, red.-
clover, alfalfa, rye, bare soil, and two types of wheat. The Hill
County data is sixteen-channel data for five agricultural classes:
winter wheat, fallow crop, barley, grass, and stubble.
The starting value w 	 for the steepest descent procedure
for selecting each successive Householder transformation
H19 H2' H3 ... was arbitrarily chosen to be wo = ( 1, 1,..., 1)T,'
Choosing starting values in this arbitrary fashion is certainly
not the most clever thing to do in the presence of the monotone
behavior of the sequence 
D (I Z H •••H	 One would expect, for
k l ) i	 1
example, that the starting values for th.^ selection of Hi+l
should depend upon the unit vectors previously selected as gener-
ators of Hi,Hi_1,...,H1 in such a way as to guarantee that the
starting value wo , for the descent procedure for selecting Hi+1'
lsatisfies
T
D ( Ik IZ) t{ i...11I K-. D (Ik !Z)(I - 2w0w0 )Hi...Hl
w w0 0
This rather rrbitrary selection of the starting vector does, as
the examples demonstrate, violate the latter inequality. The
question about how to choose starting vectors, according to the	 r• - a
latter inequality, is still an open oae and its answer would cer-	 O
tainly decrease computation time.
C-1 Flight Line Date
n=12, k= 6, m=9, D=10,660
Iteration for H1
No * Divergence	 D 
1 1982
2 3536
3 4533
4 5781
5 6910
6 7522
7 7710
8 7790
9 7838
10 7865
11 7881
12 7892
Hill County Data
n= 16, k=8, m= 5, D=636
Iteration for 111
No * Divergence	 D 
1 114.58
2 136.66
3 152.27
4 179.69
5 223.81
6 247.42
7 252.78
8 257.12
9 260.74
10 263.95
*Iteration counter
AM
No Divergence	 D 
1 371.12
2 594.75
398.623
4 400.69
5 402.03
6 402.98
7 403.74
*Iteration counter
C-1 Flight Line Data (cont.)
Iteration for H2
No
Divergence	
D 
1 7815
2 8797
3 9542
4 9785
5 9901
6 9966
7 10 1, 005
8 10,031
9 10,048
Hi.li County Data (cont.)
Iteration for H2
No Divergence	 D 
1 269.00
2 280.48
3 293.32
4 300.68
5 304.0 7
6 306.19
7 307.74
8 30 8.95
9 309.93
i
i
Iteration for H3	 Iteration for H3
No Divergence	 D 
1 7582
2 8705
3 9809
4 9947
5 9995
6 10,020
7 10 , 037
8 10,049
9 10 , 058
No Divergence	 D 
1 312.1-8
2 344.52
3 380.83
4 387.20
5 391.70
6 392.96
7 394.58
8 399.4 7
r
Iteration for H4
.I
► . _A ^
i i I I i^^-^
B I BL IOGRA-'HY
Anderson, T.W. (1958). An Introdccction to Multivariate Statisti-
cal Analysis. New York: John WLley & Sons, Inc.
Decell, H.P. and Quirein, J.A. (Oc:tob.?r 1973). "An Iterative
A)proach to the Feature Selection Problem". IEEE Cat. $CHOE
pp. 3B-1-3B-12.
Decell, H.P. and Smiley, W. (to appear). "Householder Transfor-
mations and Optimal Linear Combinations." Comm. in Stat.
	 ^I
Householder, A.S. (1968). "Unitary Triangularization of a Non-
Symmetric Matrix." J. Assoc. Comput. Mach., pp. 339-342.
Kullback, S. (1968). Information Theory and Statistics. New
York: Dover Publications.
Quirein, J.A. (November 1972). "Some Necessary Conditions for an
Extremum." Report #12 NAS-9-12777. Dept. of Mathematics,
Univ. of Houston, Texas
"Remote Multispectral Sensing in Agriculture." (1968). Report of_
the Laboratory for Agricultural Lemote Sensing Vol. 3,
Research Bulletin 11844. Purdue Iniv., Lafayette, Indiana.
Lr


Feature combinations and the divergence criterion

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server