Several theorems related to the Householder transformation and separability criteria are proven. Orthogonal transformations, topology, divergence, mathematical matrices, and group theory are discussed

Decell, H. P., Jr.

Smiley, W., III

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19750018670 2020-03-22T21:51:27+00:00Z
f4.
OFh,
r r^ or
•	 DEPA R I M ENT 01:
 MA I I I EMATICS
	 NASA CR.
§J' JNIVERSITY OF HOUSTON
	 HOUSTON ^	 , EXAS
(NASA-CR-11118719)	 HOUSEHULD6R	
N75-26742
TRANSFORMATIONS AND OPTIMAL LINEAh
COMPINATIOuS (HOUSCOn Univ.)	 14 1" HC $3.25
 Unclas
G3/64 27275
NOUSEHOLUER TRANSFORMATIONc IND
CPTIPA LINEAR COMBINATIONS
BY HFNVY P. DECELL,JR.
6 WILLIAM SMILEY
JUNE 1974 RFPORT b 38
`^910117^^^
V
r a.	 ir G
N 1 ^^ ^ G
c	 6^ m
S n	 AF
wry
PRFPAREL FOR
EARTH OBSERVATION DIVISICN.JSC
UNDER
CONTRACT NAS-9-12777
3801 CULLEN BLS D.
HOUSTON, TEXAS 77004
..	 .s	 1 -Mmw.:	 . #	 ^m	 1 •	 w^
HotAA ehotdeh thane Jo4matione and
Op,t cma,t Linear Combinati.one
June, 1974
Hervey P. Dece t, Jh. and W. G. Smiley, ITT
repo tmen.t o6 Ma them tiu
un ,eAAity o6 Houe.ton
RepoA t 0 38
Householder Transformations and
Optimal Linear Combinations
Henry P. Decell, Jr. and W. G. Smiley, III
Department of Mathematics
University of Houston
I. Preliminaries
Notation. Let Mn be the act of linear transformations on R n , and suppose
that On denotes the group of orthogonal transformations on Rn.
Theorem 1. In the norm operator topology on Mn , On is a compact topo-
logical group. (1)
Theorem 2. If AEOn , then det(A) _ *_ 1. [71
Definition 1. The orthogonal transformation A is a rotation, in case
det(A) - 1. Otherwise, A is called a reflection. Suppose that
R is the subgroup of On consisting of all rotations, and let
R denote On %R.
Theorem 3. In the norm operator topology on Mn , On consists of the com-
ponents R and R. Hence, R and R are compact. [1]
Definition 2. Let B denote	 n•{ x II x II e 1 }, the set of unit vectors in p
Theorem 4. In the norm topology on R n , the set B is compact and connected.
•	 Proof. Since B is closed and bounded, it is compact. Moreover, B is the
continuous image of Rn --(O} and hence is connected.
Definition 3. For each x EB, let the Householder transformation H x be
defined by H. - I - 2xxT . Let Hn denote { Hx : x E B}, the set of all
Householder transformations. [5)
(i
Theorem S. The set Hn is a compact, connected subset of R.
Proof. By the continuity of matrix operations, H n is the continuous
image of B under the mapping x -o-I-  2xxT . Thus Hn is compact and
connected. Choose x t B. Since (I - 2xxT ) f - I -2xxT and
(I - 2xxT ) 2 - I, it follows that I - 2xxT a 0U . Also,
det(I- 2xxT ) - -1 implies that I- 2xxT - R.
Theorem 6. (Householder) If y c R 	 61 and x e B. then there exists
a vector w c B such that (I - cwk')y - II y 11 x. 141
Proof. In the case where y - Ily Ilx, choose any w satisfying <w,y> - 0.
otherwise, let w be defined by w	
Y - 11 Y Ilx
	
.
Ily- II y IlxlI	 n
Corollary 7. For each x,y c B, there exists some w c B satisfying
Rw ( y ) - x.
It has been shown by Decell in (21, for optimal selection of linear
combinations (feature selection), that the search for an optimal solution
( k xn, rank k matrix B) may be restricted to the set of k xn matrices of
the form B - ( I k IZ)U, where U is an nxn orthogonal matrix.
H. Walker has shown that, given an optimal linear transformation (Iklz)U,
there exists a positive integer p < min{k,n-k) such that (I k 1Z)U may
be factored into the product (I k IZ)HP • H 1 , for some H l ,...,Hp a Hn.
Note that Theorems 8 and 10, with their corollaries, in addition to
r
establishing the existence of the p < rain{k,n-k) factors Hl,-••lip
yield Walker's result for a very particular sequence of transformations in
Hn (i.e., those derived by Householder's technique for upper triangularization
of UT),
-a
3
These remarks apply to all separability criteria which are invariant
under nonsingular transformation (e.g., probability of misclassification,
Divergence, B lvittacharyya distance, Chernoff distance, Transformed Divergence).
This discussion will be summarized in Theorem 12.
Theorem 8. Let fe 1 ,..., en } denote the usual orthonormal basis for R .
Suppose that { u l , ... I II n ) is any orthonormal set of vectors. Then
there exist transformations H1,...,Hn- 1 c H 	 such that:
(1) if i < n-1, then H i ••• H I uj = 
e 
	 for all j	 1
(2) in addition, if i - n-1, Hn_1 •••H l un - ;en'
Proof. If 1 - 1, by Theorem 6 there exists a transformation H 1 E Hn
such that H l u l , e l . Let p < n-	 Suppose that H...,1i p E tin
have been chosen such that if i < p, then H i • • •H,L j o ej for
all j 
_< 1. Let the vector u - HP '**B l up+ l , and suppose that u
denotes the vector in Rn-P which consists of the last n-p
components of u. Likewise, let e l consist of the last n-p
components of e p+l . Since H i is an isometry for each i =
we have H u II - 1. It follows from the relations <u,e j > - 0,
j - 1,...,p, that u is a unit vector in Rn P, Again using Theorem 6,
choose "
	
In- -2x u- x^_ c Hn-	 sv, h that Hu - e l C Ro G . Define
	
P	 P	 P	 P
'T
	Hp+l =	 I	
Z	
I -2	 Z	 (Zlxn - p)	 a H 
Z	 H	 '^n-p
where each occurrence of Z denotes the cppropriate matrix or vector
of zeros.	 It is evident that if i 
_< p+l, then for all j < i,
i	 ^	 1
4
....4 %
I	 7.
li p+l Hp ••• H I uj	 Hp+lej W	
'1
	 ej - ej . Therefore, by
ZH
induction, ( 1) holds for all 1	 1....,n-1. Finally, given the
Householder transformations H I . .... Hn-1 constructed above, observe
that ( 2) follows from	 :1H 	 N l un I. - 1 and the relations
<H
n-l
... d l un , ej > - 0, j = 1,...,n-1.
Corollary 9.	 If U	 is an orthogonal matrIA,	 then Lhere exist transformations
HI,...,Hn_i E Hn such that:
(Ln- HZI
(1)Hn-1...H1HT
	
1
Z
(2)
I2
- 1	
Z . }1n-
I...HIUT - I
7.	 tl
(3) UTI
(iz--+1 71
Z	 Hn-l...Hl
	
I
(4) for p = 1,...,n-2, there exists a unit vector 
X.,	
Rn
- a
such that
H	 I- 2 Z	 (Z l xn_p
))1p	 Z
p+1	 _p	 Z In- I-Y-n
(5) exactly one of the following holds:
A	
a) Hn-l...HIUT = I
b)	 kinHn_ 1 •.• 11 1 11T 	 i.	 where
Yn = In-1 7	
. I- 2enen E Hn
z	 1
(6) if (S) a) hods, then U	 (UT)-I
	 Hn-1 ... III
and if (S) b) holds, then U - (UT ) -1 
= HnHn-1..,H1
S
(7)	 (IkIZ)HnHn-l...E'I	 -	 (IkIZ)Hk...Hl .
Proof. Since U Is an orthogonal matrix, we also have U T r On . Thus the
columns of U T form an orthonormal set of vectors fu l , ... f un ). Choose
transformations Hl,...,Hn-1 c H, satisfying (1) and (2) of Theor:m H.
Because UT - (u l I ... jun), parts (1) through (6) are immediate
1	 consequences of this theorem. Part (7) follows from the observation
that
•	 (IkIZ)Hk+j - (IkIZ)
for all j - 1,...,n-k.
One should observe that Theorem H may be restated in the following
form.
Theorem 10. Let (ul,...,ud
there exist H1' ... ,Hn-1
(1) if 1 -< r-1,
(2)
Hn-1...H1u1
n
be an orthonormal set of vectors in R	 Men
c Hn such that
then H i ••• H l uj - ej	for all n+l-i < j < n
!e .
1
Corollary 11. Ii U is an orthogonal matrix, then there exist transformations
H Ie ...,Hn-1 E fin such that:
(1) Hn-1^^ 
H1
 
UT1	 ?.
Z In-1
(2) - ±1	 Z` Hn-1
.. . 1i1UT	 I
Z
n 1
(3) UT +1	 Z	
Hn-l..
.
H1	 I
Z In-1
(4) for p - 1,...,n-2, there exists a unit vector x
n-p 
c Rn p
'	 Y6iikt 	 ,aw+^nssriwr+w.rx. 	 .F. 	 .k	 I	 _	 I	
—.  ----- *
	 - -	
I `°-^+r^-- - ---^ ^°-.
6
such that	 T
-p (XT 
IZ) -
	
n--JL Z
	
H^ 1 	I - 2	 n-p
Z	 Z	 Ip
(5) exactly one of the following holds:
a) Ha
-1...HIU7 - I
b) HnHn _ 1 " 'H 1 U1 • I	 where
1
•	 Hn =	 - 1 	Z	 I - 2e 1 1 E Nn
Z	 In 1
(6) if (5) a) holds, then U - H	 • • •H
	
a- 1	 1
if (5) b) holds, then U • Hn 11n-1	 F'I
(7)	 (1kIZ)HnIfn-1...111	
• A(IkIZ)Hn-k.
	 "1
for soine nonsingular k xk matrix A.
Proof. Parts (1) through (6) of this corollary follow directly from
Theorem 10. Part ( 7) folloui by observing that for p • n-k+1,
0 1Z)Hp 	 (IkIZ) Ik-2x kx	 Z	 • (Ik-2xkxk)(IkIZ)
 )
Z
	In-k
and, for p >n-k+l and (n-p) + r = k,
T
	
In_
p
-2xn-
 xn- Z 
I 
Z _
	
I -2x XT	 Z
(I k IZ)Hp	(IkIZ)	 Z	 I	 Z	 n^p	
- 
(IkIZ).
	
Z	 Z Is	 Z	 Ir
A
Theorem 12. I.et tp be any real-valued separability criterion which is
in,rariant under nonsingular transformation. If S -is a rank k, kxn
*-optimal solution of the feature selection problem, then there exist
at most m = min{k,n-k) Householder transformations such that
(Ik IZ)Nm... H I 	!a i-optimal	 (i.e., ^ b (Ik ^Z)Hm ... y 1 ' ^B).
Proof. Recall that B in (Ik IZ)U, for some orthogonal matrix U. The proof
then consists of selecting m to be the smaller of tr and n-I•, and
subsequently applying either Corollary 9 or Corollary 11.
I
811. Separability Criteria and Suggested Algorithms for Optimal
(Subo,^timal) Linear Combinitions
Let ^ be any continuous real function of the matrix variable ( IkIZ)H.
Since Hn
 is compact and V (IkIZ)H . Hn - R, there is an H I E Hn such
that
	
4)(IkIZ)H1	 H.c Hn
b. ^(IkIZ)H
Theorem 13. For each positive integer i, let the element H i
 of Hn be
chosen such that
	
^P( k IZ)HiHi-1
...H
1 	 H•E Hn ^(IkIZ)HHi-1...
E;I
It follows that, for each i,
(1)	
*(IkIZ)Hi ... Hl < ^(IkIZ)Hi+1Hi•..E{I
1
(2) ^(I kI 7.)H 1 "' YH d 5 ^ (I k) ?.)H i+1 i"' 1H	 H	
for every H
(3)
^(IkIZ)HHi...Hl < ^(IkIZ)Hi+1.Hi...Hl for every
(4)
^(Ik IZ)Hi ...Hi_pHH i_ (p
+i)H 1 f ^(IkIZ)Hi+1Hi'..Hl
for every H c Hn
and p - 0,...,1-2.
Proof. As in the proof of Corollary 9, we may choose He H n
 suet
Y	 (IkI Z)H - ( 'k1 Z) and use the definition of ^
(IkIZ)Hi+1Hi^
conclude that
' I	 ^(I k 1 Z)H , ,**H l 
 - 
(I kI Z)HHi"' 1H C (I kI Z)H i+1
 H i ...E
I. 
0
9
Now let H c H  and define H - H i (...(
 . . (Hl HH1)...)... ) Hi.	 By
Theorem 10, inductively conclude that R c Hn , and therefore that
* ( I k IZ)HH i ...H l f y( IkIZ)Hi+1Hi ... Hl*
However, HH I .. . H l M H I*" H I HH 1 .. . H IH i ...H I . HIHi-1...H1H, so (2)
follows. Clearly, ( 3) holds by the definition of Hi+1' In order to
conclude (4), suppose 0 < p < i-2 and H EH n .  For this case, define
uH	 Hl ...fi i PHhi- ( P+l)...H1.
Since N c Hn , statement (4) follows from (2).
Theorem 14. If the hypothesis of Theorem 13 is satisfied and the sequence
{1y (I Z)H	 H }i	
is bounded above,
k^	 i"' 1	 1
then
lim	 l.u.b.4
	
Ii is a positive integer}.
i— (I k IZ)H i ...H1	(IkIZ)Hi...H1
Remark. It is clear that Theorems 13 and 14 remain true if "l.u.b." is
replaced by "g.l.b.", "bounded above" is replaced by "bounded below",
and the inequalities are reversed.
Note that by definition, 	 Divergence, Bhattacharyya distance,
probability of misclassification, etc., all satisfy the hypothesis of Theorems
12 and 13. These theore&s give rise to a sequential monotone procedure for
possibly obtaining a tp -extremal rank k linear combination matrix. At each
t
10
stage in the procedure,the extremal problem is a function of only n variables.
At this point, we conjecture that the process should terminate in at most
min(k,n-k) steps. This conjecture is clearly in line with the min{k,n-k)
representation of the actual TP- extremal solution. In addition, all tests
of the algorithm on real data further lead one to believe that the conjecture
i
	 is fact. It is evident that this procedure is at worst sub-optimal.
S. Marani, in (6), gives details of computational results obtained
using the sequential procedure and 3 very crude differential correction scheme
to solve the n-dimensional variational problem for the generators of the Hits
at each stage of the procedure. The initial guess used at each stage of the
process was arbitrarily set at X	 (
vn
-,	 )T.
Evert using this rough initial guess and the differential correction
scheme , convergence seems to be fairly rapid. Moreover, one would
reasonably expect to reduce iteration time using an Liproved scheme for
successive initial guesses at each stage of the procedure, together with a
more sophisticated iteration procedure.
The results obtained by Marani, despite use of the arbitrary initial
guess at each of the stages, match the results of known Divergence-optimal
solutions calculated by J. Quirein in (2). Note that the total number of
scalar variables involved in this process cannot exceed
w
	
V - n + (n-1) + (n-2) + ... + (n-(m-1))
Om - +t m-1) ,
If
where m v min{k,n-k).
P
rI 
In 131, Decell and Mayekar have c!eveloped an analytical
expression, in the case *= Divergence, for the variational equations as a
function of the Householder generators for every H i . This expression is
utilized in Marini's calculations. We should further point out that these
results only depend on
1. The continuity of
2. The compactness of H
n
3. The invariance of ^ under nonsingular transformation.
The following are several related open questions:
1. Does the process terminate in at most min{k,n-k} steps?
D	 It terminate at an absolute 4-extremum (rank-k
maximal statistic)?
.l. Given 01 and 412	 under what conditions is a 1-extremal
solution also a ^ 2-extremal solution?
4. If the process does not terminate in a finite number of steps,
what can be said of the cluster points of the sequence
{H 1 .. 000 2 ? (Recall that Jn is compact).
I
i I
III. Several Useful theorems
Theorem 15. Suppose that fi x f Hn . Then UHxUT E fins with Ux a
generator of UHxUT , for all U E On . In particular, for every
Hy E tint HyHxHy E Hn . In this case, the Householder transformation
V	
HyiixHy is generated by the vecto. x - 2 <x,y>y.
2	 Proof. Suppose U E On . Since	 II Ux II - II x II - 1, it follows that
UHxUT - U(I- 2xxT )UT + I- 2(Ux)(Ux) T E Hn . If U is chosen to be some
Hy E Hn , then V a generator of H y If X Hy is
Hyx - (T- 2yyT )x = x- 2y(yTx) - x -- 2<x,y>y.
Theorem 16. For each H. and Hy in Hn , there exists some H E H. such that
HE H - If
y	 x
Proof. By Corollary 7, there exists a transformation H E H 	 satisfying
Hy - x. Therefore
HHyH - Ei(I - 2yyT )H - H2 - 2HyyTH = H2 - 2(Hy) (Hy) T = I- 2xxT = tix.
Theorem 17. Suppose that Hx , Hy E tin . Then the following statements arc
equivalent:
(1) <X ,Y > - 0
(2) x # 1y and HXHy=HyHx-Hx+Hy-I.
Proof. Le, <x,y> = 0. Then cleac:y x - *- y. Moreover,
H KHy = (I - 2xxT ) (I - 2yyT) - I - 2xxT - 2yyT - I - 2), 	+ I - 2yyT - I
Hx + H^. - I - Ii  + fix - I - HyHx.
Conversely, suppose H XHy	HyHx . Then
Z - HxHy - HyIIX = 4(xxTyyT _ yyTXXT). Using the fact that
<x,y> = xTy	 yTx, we obtain 4<x,y>(xyT - yxT) = Z. Since x 4*_ y
..	
implies xyT - yxT 4 Z, it follows that <x,y> = 0.
12
I	 I	 k-W
13
Theorem 18. Suppose Hx , Hy c Hn . Then the following are equivalent:
(1) <x,y> _ ! 1
(2) x s t y
(3) Hx - Hy.
Proof. Let HY - Hy . Then xxT = yyT , and hence x(xTx)	 y(yTx). Since
II x 11 2	1, it follows that x - (y Tx)y. Finally,
11 x II	 IyTxI•I) y 11	 implies that	 I <x,v >I - IyTxI	 1.	 The remaining
parts of the proof are immediate.
L
REFi:RENCES
1.	 Chevalley, Claude, Theory of Lie Gi
Press, Princeton, 1946.
2. Decell, H. P. Jr. and Quirein, J. A., "An Iterative Approach
to the Feature Selection Problem", Report 1126 NAS-9-12777,
Department of Mathematics, University of Houston, March 1973.
3. Decell, Henry P. .1r. and Mayekar, Milo, "On the Variational
Equations for Householder Transformations in Feature Selection",
Report 1139 NAS-9-12777, Department of Mathematics, University
of Houston.
4. Householder, Alston S., "Unitary Triangularization of a Nor,-
symmetric Matrix", J. Assoc. Comput. Mach., 5(1958), 339-•342.
5. Householder, Alston S., The Theory of Matrices in Numerical
Analysis, Blaisdell, New York, 1964.
6. Marani, Salma, "Routine to Find the IM-iximum Average Divergence",
Report 1136 NAS-9-12777, Department of Mathematics, University
of Houston, August 1974.
7. Weyl, Hermann, The Classical Groups, Second Edition, Princeton
University Press, Princeton, 1946.


Householder transformations and optimal linear combinations

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server