Methods are given for computing the covariance matrix for the solution vector of an equality-constrained least squares problem. The methods are matched to the solution algorithms given in the book, 'Solving Least Squares Problems.

Lawson, C. L.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19770005845 2020-03-22T12:48:07+00:00Z
J I ^- I ^- ri
t
,V*
NATIONAL AERONAUTICS AND SPACE ADMINISTRATION
Technical Memorandum 33-807
The Covariance Matrix for the Solution Vector
of an Equality-Constrained
Least-Squares Problem
(NASA-CR- 1 49232)	 rHF COVARIANCE MAIRIX FOR	 N77-12788
THE SOLUTION VECTOR OF AN
EQUALITY-CONSTRAINED LEAST-SQUARES PROBLEM
(Jet Propulsion Lab.)	 18 p HC A02/MF A01	 Unclas
CSCL 12A G3/64 55820
r
A
JET PROPULSION LABORATORY
CALIFORNIA INSTITUTE OF TECHNOLOGY
PASADENA, CALIFORNIA
December 15, 1976
^'`0111213 7 
41,5'j^,
l.!]	 N
.r t
	
ANm114PUt 	 ^cyJ
^:.^t h, A7 ^^^ti
33-8U?
PREFACE
The work described in this report was performed by the Infor-
mation Systems Division of the Jet Propulsion Laboratory,
iii
PRWFI)ING PAGE BLANK NOT
v
THE COVARIANCE MATRIX FOR THE SOLUTION VECTOR
OF AN EQUALITY -CONSTRAINED LEAST
SQUARES PROBLEM
1.	 INTRODUCTION
Consider the linear least squares problem
Ex 9E f
subject to the linear equality constraints
Cx = d
We refer to this as Problem LSE denoting Lease Squares with Equality
constraints.
Methods for solving Problem LSE are described in Chapters 20 - 22
of Ref. (1).
In this note we describe methods for computing the covariance matrix
V for the solution vector x. Different methods of computing V will be discussed
which are convenient for use witr each of the different solution algorithms
given in Ref. (1). Any reference to a Chapter, Section, or Page without further
qualification is to be understood to refer to Ref. (1).
We assume throughout that the covariance matrix of f is the identity
matrix. If the covariance matrix of f is known to be something other than the
identity matrix then a preliminary left multiplication :,f E and f by an appro-
priate matrix will produce the desired standard situation. (See Chapter 25,
Section 2. )
We assume that E, C, and d are known exactly, or at least that their
errors are very small relative to those of f.
'
1
33-807
Let C be an m l
 X n matrix and let E be m 2 X n. We assume that
S
ml<n
ml +m2>n
Rank (C) = m 1
Rank 01) = n
With these assumptions Problem LSE has a unique solution vector and all of
the solution methods to be discussed apply without the need to consider unusual
special cases.
As a small numerical example to illustrate the computational methods
to be presented we use the same problem that was used in Chapters 20 - 22.
(See p. 140).
	
C = [0.4087	 0. 1593]	 d = 0. 1376
	
0.4302	 0.3516	 0.6593
E=	 f=
	
0.6246	 0.3384	 0.9666
The computations described in Chapters 20 - 22 were done using a relative
precision of 10 -8 whereas intermediate and final results were rounded to a)-,)ut
four decimal places for publication. In this note we begin with the published
intermediate results when applicable and compute using a pocket calculator.
2.	 SOLUTION METHOD USING A BASIS OF THE NULL SPACE
This solution method is described in Chapter 20, pages 134-141. It
may be summarized as follows.
Apply Householder orthogonal transformations to C from the right to
reduce C to lower triangular form. Apply these same transformations to E
from the right. Denoting the product of these orthogonal transformations by
the nxn orthogonal matrix K these operations may be represented by the equation:
2
- ---------
	F7^7T --77-----
33-807
C ri
 ©^m
K=	 l,,,
E 	 1 E2 IM2
ml n-mI
Solve the following lower triangular system for yl:
C l
 Y l = d
Compute:
f=f - EIYI
Solve the least squares problem:
EZ y2 77
Compute:
K 
Y1
x=
Y2
To compute the covariance matrix, V, :or xy£irst compute the covariance
matrix S for y2:
_ -1
S = (E2 EZ)
Then the covariance matrix for
_ IYZII
Y 	 Y 
is
	 r
00
u= OS
and the covariance matrix for x is
do
(2)
(3)
(4)
3
33-807
V = KUKT
Consider the numerical example given on pp 140-141.
In this example
[0. 1714
E 2 =
0.0885
and
-0.9317 -0.3632
K=
-0.3632 0.9317
Thus using Eq. (3) - (5) we obtain
S = (0.037210) -1 = 26.874
(5)
and
3.545 -9.044
V =	 (6)
-9.094 23.33
Note that although Eq. (3) is a valid mathematical definition of S it does not
represent the most stable way to compute S. if Problem (2) is solved using
Householder transformations, then one would have an upper triangular matrix
R such that
_ [R]QE 2 =	 (7)
0
where Q is m 2 x m2 orthogonal.
a
4
33-807
Then, as is described in Chapter 12, one could compute S as
	
S	 = R -1 (R-1) T
	
(8)
3.	 SOLUTION METHOD USING DIRECT ELIMINATION
This solution method is described in Chapter 21, pp 144-147. It
may be summarized as follows.
Assume column interchanges have been done in the augmented matrix
C
E
if necessary, to assure that the first m l
 columns of C are linearly independent.
Use Gaussian elimination to zero all elements below the diagonal in
the first m 1 columns of
[C],E
	
C d	 Cl 32 d } ml
	
G E f	 0 E2
 f} m2
m 1
 n-m 1 1
Solve the least squares problem:
E 2 x2 a f	 (9)
Solve for x  in
C1 x l = d - C 2 x2	(10)
5
433-807
Then the solution vector is
Xl
x=
x2
To compute the covariance matrix of x introduce the m l x(n - m l ) matrix H,
obtained by solving
C1 H = C2
Then from Eq. (10) we may write
x l
 = %.'1 a - ri x2	ii^
Let a denote the expected value operator. Introduce the mean values
xl = E(x l )
and
x2 = E(x2)
These mean values satisfy Eq. (11), i. e.
x l = C li d - H x 2 	(12)
Subtract Eq. (12) from Eq. (11) obtaining
(X l - x 1 ) 
_ -H(x2 - x 2 )	 (13)
from which we may write
xl - x l	[-H]
X - X =
	
_ 	 (X2x2 - x2	 I
6
(lA)
a,&-
?	 I	 - _ --- I	 __ !	 __-A 	 4--
33-807
Let W denote the (n - m l ) X (n - m l ) covariance matrix of x2 , which from
Eq. (9) maybe defined as
W = (E2 E2)-1	 (15)
Then using Eq. (14) the covariance matrix V of x can be written as
-H	 HWHT -HW
V =	 W[-HT I,
I	 -WHT W
Consider the same numerical example as before, solved by this method. (See
p. 147). We have
[0. 1839
E 2
 =
0.0949
and
H = C 1 C 2 = (0.4087) -1 (0. 1593) = 0.38977
We compute
W= (E2 E 2 ) -1 = (0. 042825) -1 = 23.351
and
	
3.548	 -9.101
V =
	
1-9.101	 23.351
Note that Eq. 1 15) is a valid mathematical definition of W but not a
recommended computational formula. See the remark at the end of Sec. 2 for
suggestions for a more stable way of computing W.
7
Ob..
33.807
4.	 SOLUTION BY WEIGHTING
This solution method is described in Chapter 22. It may be summarized
as follows:
Suppose the data are scaled so that the elements of largest magnitude
in the matrices C and E are approximately the sa.ne  size. Introduce a scale
factor, E, such that E 2 is smaller than the working precision. For instance
set E < 10 -4 for Univac single precision arithmetic and E < 10 -9 for Univac
double precision.
Solve the least squares problem
	
I C 	 d
	
Le	
x =	 (18)
	E
	
i f..
using Householder or Givens orthogonal transformations.
Solving the problem by either of these methods involves triangulariza-
tion by left multiplication by an orthogonal matrix O:
	
[]	
C	 d ^m1
C d
Q	 = EE 1 Ef l ^n - ml
E E	 if	 J
0	 if 	 ^m l
 +m - n
i.	 1
Then x is obtained by solving the upper triangular system
	
C	 d
X=
	
EE 1	Efl
The condition number of this problem is very large (about E -1 ) however this
does not affect the accuracy of the solution because of the special structure of
the matrix and right-side vector.
8
33-807
The covariance matrix, V, of x is
TC -1	 C -1
	
V E2	 (17)
eE 1	EE^`1
C
Even though the triangular matrix
	
	
has a large condition number
EE 1
its inverse can be computed without numerical difficulty.
Consider the example used before. The weighted problem to be solved
(see p. 156) is
	
0.4087	 0.1593	 0.1376
x 
	
0.4302E	 0.3516E	 19 0.6593E	 (18)
x2
	
0.6246E	 0.3384E	 9666 fj
Since we will be using 4 or 5 place decimal arithmetic we could choose any
value of E < 10 -3 . The point is that for any two numbers, a and b, of compar-
able magnitude E should be small enough relative to the computational precision
so that the computed value of a 2 + ( Eb)2 will just be a 2 . For our numerical
example we will not assign a specific value to E but will use the computational
rule that the computed value of an expression of the form a 2 + (Eb) 2 is a 2 when
a and b are of the same order of magnitude.
The data arrays of Eq. (18) can be triangularized by Householder
transformations to obtain the equivalent problem
-0.4087	 -0.:5930	 -0.13760
x 
0	 -0.20698E
	
=	 -0.80403E	 (19)
x2
0	 0	 L 0.43606 fj
9
i	 r
33-807
Solving the nonsingular system represented by the first two rows of Eq. (19)
given the solution vector
13.88
1. 1?74
46
x -
Let R denote the leading 2 x 2 triangular matrix in Eq. (19). We compute
- 2.4468
	
1.88311-1
R-1 -
0	 -4. 8314E -1
hen using Eq. (17) we compute the covarian r _ matrix V of x as
V = r	
_2 R
-1 (R 1 T) =
1
3.5461 -9.0980
9.0980 23.342
This computational procedure looks peculiar in some ways but it is
valid. For e%ample the upper left element of R -1 , namely -2.4468, is entirely
lost in the roundoff error when the product R-1 (R-1)T is computed and this
results i n the computed V being singular whereas R -1 was clearly nonsingular.
This is exactly the right thing to happen, however, since the covariance
matrix V for problem LSE should be singular and should not be influenced by
the upper left element of R-1.
Close analysis of this weigi.fed method (See Exercise 22.40, p. 157)
shows that with sufficiently small 1 this is just a sneaky way of performing the
direct elimination algorithm treated in Sec. 3 of this note (Chap. 21 of the book).
5.	 ONE MORE APPROACH
Still another way of looking at Problem LSE is presented on pp. 141-
143. As is noted there we expect that this approach may not have practical
value but may be of theoretical interest.
10
33-807
Let K be the n x n orthogonal matrix defined in Sec. 2 of this note
(Chapter 20 of the book). Let K be partitioned as
K = [Kl
	
K2 l^n
m l n - ml
Define
E = (EK2 ) (EK2 ) + E
where the superscript 'Y' denotes pseudoinverse. Define
C
A
A =
A
E
Then, as is proved in Chapter 20, the least squares solution of
[d]
A
Ax =
f
is the same as the solution of problem LSE:
Cx = d
Ex f
To compute the covariance matrix of x, regarding x as the solution of
Eq. (20), we first write
A 
+
[d]
X = A
f
MW
(20)
2
11
33-807
d
Assuming the covariance matrix of f is
0 0 } ml
0 I Im,
m l
 m2
it follows that the covariance matrix, V, of x is
0 0
	
V = A+	 A
+T
0	 I
From Eq. (20. 30) on p. 141 we know that A + can be written as
A + = [ C + - K2 (EK 2 ) + EC + , K2 (EK2 ) +1 'jn	 (22)
V
	m l	m2
Substituting Eq. (22) into Eq. (21) gives
	
V = K2 (EK2 ) + (FK2 ) +T K2	(23)
From Eq. (1) we have
EK2 = E2
and thus Eq. (23) can be written as
V = K2 E 2 E 2 T K2
K2 (E 2 E 2 ) -1 KZ
This last expression is identical to the right-side of Eq. (5). Thus we
obtain the same representation of V as in Sec. 2.
12
(21)
No-
__ - 	 I 	 i	 I
	 I 	 I!	 i1.
33-807
REFERENCES
1.	 C. L. Lawson, and R. J. Hanson, "Solving Least Squares Problems",
Prentice-Hall, 1974.
13	 NASA—M—Cant. L.A.. Cold


The covariance matrix for the solution vector of an equality-constrained least-squares problem

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server