Monte Carlo algorithm for mean square approximation

Meisel, W. S.

English

NASA Technical Reports Server

J. . 1- . .  This w o r k  was supported in pa r t  b y  (he Nat iona l  Aeronautics 
and Space AdininisIration under. G r a n t  No. N C  - 0 5- 0 18-0-14. - 
. . _  . 
. L  
https://ntrs.nasa.gov/search.jsp?R=19680020913 2020-03-12T10:19:01+00:00Z

. -  
-, where h(x) is a given function, { gi] is a s e t  of linearly independent func- 
- ,  
tion, and x is a vector of dimension n. Simply, we w i s h  to find the best  
\ 
. .  . 
meag-square approximation to h(x) over the regionR. i . c. 
Substantial difficulties may  a r i s e  in the solution of this problem. 
.- The region R may be i r regular ,  the vector x of high dimension, the 
functions F(x) or h(x) complex--all making the prerequisite integrations 
difficult, 
many direct  methods. 
The order  of approximation R may be very high, frustrating 
The probability distribution F may  be unknown, . . 
.. . .  
. defined only by a finite s e t  of samples. Finally, h(x) m a y  be determined 
only by. experiment upon some system (a ''black box"), and hence difined 
'r. 
' . at only a limited number of points [ 1) .  
. .  . .  
. .  
. The work below .studies a n  approach which is of varying usefulness 
* in  all the above cases .  
ra ther  than its analytic form.  
W e  use samples of :he probability distribution 
The samples may resu l t  f rom some physical 
. _  
' .  . 
. process  or be generated artificially [ 2,6,7] .: In either case,  we assume 
: that the number of samples and their dimension is such that the problem 
must be solved without their storage; that is, w e  require  that a sample be 
used in computation only when i t  is f i r s t  generated and not a t  all thereafter. 
W e  further assume that Rj the number of coefficients to be determined, is 
sufficiently la rge  that the inversion of an  R XR m a t r i x  is computationally 
, . .  
". 
. 
. -_ 
* infeasible. This la t te r  assumption is related to high dimensionality: a 
! 
i 
. general  fifth-order polyn 
252 coefficients. 
. *  . .  
. .  
*- 
The Algorithm 
x 
J The procedure proposed solves exactly the problem of minimizing 
M R 2 
j=1 . J i=l 
. 1  
M 
- (h(Y.1 - c a.'s.(y.)) 3. 1. 1 J 
. .  
-, where Y = [y,,y,, . . . ,y ] a r e  sample points generated according to the M 
distribution F. 
the'accuracy of the approximation increasing with M.. In the case where F 
is unknown'and Y represents  our only knowledge of it, 
consider (2) the exact problem. 
This is clearly an approximation Bf the problem in ( l ) ,  
5 
may a s  well 
_ .  
c 
. Setting the partials of (2) with respect  to each cldefficient equal to 
zero gives the necessary condition for a solution vector a = (a U , . . . ,a ): 
1' 2 R: 

8.. by d. aid a.. by c . .  . 
33 . J  1J 1J 
. .  
. .  
\- . 
~. 
,.  
. . .  -4- 
. .  _ _  . _ _ _ _ . .  ~ ....  ... . 
- .  
. - .  . ’ 
:. . . : _ . C - .  . 
Ca,, a,, ..# * 6,) such that 
, .  
- 5- 
, ~. . . .  . . . .. *-. .. 
. -  
By (20) and' (21), 
/ 
.. 
6.. = (g.,d.[ g - c 8 - * * * -  C 8. 1) 13 J i , i  li 1 i-1,i 1-1 (23 1 
. .  
= ii.[a..- c c - ...- c ' *c  -3 9 1 IJ li lj i-1,i i-1,j 
-6- 

. a solution to the original problem (2). The a. may be determined by setting 
1 
. .  - 
c ai = c 8. b.(x) * ,  
1 1  
. .  
and equating coefficients of g.. By induction, one may show that equations 
1 
. .  
(15) and (16) hold. 
The ' reader  may verify that the algorithm can be execut 
Equations (S) - (16) constitute this constructive process. 
! e .  
. ,. :listed. 
-\ 
Uniqueness of Solution 
. .  
> Clearly, solutions to (2) wi l l  not always be unique. If the sample P t' t -  
..I . -  7 
points l i e  on a hypersurface of dimension l e s s  than R ,  infintely many . .  
i 
I solutions a r e  possible. 
i _ I  * .  . I _  . . -  
> 
On the other hand, the construction of the previous section indicates 
. .  that a unique solution exists if the vectors  g ,g  , . , . , g  
pendent, since in that ca se  cl. always esists.  
a r e  linearly inde- 
1 .  2 R 
Since the g.(s) a r e  linearly 
I 1 
. .  . . . .  - .  
e \  . - .  . ’ 
* .  
.. . .  . _  
. I - .  - 
. .  . . .  
. independent functions of high dimension, the previous condition wi l l  be 
_ -  . .  . . .  
. .  satisfied in  practice “almost a lways  ‘I. . .  - 
4 -  . -  . -  I 
. .  . . -  
. ,. 
. . -_ 
. Operation Count 
The algorithm presented wi l l  often give significantly better results 
than solution of (3) directly [ 5 3  if operation count is a valid standard of 
tion of (3) by Gaussian elimination requires 
! 
2 R  - 3 - R  - -  
3 3 
. a  
- <  
. .  . .  
. *  
. - 4 R 3  
a 
’ operations, counting only multiplications and divisions [ 3 1. The algorithm 
a 
presented requires - . 
R 3  R2 4 i-. ---R 
3 
- i - -  6 2 
t 
. .  
:< J .  . _  
operations (multiply and divide) and R square roots. Other than the square 
. .  
. *  
. . . . . . . 
unnecessary; working ‘storage is limited to 
- ’  
‘-or$er approximation, if found necessary,  may be derived with little recal-  
- _. culation. . n  . 
. . .  I .  
- 4  ’ 
If the distribution function is known, the usefulness of the algorithm 
’ will  depend on the ease  of generating random numbers with that distribution 
.~ . -  - ‘ 
relative to the difficulty of direct  solution or  finding functions orthonormal 
with respect to the distribution function. 
mentioned in the introduction may force recourse to the Monte Carlo 
approach. 
. .  
Several of the other complications 
! .  
E . .  
. This algorithm has proved useful in abs t rac t  patt.ern recognition 
. .  
where both h(x) and F(x) a r e  unknown 1 4 1 .  . .  
I 
i 
.: . . 
. .  ._ . .  . .  . "  
. -  I .  
. .  - , 
> .  - .  . .  . .  
j : 
/ c , *  e- ' .  
- , .  . f--.. . .  . -  . 
. . .  
* .  
-- References 
. *  - -  . 
' . I .  
. _. 
El)'  Aieerman, M. A . ,  E. M. Braverman, and L. I. Roxonoer, "The 
. * Method of Potential Functions fo r  the Problem of Restoring the 
.Characterist ic of a Function Converter f rom Randomly Observed 
, Vol. 25, NO. 12, pp. 1705- . .  
1 .- . .  . -. . I 
.' . 
3 Butlerl J. W., "Machine Sampling f rom Given Probability Distribu- 
tions, . ,  =.. Symposium on Monte Car lo  Methods, H. A.  Meyer (ed. ), 
.'. . Wiley, 1956. 
- .  
b. 
. . -  . .  * -  Wiley, .1966. 
' 1 3 )  Isaacson, E , ,  and H. B. i(eller, Analysis of Numerical Methods, 
! >c . . .  
. -  - .  . .  . -  
a 
' [ 4 ]  Meisel, W. S. "Least-Squares Methods in Abstract  Pat tern Recog- 
nition, '' to be published. I 
. .  
[ 53 
. 
. [ 6 1  
Ralston, Anthony, and H. S. Wilf (ea. ), Mathematical Methods for  
Digital Computers, Vol. 1, Wiley, 1960. 
Yon Newnann, John, "Various Techniques Used in Connection with 
Paper  No, 13 in  Monte Carlo Methods, NBSApplied 
Mathematics Series No. 12, U. S. Government Printing Office, 1951. 
- - Random Digits, 
, -  - - I  - - ' [ ? ] .  Votaw, D. F., Jr., and J. A. Rafferty, "High Speed Sampling, '' r . .  
'MTAC, Vol. 5 (1951), pp. 1-8. . 
. .  
\ _ _  
.- . .  . .  
. .  .. 
. \ .  
. .  . -  - .  . .  
- _ .  - - L .  . .  - .  
. .  
, * .  
. - .  
* .  
I .  . 
. .  .. 
. ' .  
. -  
* 
. .  . .  . .  . I  
.- - . .I ..5;<. , . . , . .. - - .. .. - :.. . -. , 
L - 7 .  . 
f .  
. r  . . . 
. ... . ..i-.___. . - - ... ' .  . . .*-*..4.. . . .  
-. 
1, e-. 
\ '  


A Monte Carlo approach to mean-square approximation

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server