Stochastic approximation scheme with accelerated convergence propertie

Newbold, P. M.

English

NASA Technical Reports Server

Office of Naval Research  
2 ont r ac t N O  0 0 i 4 - 6 i - A - 0 2 38 - 0 0 0 6 
NR - 372 - 012 
National Aeronautics and Space Administration 
Grant NGR -2 2 - 007 - 068 
A STOCHASTIC APPROXIMATION SCHEME WITH 
ACCELERATED CONVERGENCE PROPERTIES 
BY 
P. M. Newbold 
Technical Report No. 545 
Reproduction in  whole or in par t  is  permit ted for 
any purpose of the United States Government. 
October 1967 
The r e s e a r c h  reported in  this document was made possible through 
support extended the Division of Engineering and Applied Physics ,  
Harvard University by the U. S. Army Research  Office, the U. S. 
Air  Force  Office of Scientific Research and the U.  S. Office of 
Naval Research  under the Joint Services Electronics  P r o g r a m  by 
Contracts N00014-67-A-0298-0006, 0005, and 0008 and bythe National 
Aeronautics and Space Administration under Grant NGR-22-007-068. 
Division of Engineering and Applied Physics  
Harvard University Cambridge, Massachuset ts  
https://ntrs.nasa.gov/search.jsp?R=19680003748 2020-03-12T11:46:13+00:00Z
A STOCHASTIC APPROXIMATION SCHEME WITH 
ACCELERATED CONVERGENCE PROPERTIES 
BY 
P. M. Newbold 
Division of Engineering and Applied Physics  
Harvard University Cambridge, Massachusetts 
ABSTRACT 
An examination of the f i rs t  o r d e r  s teepest  descent method 
for  minimizing deterministic functions leads to  the specification 
of sufficient conditions assur ing  the convergence with probability 
one of a new stochastic approximation algorithm. The algorithm 
is interpreted as a stochastic descent method using a constant, as 
opposed to a continually decreasing s tep length. The c ruc ia l  con- 
vergence condition i s  that the variance of the gradient should be 
zero  as well as a minimum at the solution point. 
The rather  narrow c l a s s  of problems to which the new 
algorithm is applicable can be extended to include many useful 
problems by employing accelerator  methods adapted f rom the 
field of Monte-Carlo techniques, both to sat isfy the condition, and 
to acce lera te  convergence. 
Simple i l lustrative examples demonstrating the effective - 
ness  of the new algorithm a r e  presented. 
1 T-A- -~- - . .&: - -  
I . M I L  I UUUL L l U l l  
The convergence conditions for the basic stochastic 
approximation algorithm originated by Robbins and Monro [l] 
a r e  now well known. Important resul ts  are due to  Blum [2], 
Dvoretzky [3], and Gladyshev [4]. The cruc ia l  condition for  
convergence with probability one i s  that  the step s ize  should 
progressively decrease  in  such a way that the stochastic var ia -  
tion in  the correct ion t e r m  dies out. 
Although the stochastic approximation algorithm can be 
given numerous interpretations,  one par t icular ly  fruitful point 
of view i s  to in te rpre t  i t  as a stochastic descent method for  
finding the expected minimum of a stochastic function. In this  
light it i s  logical to investigate possible correspondences with 
determinis t ic  descent algorithms. The determinis t ic  counter- 
pa r t  of the Robbins-Monro algorithm, the first o r d e r  s teepest  
descent method, differs only i n  that a constant s tep length can 
be used, thereby resulting in a fas ter  convergence for a n  
equivalent problem. 
is  taken to mean that deterministic problem which is identical 
with some stochastic problem under consideration except f o r  the 
absence of the pr imary  random variable(s).  
In this paper the term 'equivalent problem'  
The present  paper is concerned with developing a 
stochastic approximation algorithm whose ra te  of convergence 
i s  comparable to that of the deterministic s teepest  descent 
method for the equivalent problem. This algorithm uses  constant 
step lengths a s  opposed to the conventional schemes,  and is 
shown to converge under the appropriate conditions with pro-  
bability one. 
2 .  Convergence of the Deterministic Algorithm 
As an introduction it i s  useful to derive briefly sufficient 
convergence conditions for the deterministic steepest  descent 
algorithm; i t  will  be found in  the next section that these a r e  very 
s imilar  to  the stochastic convergence requirements .  
the deterministic algorithm 
Consider 
for minimizing with respect  to u some function H(u). 
A aHL (uk) 
i s  assumed to exis t  everywhere; Uk i s  an 
auk 
(uk) = 
n-vector. 
in  f(u) ) a t  u = 0. 
under the following conditions : 
Let there  be a unique solution point (i. e . ,  a zero  
The algorithm converges to  the solution point 
the inequality being s t r ic t  except a t  the solution T (a) \f(q 3 0 ,  
point; (2) 
Proof:  
Forming the inner product: 
-2-  
Using the inequality of Eqn. ( 3 ) ,  
T 2 T  T u u - { 2 P  - P d}Ukf(uk) ; U k t l  U k t l  k k  
and ii Eqn. (4 j  is satisfied 
Hence i f  u 
Taking the infinite sum of Eqn. ( 5 )  
i s  finite, then uk and f(u ) a r e  finite for all k > 1. 1 k 
m 
T 
G U ~ U  1 1  - {Zp - p‘d} 1 uzf (u , )  ; 
k= 1 
ukf(uk) < a  , which must then imply that f(u ) + 0 uk’ k hence 
k=  1 
and so the algorithm converges.  
3 .  Convergence of the Stochastic Algorithm 
In considering the stochastic approximation algorithm, 
nconvergencen will be used in  two senses .  “Deterministic 
convergencen will be taken to mean the convergence of the 
algorithm in expectation; that i s ,  E\ --i* 0 ;  
%tochastic convergence ‘I will be taken to mean the reduction 
of the variance of the correct ion te rm p f ( x )  to ze ro  as the 
solution point is reached. 
very s imilar  conditions hold on the one hand for the convergence 
of the deterministic algorithm given above; and on the other f o r  
k + . 
In this section i t  will be shown that 
the deterministic convergence of the stochastic algorithm to be 
de s c 1- ibe d . 
-3- 
The problem is now to minimize with respect  to u the 
function EH(u, 5 )  where E is a vector of random variables  with 
zero expectation and finite variance.  The stochastic descent 
algorithm is 
- - Uk - pf(uk) 9 k t l  U (7) 
where as before the s tep length P i s  a constant, but now 
which is a random variable .  
moments of f(u ) exist ,  and let the unique solution point be 
Suppose that all the relevant 
k 
u = 0 at  Ef(u) = 0. Then the algorithm converges to the solu- 
tion point under the following conditions: 
(a) ,,TEE f(u k ) 2 0 , the inequality being s t r ic t  except at the 
solution point; ( 8 )  
Here E 
of the random variable E 
Proof: 
denotes expectation with respec t  to the cur ren t  sample E 
k ’  
Taking the total  expectation (i. e . ,  on all samples  of 5 
from f, through E ) of the inner product fo rm of Eqn. (7)  k 
Using Eqn. (9) af ter  taking i t s  total  expectation also, 
-4- 
and if Eqn. (10) is satisfied 
T d Eu u T EUk+l U k t l  k k '  
T 
k k  
Assuming the moments of u1 t o  be finite then u u i s  a super -  
martingale [ 51. Again taking a n  infinite sum of Eqn. (1 1 ) 
T T 2 T 
Eu,u, d Eu 1 1  u - { Z P  - P d } C  Eukf(uk) , 
which implies that 
i T  Eukf(u k ) c w . 
k= 1 
Using the expectation of Eqn. (9 ) ,  Eqn. (13) implies that 
and since it is universally t rue that 
then 
(13)  
Hence Ef(u ) + 0 (but not necessarily monotonically unless Ef(u) 
is monotonic in u ) ,  and so u + 0 a s  k -3 13 ; and the stochastic 
algorithm converges in expectation. 
k 
k 
The close similari ty in the conditions for each of the two 
proofs is  to be noted. Intuitively i t  may  be expected that the 
ra tes  of convergence (in the sense of deterministic convergence 
- 5- 
in the stochastic c a s e )  will be substantially the same for both 
the stochastic problem and i t s  equivalent determinis t ic  problem. 
Only a few fur ther  steps a r e  required to show conver- 
gence with probability one under the same conditions. 
Proof: 
It has a l ready been shown that 
co 
T v 2 Eukf(u k ) < OD 
k= 1 
Hence over some subsequence of the sequence {k} 
T 
k k  E f ( \ )  + 0 with probability one. But u u is a super -  
$ 5  T martingale, implying that u u and hence a l so  u converge 
with probability one to  a random variable [5].  
k k  k 
These two 
facts a r e  sufficient for convergence of the algorithm with 
probability one to the solution point. 
4. Interpretation of the Convergence Conditions 
It is easily seen that the implication of the convergence 
condition of Eqn. ( 9 )  is that a t  the solution point not only is 
Ef(u) zero,  but so  a l so  is Var{f(u)}. 
from the corresponding condition for the original stochastic 
approximation scheme. 
[4] there  is the looser requirement that Var{f(u)} should have 
some finite minimum, not necessar i ly  zero,  a t  the solution 
point. 
This differs quite sharply 
As is given by Gladyshev for  example 
At f i r s t  sight this is a monumental difficulty, since very  
- 6- 
few functions f(u) ,  excluding even the l inear case ,  satisfy 
this  condition. 
to problems in which there i s  simple multiplicative noise. 
Example 1: 
Apparently the new method can only be applied 
Let f(u) = u t uf where f has ze ro  mean and unit variance.  
This sat isf ies  the conditions given by Eqns. (8) through (10); the 
solution point is at  u = 0. 
simulation of the descent algorithms over 2 5  i terat ions.  
the new algorithm (curve 2 )  i s  a considerable improvement over 
the original stochastic approximation scheme (curve 1) where in 
this  case  the s tep length P decreases  a s  l/k from i t s  initial 
value. 
the steepest  descent method i s  given for comparison (curve 3 ) .  
Figure 1 shows the resu l t s  f rom the 
Clear ly  
The solution of the equivalent deterministic problem by 
5. Application of Monte -Carlo Accelerator  Techniques 
The c l a s s  of problems which can  be solved effectively 
by use of the new stochastic dzscent algorithm can be consider-  
ably enlarged by borrowing accelerator  techniques from the 
field of Monte-Carlo methods. To  date there  seems  to have 
been little cross-fert i l ization between these two fields: the 
monograph of Hammersley and Handscomb [6]  has been found 
very useful in the development of the work in this section. 
The so-called naive Monte-Carlo method i s  itself a 
stochastic approximation scheme of the original type using a 
harmonically decreasing step length. Let  f .  be a sequence of 
1 
i 0’ 
!OO 
-! 
10 
-2  
10 
l lull 
- 3  
10 
-4  
10 
12 
loc 
f (u) = u + u t  
E t  =O; Varc z i . 0  
0 5 40 15 20 25 30 
FIG. 1 MINIMIZATION OF A SIMPLE FUNCTION WITH 
SAMPLES k 
PURELY MULTIPLICATIVE NOISE. 
< ,.,,,,,,.dent -AC.nT)- randem s a m p l e s  from. s e m e  prnhabili ty distribution; 
- 
let g ( e . )  be a sequence of random variates  with expectation g. 
1 
Then u = g(ei)  i s  an unbiased est imator  of E .  This sum- k k  
i =  1 
mation can  be written iteratively in  the form:  
which is  immediately recognizable. 
It might be expected that Monte-Carlo acce lera tor  
techniques would be of considerable use in  the present  algorithm, 
in  the same  way that they a r e  for the conventional stochastic 
approximation algorithm. On examination, two methods in 
par t icular  a r e  immediately applicable; the antithetic var ia te  
method, and the control variate method. 
(a) The Antithetic Variate Method 
The basic idea of the antithetic variate method is to use 
instead of the sample g ( e . ) ,  a linear combination of two negatively 
correlated samples such that the expectation remains  unchanged. 
If 5 .  i s  a sample from a symmetr ic  distribution with ze ro  mean, 
and g(5,) i s  predominantly odd in 5. 
1 
1 
then the substitution 
1' 
i s  both simple and effective. 
Consider now the application to  the new stochastic approxi- 
mation algorithm, replacing f(uk) by f"(u ) in  Eqn. ( 7 )  where k 
- 9- 
Under the above assumption on the symmet ry  of the distribution 
for ek, E?(\) = Ef(uk) so that if the modified algorithm con- 
verges, it  will do so to  the solution point. If i n  addition 
.L 0,.
where u represents  the solution point u = 0, then exactly as i n  the 
equivalent deterministic problem, convergence to the solution 
point can be achieved over some range of P for functions f(u) 
which do not increase too fast with u (i. e . ,  Eqn. (9) can be 
satisfied over all the interesting range of u ) .  
Example 2: 
t For  this example 2 k k  Let f(uk; = (1 t 5 )u t 0 .  I ( , )  
Eqn. (9)  does not hold, so the new dlgorithm will not converge 
with probability one; the original stochastic approximation 
scheme with decreasing as l /k  does so however. On the other 
hand 
r"c'k) = ( l  Ek)\ 2 
which satisfies Eqns. (9)  and 
3 2 ' '* l(uk ' 3%E,) 9 
14)'k, so that the moc ified new 
algorithm will converge with probability one. F igure  2 shows 
the resu l t s  of simulation of the various algori thms over 2 5  itera- 
tions. The modified new algorithm (curve 3)  again converges 
:$ 
probability of the algorithm diverging, but in  pract ice  this does 
not invalidate the scheme. 
Actually in  this example for any finite P there  is a small finite 
-10- 
40' 
toc 
-I  to 
- 2  
to 
I lul l  
-3 
io 
I-d4 
I I I I I 
HARMON I C p =O. 5/k 
CONSTANT p = 0.5 
SAMPLES k 
FIG. 2. MINIMIZATION OF A FUNCTION USING ANTITHETIC 
VAR I ATE S . 
considerably faster  than the original stochastic approximation 
algorithm (curve 1).  If the antithetic variate technique i s  not 
used, stochastic convergence is not a s su red  (curve 2 ) .  
(b) The Control Variate Method 
Essentially the control var ia te  method spli ts  the var ia te  
g( ti) into the sum of two pa r t s :  g(Ei) = ;(si) t gc(Ei). The spli t  
is made so that the expectation of g ( E . )  may be found analytically. 
Let this expectation be E ;  then a n  unbiased est imate  of the 
expectation of g(ei), denoted by E ,  is given by 
rcI 
1 
i =  1 
and this estimate will have a smal le r  variance than the est imate  
computed by the naive Monte-Carlo method for the s a m e  value 
of k. 
Turning to the present descent method for solving Ef(u)  = 0 ,  
a similar  split can be made:  
Ef(u) = EF(u) t Efc(u) 
where E?(u) = f (u)  can be found analytically. 
the sample f (u  ) i s  replaced by 
, 
A 
Then in Eqn. (7)  
k 
A 
(‘k) * 
Used by i tself ,  the control variate method will not be able to 
widen the c l a s s  of problems which the new stochastic descent 
algorithm can handle unless fortuitously ck can be contrived 
only to  enter  multiplicatively into f (u,). C However used in 
-12- 
As in the previous example f(u ) = (1 f ck)uk 2 f 0. 1(uk f f k )  3 . k 
The antithetic variate saniple f ' ( u k )  given by Eqn. (16) is split 
into the two par t s :  
F o r  this example, then, f ( u  ) = 2uk if 5, has unit variance.  
descent algorithm i s  given by Eqn. (7), substituting Zu t fc(u,) 
for f ( \ ) .  Figure 3 shours the improvement obtainable by the use 
of a combination of the antithetic and control variate methods 
The k 
k 
(curve 2 )  over  the use of the antithetic var ia te  method only (curve 1). 
6. Conclusion 
It has bc,eii shown a t  any rate for the simple and ra ther  
contrived examples given in the previous sections,  that a con- 
stant s tep  length algorithm con\rerges considerably f a s t e r  than 
the usual stochastic approximation algorithm. 
reasonable in  general  (except maybe for pathological problems)  
if the variance of the parameter  u can be reduced fast  enough. 
Hopefuli? t h i s  can be  achieved by  using Monte-Carlo acce lera tor  
techniques. 
This i s  intuitively 
Further  examination of deterministic descent methods is 
likely to ledd to new algorithms for stochastic optimization. The 
-13- 
io’ 
10° 
Id’ 
102 
l lul l 
-3 
10 
lo4 
lo5 
I I I I I 
f (u l=u(  1 +t2, + 0.1 ( u +t 13 
E t  =O ; Var ,$ = 1.0 
CONSTANT p =0.5 
\ (antithetic) 
CONSTANT 
(anti t hetic, 
control 
p=0.5 
I 
5 10 15 20 25 
I I I 
SAMPLES k 
FIG. 3 INCREASE OF CONVERGENCE RATE BY CONTROL 
VARIATE METHOD. 
ideal  goal would be to c rea te  a range of algorithms so that most 
problems could be solved as quickly and efficiently f rom the 
computational point of view a s  their  deterministic counterparts.  
-15-  
RWCEDING PAGE BLANK NOT FILMED. 
1. 
2. 
3. 
4. 
5. 
6 .  
H. Robbins and S. Monro. "A Stochastic Approximation 
Method, I '  Ann. Math. Statistics, 22, No, 1 .  (1951). 
J. A. Blum. "Multidimensional Stochastic Approximation 
Procedures ,  I t  Ann. Math. Statistics, 25, No. 4. (1954). 
A. Dvoretzky. 
Third Berkeley Symp. Math. Statistics and Probability, I, 
(1 956). 
"On Stochastic Approximation, I '  Proceedings,  
E. G. Gladyshev. "On Stochastic Approximation, I t  Theory 
of Probability and Applications, 10, No. 2. (1965). 
W. Fe l le r .  An Introduction to  Probabili ty Theory and I ts  
Applications; Vol. 11, Wiley, New York, 1966. 
J. M. Hammersley and D.  C .  Handscomb. Monte-Carlo 
Methods; Methuen, London, 1964. 
-17- 
" 1  M.,.L ..'h.. ..... U .... -r 
.".h L.,.lO" 
111 C.,."." .d 
0 . u  ".< v 7 llYl 
c-.-,,, 
At?". ......... " . A r m , . * C n - i . l U i f E . I  .... .... LII""I-Ollh. ........... 
C-..,.."d.. ....... U r  UL..B. ......... 
Pd"t Mas". C.lil0.d. 9 x 4 ,  
h r . < C ' ,  ....... L.I,.:Dn "'F,.<-:_,<. .............. bl, "I re:.,*, 
C-bndc Y.... ..... 
C-""dI".Mllr.. 
mtlrr .. N.I.1 ........ S,."ll ..... 
101 .*.# Zeh.", 
N.. ..... *- .... 1w.1, 
sv.n..d. . 
Y.  I *.I, YL"Y.**..d.m(. ..................... 
Uric la s si fied 
October 1967 
NO001 4- 67-A- 0298 - 0006 and 
la. C O N T R A C T  O R  G R A N T  NO. 
Division of Engineering and Applied Physics 
Ha r va r d Unive r s i  ty Cambridge, Ma s sac  h u s  e t  t s 
19 6 
98. O R I G I N A T O R ' S  R E P O R T  N U M B E R ~ S )  
1 Unclassified 
2h .  C R O U P  
t .  S U P P L E M E N T A R Y  N O T E S  
R E P O R T  T I T L E  
I 2  S P O N S O R I N G  M I L I T A R Y  A C T I V I T Y  
Office of Naval Research  
A STOCHASTIC APPROXIMATION SCHEME WITH ACCELERATED 
CONVERGENCE PROPER TIES 
I .  D E S C R I P T I V E  N O T E S  (ripe of reporf and.inclusrvr d a l e s )  
Interim technical report  
i. A U  T H O R ( S )  ( F i r s f  name,  middle In i t ia l ,  l a s f  name)  
P. M. Newbold 
. R E P O R T  D A T E  78. T O T A L  N O .  O F  P A G E S  176. NO. O F  R E F S  
An examination of the f i r s t  o r d e r  steepest  descent method for  minimizing 
determinis t ic  functions leads to the specification of sufficient conditions assur ing  
the convergence with probability one of a new stochastic approximation algorithm, 
The algorithm is interpreted as  a stochastic descent method using a constant, as 
opposed to  a continually decreasing step length. 
is that the var iance of the gradient should be z e r o  as  well as a minimum at the 
solution point. 
applicable c a n  be extended to include many useful problems by employing 
acce lera tor  methods adapted f rom the field of Monte-Carlo techniques, both to  
satisfy the condition, and to  accelerate  convergence. 
Simple i l lustrative examples demonstrating the effectiveness of the new 
algorithm a r e  presented. 
The c ruc ia l  convergence condition 
The ra ther  nar row c lass  of problems to which the new algorithm is 
FORM 1473 (PAGE 1 ) I D  1 NOV 6 5  Unclassified 
Securitv Classlf icat ion 
A-31408 
S I N  01 01 - 8 0 7 - 6 8 1  1 
U nc la s si f i  e d  
- Security C1,issification 
1 
K F Y  W O R D S  
Stochastic approximation 
Stochastic gradient descent method 
Stochastic optimization 
Monte -Carlo accelerators  
L I N K  A 
R O L E  - W T  
L I N K  B 
R O L E  W T  - 
L I N K  C 
? O L E  W T  
TJnclassified 
A - 3 1 4 0 9  Security Classif icat ion 


A stochastic approximation scheme with accelerated convergence properties

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server