A problem is considered which arises in the numerical analysis of the degenerate partial differential equations of stochastic control theory. The relationship between diffusions and finite state Markov chains is considered. The role of probability theory in numerical analysis is illustrated

Kushner, H. J.

English

NASA Technical Reports Server

m? i-u: V-" "-*'• '
CONVERGENCE 0V TILS DISCRETIZATION EQUATIONS WlilCH
ARISF I?! STOCHASTIC CONTROL •• .
=^
" H. J. Kushner : "'.
Center for Dynamical Syst.ems
Divieiou of Applied Matfie-jmatlcs.:
Brown University ;
Providence, Rhode Island
Oaiv.) Sep. ,1971 18 P
»
«ro«
H72-21568
Onclas
G3/19
This research was supported in part by the Air .force Office of Scientific
Research, AF-AK)SR 693-67D, in part by the National Aeronautics and Space
Administfation, KGL 1*0-002-015, and in part by .the National Science Founda-
tion, GK ' '
https://ntrs.nasa.gov/search.jsp?R=19720013918 2020-03-11T18:56:25+00:00Z
ARJTMCT
for
' Or Tllii DISCRETIZATION EQUATIONS WHICH
ARISE IN STOCHASTIC CONTROL
by H. J. Kushuer
.
We consider a problem which ariaes in the numer.ic.-i.] analysis
of the degenerate partial differential equations which arise in stc-
cb.as1.ic control theory. The- result will also shed eoine light on the
relati one-hip between diffusions and finite state Markov chains, and
illustrates a role of probability theory in numerical analysis.
Consider the degenerate linear elliptic equations
V + k(x) = 0, x e G,
(1) V(x) -<p(*), * e &}
*•• Z a - -
J
 d>.dx i dx
* v
G is a boaiidtd op«n set, k and <p are continuous and a. . (x ) *
£ o (x)a (x), where 0 and f. satisfy a Lipschitz condition in
. IX JK ij
x. Equation (1) arises in stochastic control as a formal representation
of the cost
T
(2) C(x) = E / k(x )ds + E cp(x ), x e G,
0
for the diffusion
(3) dx = f(x)dt + ff(x)dz,
where z is a vector valued Wiener process and T * inf(t: x. i G}
V V
lilfir represent z-tiout can be dcvejopcd for^discounted, and time depen-
dent cost::, and our ccnvtrgi i.e..1 r e&ul t^ can be extended to these cases.
We con... ider the proUU-m: Discretize (l) with finite difference
interval h (or h. if it depends on the direction). Solve the finite
difference equations, to obtain a function V. (x) at the finite dif-
ference points. Then v/hat happens to V.(x) as h ~»0? The question
is important sincr (1) cloe^ not usually satirfy the classical condi-
tions which clarrical numerical analysis if based upon, and if we
i
are to have confidence in the finite difference solution, it is es-
sential to know that V,(x) -»C(x) irrespective of whether (1) has
strong derivatives or not; i.e., that (1) can legitimately serve as
an intermediary in obtaining (2), or an approximation to (2).
It is shown that, whether or not (l) has a smooth solution,
•
the solutions of finite difference equation* converge to (2) under
quite broad conditions, if the finite difference schemes are care-
fully chosen. The analysis is purely probabilistic, and uses only
probabilistic properties of (2), (3), and the solutions of the finite
difference equations.
ace of the Discretization Equations Which
Arise in Stochastic Control
H. J. Kusbr.er
Introduction. \l*.. will consider a problem which arises in the numeri-
ca] analysis of the degenerate partial differential equations v?hich arise
in stochastic control theory. The result will also shed some light on the
relationship between diffusions and finite stat* Markov chains, and illus-
trates a role of probab.1 .lily theory in numerical analysis. The discussion
will be informal. See H. J. Kushner, "Probability Limit Theorems and the
Convergence of Finite Difference Approximations to Partial Differential
Equations", J. Math. AriaJ. and Appl., 1970, for acre results.
The Problem. Consider the degenerate linear elliptic equations
(1) CV + k(x) « 0. x c G,
V(x) = f(x), x € dG
'
vhere G is a bounded open set, k and 9 are continuous and a. .(x) *
£ cr,, (x)a.. (x), where a. and f. satisfy a Lipschitz condition in x.
J K. J lC 1J X
Equttion (1) arises in stochastic control as a formal representation of
the cost
T
(?) 0(x) . t 1 k(x )<ln + K«<x ), X r. H,
* f\ *0
for the diffusion
dx « f(x)dt + o(x)dz,
where z is a vector valued Wiener process and T = inf{t: x, / G). Simi-
t Tl
lar representations cad be developed for discounted, and time dependent
costs, and our convergence results can be extended to these cases.
We consider the problem: Discretize (l) with finite difference inter-
val h (or h. if it depends or the direction). Solve the finite differ-
ence equations, to obtain a. function V.(x) at the finite difference points.
Then what happens to V,(x) as h -»0f The question is important since (1)
does not usually satisfy the classical condition* which classical numerical
analysis is based upon, .and if we are to have coafidence in the finite dif-
ference solution, it is essential to know that Vh(x) -»C(x) irrespective
of whether (1) has strong derivatives or not; i.e., that (l) can legitimately
serve as at. intermediary in obtaining (2), or an approximation to (2).
For Equation (l) let the difference interval be h (in any coordinate
direction, for convenience only) and let e. be the unit vector in the i-th
coordinate direction. Suppose that G is strictly contained in a hypercube
H with sides [-A,A], Define the set of nodes R*" in Rr by R^ «=
{(n^ , ...,nrh), n^ ranging over 0, +1, +2,...}. Define G = G 0
In order to expose- t;vj method, and not get involved with the rather
lone, finite difference equations arising when mixed second derivatives occur,
we let a.. . * 0 for i / j. There is no trouble in extending the method
ij
to the more general c a ;->•_•..
The following finj.tr> difference.1 approximations will be used.
, ,V(x «• eh) - V(x)
(3.) Vx ~i j ,i v V(x) - V(x - e.h) >
where the upper terra of (3a) is used if f^ (x) > 0, and the lower other-
wise. (This usage will be carried throughout)
V(x + e h) - 2V(x) + V(x - e.h)
(3b) V (*) -- i-
 f - i-
xi i be
The reason for the choice (Ja) will appear shortly.
If V,(x) denotes the solution to the finite difference equations, then
•r
usirg (3) for x e . Gfa, (1) yields
,
h
v
 y
x + e i h rhfi 4
x) = L — ^ -7—1 — i
*W
*«+
where
Q (x) . 2 La + Z h | f J
i i
Define V (x) ~ C | ' ( x ) f 'c-r x c R*" - G . Re-write (] ' ) as (vith the obvious
identif.icution of term-)
Vh(x) - >.', V h ( x + eih)ph(
P(x)k(x)
, V(x) = cp(>:), for x e R^ - G, .
Now the reason for the choice (.3) wi.13 become clear. Note that since
the p,(x,y) > 0 and sum to at nost unity, and can be defined for all
x, y e R,. they can be considered to be trpnsition probabilities for a Markov
chain on the grid R..
Denote the sequence of random variables of this Markov chain by l£)'
Thus P(ik+1 + he^ = Ph(x,x + eih)J etc. Define
inf{k; '
Now we proceed to investigate the behavior as h -»0. Suppose (not very res-
trictive - see example) K N < K. < «. The solution to (l") can be written
as
(I"') Vh(x)
k=0
Equation (i"r ) looks something like a Reiaarm sum approximation to (2),
and, indeed this-- is= tin idea to be pursued. We vill show that the measures
.
of a suitably interpolated U *} converge to those of x in a certain
n t
sense, and that thjf; iTTif.Vut; that V,(x) converges to C(x) for x on any
G,. This hole.'. irrer,j, ctj.vr of vhethor strong derivatives of (l) exist or
not.
•
A key ftx-l is Theorem 1, drawn froT Gikhman and Skorokhod, "introduction
to RHndoro ProCt'o^es'' , (in Russian), Izdatclctvo Kauka, Moscow, 1966.
Theorem .11 Let C[0,T] ~ fi be the set of Rr valued continuous func-
tions on the in-Leryal [0,T], Let y (t), y(t), t e [0,T] be continuous pro-
cesses with paths in the (topolo^ ical) space ft. Let \JL and M. be the
measures induced on fi by the processes y (•) and y(*), resp. Let (for
0 < t1 < t" < T)
lim Tim P ( «up | yn(t« )-yn(t")| > e > 0}
C
 '"
for any c > 0. Let the finite dimensional distributions of (yn(t)} con-
verge to those of y(t). Let F(*) be a bounded and continuous (w.p. 3.)
Borel functional on the topo logical space ft. Tben
In order to exploit Theorem A, the process (!) must be related to
a suitable continuous time process {I (t)}.
By a comparison of (l": ) and (2), we note that the "discrete time"
cost rate ii, i-h(x) times th-:- continuous time cost. In an intuitive sense,
one ftep of the disc; de p'.vcesi £ ' should take Pu.(4, } units of realK. n ic
timt?. Thu^ tho i'ojicuvinp. definition IK natural. Define the time sequence
(t } by ( .sam>rt:'ui!<'c ;u-c, | ! t (1 ;;!,:; of function are deleted for simi'licity, andK
a, ( ^ ,). •'*•, are uKfd ant; rchangcably)
*-! K- iC
v .., h
Z ^
,
Define a process I (t) by
6 (tO c ^vK K
at th<? times {t,}, and for t. < t < t,
 f by the linear irrterpolati
(t
 '
.
Thus the continuous process £ (t) is piecewift* linear and changes slope
at the random break points {t, } only.
The use of £ (t) is a natural way of relating {£ ) and x.
The main results are»
Let measures ^ and p. (on the topological space fl « C[0,T]) cor-
.,
on
respond to processes | (t) and x., resp. t e [0,T], The conditions (Cl)-
used in the sequel are quite natural for a large class of problems,
and are illustrated in the exawple.
Thiorun el. A.vsiur.g
(Cl) f4 ( 0 u^d a . . . ( f ) are uniforraJy boupled and batisfy a uniform
LJpsc-hi.ty oondi tion. (Kf c- •" I a = aj ' .)
Q(CP) Lft 6 egiia] h or h . t'or real positive K. 3e_L
(C5) Let a(0, have the form
an r X r *vhere IT ' (£) hafi^unlfor/Rly bounded tergist and 0_
Theti the finite dimensJortal distributiorCof the proceeds s (t) con verge
to thoae of the proco.s^ x, ttiid^ for 0 < t1 < t" < T, and e > 0
liin P { sup
t'-t"| <
- 5 (t")| > €} . 0.
Remark. (C?) means the following. Either we allow .. > e > 0
for some real e , in which case 5. = h , or we allow Z.a (|) = 0 and
L|f.| > c > 0, in which case 6 = h. Thus 2. cases are considered - one
case in which there is always some diffusion scBewhere, and one case in
which there is no diffusion - but where the velocity of x. is never 0.T
In the intermediate caye the ratio
.
8may be infinite, invalidating our proofs. The first case IB one of great im-
portance.
Coro Llary 1. Assimu^ ( tU)~(^)- Let F( • ) be a bounded continuous func-
tion on C[0,T] w.p.l. (relative- to M). Then (with £ * X * x )
Theorem 2 uses condition (C6): There is an h >0 so that for h < ho>
G satisfies: Let I = (a,b) be a line connecting two adjacent (along co-
ordinate directions) points (&,b) of the grid R, . If a and b are
both in G, then so is the line t connecting tfeem.
(C6) can be weakened in »any ways - but there seems little point in
•
complicating the condition here. It is certainly satisfied (for any b_)
™
for convey. G. (c6) is used to assure that the first passant1 times from
G, of both t (t) and I, are approximately the same time. I.e., if
t (t) leaves the G between the n-th and n+l-at steps of |., then
N. = n. It is used to avoid the possibility illustrated in the Figure (for
•11 small h), where if the discrete process | jumps from a to b at
•
time n, it has not actually left G, but the interpolated process 5 (t)
T Tleaves right after time n. Let N. * max(n: t < T} and M. = min(N.,N.)b n h* b
Theorem J. Assume (C.l)-(C3) and (C6). Lejt k(-) and < p ( « ) be uni-
forinly continuous and bounded on some open set containing G = G + oXJ.
Let -i denote the first (random) time that the process x. leaves G
(T = inf{ti x / 0}), rmo suppose that T fl i =»in(T,i) it: continuous
v
w.p..1. (The w.p. 1. j^ la t -UK i ) L i_s r & j at ive t o n) cni C[0,T]. Donate T.
inf{t: Ih(t) / G). Then
TOT TOt
E / k(x )ds = lira E /
X
 0 h -»0 0
Mh-l
= im E E jcibi
h- .0 C 8=0
.
ESKx™..) = lim Eq;(th(TnO) - lim E ^ C ) .
h -»0 h -»0 ^3
Theorem k. Assume the conditions of Theorem 2, and letj for some
(5) p*5 leaves G at least once by tiltie - * >0
where M is independent of x 6 G, and h > 0, for small h. Then the
V.(x) given by (!"') converge to (2 ) a£ h -»0, unifomly in x in. G;
i^ .e.,, the solutions of the finite difference equations (1*, 1") converge to
the veak solution (2) of Equation (1), as h ->0«
.
\II)
t:J !
fr
Figure I
The conditions imposed in Theorems 2-k are rather natural for a large
class of problems. In order to illustrate this, a brief check of their
validity will be made for a ?-dimensional problem. It should be clear that
the example is typical of a large clars. Although the basic problem arose
in numerical analysis, the approach taken here, as well as the conditions,
are probabilistic. Hc-nce, the checking of the conditions involves probabil-
istic calculations on the underlying processes. Let
••
dx, = 2a
p 0'
IP v<
0crf
dx2 = f2(x)dt + v dz
where v is a constant and the f satisfy (Cl). Let
We seek to solve
i * *2 in
11
/V(x) + k(x) = 0 in G
V(x) -- <|>(x) ori dG
where k( •) and <p(*) nrf continuous and bounded,
and G is the box
+ f iW ,_ . y,.
G = {x: \ x ± \ < A).
Thus (C6) holds for all h > 0.
Note that / is degenerate and G has corners; hence, classical theory
cannot be used to BO I've the convergence problem »B b -# 0.
We need only show that T fl T is continuous w.p,1. (relative to n)
on C[0,T], and that (^) holds. Let o> be a generic point of C[0,T] » H.
T 0 t is coutinuouB at any path which is not tangent to the boundary at
TOT.
Returning to the problem, refer to Fig. 2. Jt is clear that if tan-
gencies at the boundary occur only w.p. zero, then, by virtue of the con-
tinuity of x (co) w.p. 1., (T n i)(o>) will be continuous w.p. 1. This will
now be shown to be the case.
We observe that
(a) for x > 0 x nu3t increase (as ti»e increases) since dx. «
cT J.I- J.
x_dt in G. Hence, w.p.1., points on the boundary section L. (Pig. 2) are
••
12
•
_
not aec<: r:;.ible. Similar ly for L..
1
X
r
L,
*1
L. f"
I I
" r
y^— ^-Jl
V
,
*-o.
.
1 1 6
-/•
vX,
' ^
A
' r~
1L,I
*•
.
•
V
cK
^
L
\
•
/^ ^ X|
\
•
:!
•
,J
'
-
.
1
Figure 2
.
(b) Also, since x^ > 0 on L,. the path cannot be tangent on L,.
2t JL* J-
and similarly for L .
(c) Owing to the dominant effects of the diffusion on movement in the
vertical direction, the points on L and L we regular in the sense of
Dynkinj i.e.,
lim P { x c
6->0 X
x c G f or aD. ft > c > 0} = 0
(d) PX{T = T) = 0
i
'
;•
'-
(f) Siucf x i« ec'ulj.nuou.s, w.p. I., Hurt are t.,(<o) > 0, w.p.l.,
* s
>0 w.p.1. flo t!ml
distaricf (x / v, exterior of CJ) > ei,(^),
t ~ ' - f M
for all t < T(OJ) if 1(0^ < T.
(a)-(f) imply continuity w.p.l. of T 0 t.
o
Only (5») remains to be proved. Let H = to/K-ih > ^or any
define
M (0 » number of positive «teps of 0>c.
A sufficient condition for (5)
arid
k < N
M (I) = number of negative Heps of {^
 2, k < N.
- M_(S) > 3 >
For some real K, we have the bounds,
Let (u } be a Markov process on fo, +1, +2,...} with transition pro-
Ik
•*•*
bability
_
.
Define M (u) aria logon .-;.ly to M ( £ ) .
.
)-
The mean value of VL , - u Is -2Kb, and its •variance is 1 - (2kh) , Now
(u) - N(-2
•."•
•
The left term in brackets converges in distribution to the normal «ejro
an and unity variance random variable, and the right band term in the
brackets is strictly less than some K, < » for Mall h. Thus, for
•
wbich proves (5).
°-V"(lt)-^^"*"5ir<'r'
•
The proof of the convergence of the finite dimensional distributions
h
of 6 (t) to those of the diffusion xt proceed* in a series of steps.
First the representation (6) is proved, where (oO ar* orthogonal for
:
* • . • ' ' .1' ' >,- I I", f : • ' ( • ] < c .• riri; . :r^ }u ^ ' j 'vj i ' l . i o'ia1 1-o ^t riud the- fJ> 'K K
!
'n i Ug.lblc" .f--»j' c i < - :l I h.
?h
''i-' V'k ^"V
'!'!• i Lin jfterpo.'tut..iop of ( < • ) it !)pj:r\ jy5ni--i ' tr(] by tin: interpolation, of TJ (t)
or
'\
0
Thf f in j t^ dimcn;-; 'ona] d i f t r jbul ions of the interpolation of (7) ere ap-
i-o> i jnsteJ by Lho^p of the intex-polation of (as h -»0, small A),
(8) ~hyk+l
ri thi. sum., of Atc are approximately
of (B) by ,-( diffusion.
0
and finally the interpolation


Convergence of the discretization equations which arise in stochastic control

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server