Zaremba, Leszek S

Elsevier - Publisher Connector 

Optimality principles of dynamic programming in differential games 

JOURNAL OF MATHEMATICAL ANALYSIS AND APPLICATIONS 138, 43-51 (1989) 
Optimality Principles of 
Dynamic Programming in Differential Games 
LESZEK S. ZAREMBA 
Associate Editor, Mathematical Reviews, 
416 Fourth Street, P.O. Box 8604, Ann Arbor. Michigan 48107 
Submitted by Leonard D. Berkovitz 
Received May 7, 1987 
1. INTRODUCTION 
A two-person zero-sum differential game is a dynamical system whose 
dynamics are described in the general case by a system of differential 
inclusions 
i.(t) E F,(t, x(t)), x(r*) = x*, XER”, (1.1) 
Jxt) E F,(c Y(f))7 Y(t*) = Y,, y E Rk, (1.2) 
with phase constraints of the form 
x(t) E N,(f), Y(l) E N,(f)? t> t,, (1.3) 
where N,( .), N,( .) are set-valued maps with closed graphs. By a solution 
of ( 1.1) (resp. ( 1.2)) we mean an absolutely continuous function x( ) (resp. 
y( . )) satisfying ( 1.1) (resp. ( 1.2)) almost everywhere. The game stops when, 
for the first time t > t, (henceforth abbreviated by T, = T, {t*, x,, y,; 
x( ), y( )}) the triplet (t, x(t), y(t)) hits a prescribed terminal set 
Mc R”+kf’. At time T, player I receives from player II the payoff 
= AT,, x(T,wM), Y(T,)) + lTM NC 4th y(f)) df. (1.4) 
‘* 
Concerning the information available to both players during the course of 
the game, we assume they can employ any lower strategy; this concept will 
be introduced in this paper (Definition 2.1) and is more general than the 
notion of a lower n-strategy introduced in [4, p. 4001. Under Axioms 
(Al)-(A6) we prove the optimality principle of dynamic programming for 
43 
0022-241X/89 53.00 
Copyright $: 1989 by Academic Press, Inc 
411 rights of reproductmn ,” any lorm rcaerved. 
44 LESZEK S. ZAREMBA 
player I and player II in differential games. This principle was originally 
formulated and proved by Bellman for optimal control problems under the 
assumption that the optimal cost function was twice differentiable [ 11. 
2. ASSUMPTIONS AND STRATEGIES 
Throughout the paper, t,, x*, y, are specified values with (t,, x.+, y,) 
being the initial point. We shall consider admissible points (t,, x*, y,) 
only, i.e., those for which x* EN,, y, eN,(t,). When N,(t) = R”, 
Nz(t)=Rk, tat.,., we actually deal with a differential game without phase 
constraints. We make the following assumptions on the game: 
(Al) the graphs of set-valued maps N,( .), N,( .), and the terminal set 
M are closed in R”+‘, Rk+‘, Rn+k+l, respectively; 
(A2) the functions g( .) and h( .) occurring in (1.4) are continuous; 
(A3) for any (t, X)E N,, the set F,(t, x) is convex; we assume the 
same for F, ; 
(A4) (i) for each x E N,(t), the set F,(t, x) is compact; (ii) for each x, 
F,(t, x) is measurable in t; (iii) there exists x* E R” and an integrable 
function a(t) > 0 such that, for t 2 t,, the following condition with H 
denoting the Hausdorff metric holds: H[F,(t, x*), {0)] <a(t); (iv) F,(t, x) 
satisfies a Lipschitz condition in x, i.e., H[F,(t, x,), F,(t, xl)] < 
b(t) . llxZ - x, 11, where b(t) is an integrable function; the same is true for 
F,(t, Y). 
Observe that (A4) will hold true if: (i) F,(t, x)=fi(t, x, U), FJt, y)= 
f2(t, x, V); (ii) the functionsf,( .),f2( .) are continuous and Lipschitz in the 
spatial variable; (iii) U and V are compact subsets of some Euclidean 
spaces. Concerning regularity of lower and upper values of the game to be 
defined later, we assume less than continuity. Namely, we make the 
following two assumptions: 
(A5) the lower value W(t, x, y) is lower semi-continuous in y; 
(A6) the upper value W(t, x, y) is upper semi-continuous in x. 
There are several papers giving sufticient conditions for continuity of 
lower and upper values of a differential game; see, for example, [2, 31. 
These papers, however, deal with lower and upper values defined in a 
slightly different manner than E(t, x, y), lT(t, x, y). Clearly, in several 
situations those concepts coincide. In our setting, which is general enough, 
the functions @‘(t, x, y), @(t, x, y) may not be continuous because closed- 
ness is the only assumption imposed on the constraint sets N, = {(t, x): 
xeNl(t)} and N, = ((t,x): YEN,(~)}. However, it was shown in [6, 
OPTIMALITY PRINCIPLES IN GAMES 45 
Propositions 2.1, 2.21 that for generalized pursuit-evasion differential 
games and games of fixed time duration, the lower value l/((t, x, v), defined 
in a very similar manner as r(t, x, y), is lower semi-continuous in y. 
It was also shown in [6, pp. 16&161] that the upper semi-continuity in 
x and lower semi-continuity in y imply that, for each pair of trajectories 
x( .), y( . ), the lower value _V((t, x(t), y(t)) is continuous from the left in 1, 
although it may not be continuous from the right [6, p. 1621. The reason 
for this was that N,( .), N2( .) were not continuous in the Hausdorff metric 
sense. If there were, this author would expect the continuity of 
_V(r, x(t), y(t)) and p(t, x(t), y(t)) along each pair of admissible trajec- 
tories. The problem of continuity of p(t, x, y) and W( t, x, y) is less 
obvious and will not be treated here. 
For each point x0 E N,(t,) (yO E NJf,,)), t, 2 t,, denote by X(t,, x0) 
(resp. Y(t,, yO)) the space of all admissible trajectories of player I (II) 
defined on [to, co) and emanating from x,(y,). It is known [S] that, under 
Axiom (A4), the system of differential inclusions has a solution through 
each initial point (to, x0) (resp. (to, yO)) on [to, co). 
By a finite partition n = (ti) of a ray [a, cc ) we mean a partition, such 
that, for each [In, i], finitely many partition points t, belong to [a, i]. 
DEFINITION 2.1. By a lower strategy c( = a(t,, .x0, yO) for player I we 
shall understand a sequence of maps 
cci(fi, Y( 1) + Ct;+ 1, x(s)), t, <s<t,+,, i30, 
such that the resulting partition is finite and ~1, is non-anticipatory. 
It follows from this definition that at time t, player I chooses the first 
partition point t, and the trajectory he will be moving along on the 
segment [to, t,]. At time t,, based on the knowledge of X(S), y(s), 
t, 6s d t i, he chooses the consecutive partition point t, and the trajectory 
he will be moving along with on [t,, tz], and so on. 
In a similar way, the notion of a lower strategy b for player II is 
understood. It is obvious that each lower n-strategy [4, p. 4003 is a 
particular case of a lower strategy introduced here because, in the case of 
lower n-strategy, the partitions do not depend on the opponent’s trajectory. 
Denote the space of all lower strategies for player I (II) by 
A(to, x07 Yo) (B(to, x0, Yo)). 
The notions of lower and upper values of the game are defined in a 
standard way, namely 
Yro, x0, yo)= w=suP{p+(~): MEA(43, -%, yo,}, 
W(t,,x,, yo)= ~=inf{P~(B):BEB(f,,x,,~,)}, 
(2.1) 
(2.2) 
409.‘1381-4 
46 LESZEK S. ZAREMBA 
where P+ (a)(P-(j?)) is the least (greatest) value of the payoff P that is 
ensured by a strategy a(p), that is to say, 
P+(a)=inf{PCfo,xo, h;ab+))~ ~(91: Y(+ Y(b Ye)), (2.3) 
p- (8) = suP{W,, x0, Y~;x(~ IW))~: wm,, 4. (2.4) 
The game is said to have a value W if W= W= m Clearly, each lower 
strategy a (the same refers to any lower strategy p) is a non-anticipating 
operator from Y(rO, y,) into X(t,, x,), which means that if v,(t) = y2(t), 
t,Qt<i then tx[yl(.)](t)=a[yz(.)](t), t,<tdi. The spaces of such 
defined strategies (non-anticipating operators) will be denoted by 
Al(to, x0, vo) and Bl(to, x0, yd, respectively. Thus A(t,, x0, y,) = 
A,(&,, x0, yo) and B(to, x0, yo)= B,(b, x0, yo). Awing as in 14, p, 4001, 
one can show that a unique outcome results from two given strategies 
a E A ,(to, x0, vo), BE 4to, x0, vo) as well as from a E A(ro, x0, yo), 
/? E Bl(to, x0, yo), i.e., a pair (x*( .), y*( -)) of trajectories with the property 
ac.Y*c ))=x*(.), P(x*(.))= Y*(.). (2.5) 
Therefore, we get the inequalities 
W(to, x0, Yo) G Y= sup P+(u), 
IEAI(IOJOO,YO) 
Remark 2.1. Each trajectory X( .) E X(to, x0) (respectively jj( .) E 
Y(r,, yo)) may be treated as a trivial strategy Cr E A(t,, x0, y,) 
(BE B(to, x0, Yo)). 
To see this, set &(y( .))= X( .), y( .)E Y(t,, yo), to <t< cx3 (similarly 
for /7). 
3. BASIC RESULTS 
The following set-valued map W( .): [t*, 00) plays an important role in 
our considerations. 
Set (x, y, r)E W(t), t, <t < 00, if and only if x E N,(t), y E N2(f), 
and @(t, x, y)+rd W(t,, x*, y,). Thus, if x~N~(t), yeNI( and 
0, y, r)$ W(t) then, by (Ml, W&x, y’)+r> L+lt*,x*, Y*)+E for a 
certain positive number E and y’ belonging to some neighbourhood of y. 
Clearly, (x,, Y,, 0) E Wt,). 
OPTIMALITY PRINCIPLES IN GAMES 47 
LEMMA 3.1. Let assumptions (A3)-(A5) hold. Then, for every (x, y, r) E 
W(t), every 6 > 0, and every strategy a E A(t, x, y) there exists an admissible 
trajectory jj( .) E Y( t, y) such that either (t+S, a(y(.))(t+S), j$t+S))EM 
forsomeO<S<6, orelse(a(y(.))(r+6),y(t+6), r,(t+d))E W(taG), i.e., 
W(t+6,a(y(.))(t+6),y(t+6))+r,(t+6)6~(t,,,~,,y,), (3.1) 
&et-c 
r,(t+s)= j’+  ^ h(s, atA.))( Y(s)) A++. 
I 
ProoJ Let us assume that, for each y( . ) E Y( t, y) and 0 < 6~ 6, we 
have (t + 5, a(y( .))(t +6), Y(t +a))$ M. We will prove our result by 
assuming that the assertion is false and arriving at a contradiction. Notice 
that it is sufficient to consider a trivial strategy r E A(t, x. y); by 
Remark 2.1, a may be identified with a trajectory x( . ) E X( t, x). We thus 
get a point (x, y, r) E W(t), a trajectory X( .) E X(t, x), and a number 6 P- 0 
with the property that 
.f(r t S), yjt + S), r(t + S)) 4 W(r c 61, Jf.jE Y(f, i’h 
that is to say, V(t+S, .f(t+S), y(t+s))+r(t+h)> W(t,, xt,, y*)+c:,,, 
for some E,.( ,, where 
r(t + 6) = r + j”” h(s, X(s), y(s)) ds. 
I 
(3.2 
In addition, (t+S, Z(t+&, y(r+S)$M for each ye Y(t, y) and 
0 G 6~ 6. Since w(t, x, y) is I.s.c. in y, there must exist for each 
.F( ) E Y(t, y) a neighbourhood E,, ) of ~(r + 6) such that W(t + 6, X( I + 6), 
y(r t b))+r(r+6)> W(t,, x,, y,)+ 1 ‘E,, ) for ally(.) satisfyingy(r + 6)~ 
E,,,. The set K= (v: y = ir + S), y( _) E Y(t, ,v]] k compact because the 
space of all solutions of (1.2) considered on the segment [I* t + a] is 
compact in the Banach space of continuous mappings equipped with 
the sup norm [4]. Therefore, one can find finitely many open sets, say 
E yIC.,, . . . . E,,,(.,, which cover K. In this way, we have shown that, for each 
Y(-)E Y(t, y), we have F’(t+& Z(t+6), y(t+6))3 W(t*,x*, y*)+E, E= 
min{E,,, + . . . . E,,(.) }. By the definition of the lower value, there exist lower 
strategies a,(, + 6) EA(t+6, Z(t+S), y(t+S)) with 
p+(a.vcr+6J )+r(t+b)> ry(t*,x*, y*)+&E. (3.3) 
Now, it is easy to define a lower strategy aI, E A(t, x, y) satisfying 
P+(n,)+r3 Gy(r*,x*, V*)+t.F (3.4) 
48 LESZEKS. ZAREMBA 
which yields W(t, x, y) + r > F(t,, x.,., y,)+ $E, a contradiction to the 
assumption (x, y, r)E W(t). To this end, for each y( .) E Y(t, y), denote by 
y”( .) the restriction of y( .) to [t + 6, co) and set 
It is obvious that (3.4) holds. i 
PROPOSITION 3.1. Zf (x, y,, r,) E W(t), n = 1, 2, . . . . with t, < t < 00, and 
the sequence (y,, r,) converges to (y, r), then (x, y, r) E W(t). 
Proof: Since y, E N*(t), n = 1, 2, . . . . y must also belong to N,(t). 
Therefore, assuming that (x, y, r)$ W(t), we obtain V(r, x, v) + r > 
w(t,, x*, u,) + E, for some E > 0. Making use of the lower semicontinuity 
of IJ’(r, x, y) with respect to y, we get lJ’(t, x, y’) + r > kV(t*, x,, y,) + 4 E 
for y’ belonging to some neighbourhood E of y, which contradicts the 
assumption (x, y,, rn)E W(t) because the latter is equivalent to 
!Nt, x, y,) + rn G Wt,, x*, v,). I 
THEOREM 3.1 (the optimality principle of dynamic programming for 
player I in a differential game). Zf (t*, x*, y,) +! M, then there exists a 
number S such that, for each 0 < 6 6 6, we have 
sup min 
%EA(l*,x*,y*) l’(.)E Y(‘*.Y*) i 
-w(t* + 4 4.F(‘)Nt* + 61, j(t* + a) 
+I 
r,+S 
h(s, cc(j(.))(s), Y(s)) ds = kJ’(t,, x*, Y,). 
‘r 1 (3.5) 
Proof: Since the terminal set M is closed and X(t*, x*) as well as 
Y(r,, y,) is a set of equibounded functions on each finite interval 
[ 5, Corollary 2.11, one can find a number S> 0 such that, for all t, < t < 
t, +& X(.)EX(t*, x,1, Y(.)E Y(t,, Y,), we have (t, 4th y(t))$M. BY 
invoking Lemma 3.1 with (t, x, y) = (t,, x.+, y.,.), r = 0, we obtain that (cf. 
(3.1)) 
sup min 
aEA(r*,x*,y.)y(.)EY((*,y*) 
-w(t* + 4 4.F(.))(t* + a j(t* + 6)) 
+r,(t, + 6) G IHf,, x*, Y*). (3.6) 
Let* x,(t,, x,1, Y,(f,, y,) stand for the restrictions of X(t,, x,), 
Y(t*, y,) to Ct.,., t, + E]. Denote by x,( -), y,( -) generic elements of 
X,(2,, x,), Y,(t,, y,), respectively, and by v”( -) the restriction of y( .)E 
Y(r,, y,) to [It, +a, co). Thus y( .) can be written as the pair of functions 
OPTIMALITY PRINCIPLES IN GAMES 49 
(y,(.), ye(.)). Notice that each strategy c(EA(~.+, zc*, v,) and each E>O 
uniquely determine the strategy CI,: YE(f*, y,) + XE(t*, x*) from the 
equation or,(y( .))(t) = cr(y( . ))(t), t, < t < t, + E. They also determine the 
family of strategies a;,., EA(t* +E, a(j(.))(t, +c), j(t, +E)) according to 
the formulae 
for y( . ) satisfying y,( .) = j,( .). Observe that x;,. , usually means something 
different than x$, .) even if j(t, + F) = j( t, + E) and, in addition. 
cc(j(.))(f, + E) = ~(j( .))(t* + E). To prove (3.5), observe that 
by(t,, .x*, y,) = sup sup mm min 
.v, ( ) I”( ) 
z&*,x*, y,; (Lx,, q.&g+ .v”(.)), Y(.)II~ 
where y( . ) ranges over Y( t, , y, ) and j,( . ) (resp. j”( . )) ranges over 
YE(t*, y,) (resp. Y(t* +E, j(t, +E)). This means that the strategies cl;,., 
range over the spaces A( t, + E, a,(~,( .))( t, + E), y,( t, + 6)). The number 
in the right hand side of the last equality (henceforth denoted by C) is the 
best result which can be obtained by player I assuming that he announces 
his choice of a strategy c( at the beginning of the game (at time t*). 
Therefore C Q D, where 
D=supminsup~~.~{PCt,,x,, Y,; (G~;.,.,)(Y,(~~ YE(.)), Yt.11) (3.7) 
1‘ .C,( I @II 1 
is the best result which can be obtained by player I under the following 
information pattern: at t, player I announces c1,, i.e., the “first part” of his 
strategy c1= (a,, cl;,.,), where y(.) ranges over Y(t,, y,) and next (between 
t, and t, + E) player II announces the path j( .) he will be moving along. 
At I, + E player I, knowing j( . ), announces a&. ), i.e., the “second part” of 
his strategy. The equality C = D means that player I will gain nothing if 
player II announces his choice j,( .) at any time between t, and t, + E 
including t, -t E. In (3.7) the sup over all a;,( .) may be replaced by the sup 
over all a-;(. ) because before choosing a;(.) player I knows j,( . ), the choice 
of his opponent on [t *, t, + E]. Note that, for fixed tl,, and j,( .), we have 
supminPCt,,x,, Y,; (4,$,.,MY(~)), Ut.11 vr’(. 1 6, 1 
I* +e 
= 
s MS, ~,M.))(s)> j(s)) ds ‘* 
+ Wt, + E, a,(.F(.))(t, +&h j,(t, + ~1) 
50 LESZEK S. ZAREMBA 
because the sup over a;(., and the min over y”( .) do not refer to the first 
integral in the right hand side of the last equality. Taking infimum over 
j,( .) and supremum over a,, one sees that the number in the left hand side 
of (3.5) equals D. Thus (3.5) is equivalent to the equality C= D. The proof 
is thus completed since C < D and (3.6) states D < C. 1 
Since the set of all outcomes resulting from a given strategy a E 
A(t,, x*, y*) and all y(-)E Y(t*, y.J is the same as the set of all outcomes 
resulting from u and all strategies p E B(t,, x*, y,) (cf. Remark 2.1), we 
arrive at 
sup min 
LxEA(l*,X*,Y*) fiEB(‘*J*,Y*) 
wt* + 4 4B)(t* + a B(a)(t* + 6)) 
where W, = lj’(t,, x*, y, ) and a@), b(a) stand for x*( .), y*( .), resp. with 
[x*( .), y*( .)] being the unique outcome resulting from (a, p). According 
to the notation employed in the proof of Theorem 3.1, it would be more 
natural to replace a and B in the last equality by a, and fl,, respectively. 
Using Theorem 3.1 once again, one can easily prove the equality 
+i 
t*+q+e2 
h(s, dP)(s), B(a)(s) ds= !YCt,, x*3 Y,), 
‘* 
with an appropriate interpretation of a = (a,,, cl,*, aEltE2), p= (fi,,, p,,, 
/?“““‘). Thus, player I will gain nothing if player II announces j?,, between 
t, and t, + E, including t, +si and p,, between t, +si and t, +si +Q, 
including t, + cl + E*. We can, clearly, generalize this remark for any 
natural number n 2 3. 
Arguing as in Theorem 3.1 with (A5) replaced by (A6) one can prove 
Theorem 3.2. 
THEOREM 3.2 (the optimality principle of dynamic programming for 
player II in a differential game). If (t,, x*, y,) $ M, then there exists a 
number 8 such that, for each 0 < 6 < 5, 
inf max 
BEB(l*J*.Y*) .f(.)EX(I*,.X*) 
(t* +4 $t, +Q, P(-fi(.))(t, +@I 
+J 
I, + 6 
MS, X(s), W(.))(s)) ds= PO,, xz+c, Y ). 
‘* 
Remarks similar to those following Theorem 3.1 apply here. 
OPTIMALITY PRINCIPLES IN GAMES 51 
ACKNOWLEDGMENTS 
I thank Professor Felix R. Albrecht for his valuable comments during the writing of this 
paper. I also thank Professor Leonard D. Berkovitz for his improvements. 
REFERENCES 
I. R. BELLMAN, “Dynamic Programming,” Princeton Univ. Press, Princeton, NJ, 1957. 
2. A. FRIEDMAN, “Differential Games,” Wiley, New York, 1971. 
3. L. D. BERKOVITZ, Differential games of generalized pursuit and evasion, SIAM J. Conrrol 
24 (1986), 361-376. 
4. L. S. ZAREMBA, Existence of value in a differential game, Systems Control LR[r. 1 (1982). 
399403. 
5. L. S. ZAREMBA, Existence of value in generalized pursuit-evasion games, SIAM J. Control 
22 (1984), 894-904. 
6. L. S. ZAREMBA, Remarks on differential games of survival, Control Theory Adu. Tech. 1 
(1985), 155-163. 


file:///data/core-remote/dit/data/elsevier/pdf/45a/aHR0cDovL2FwaS5lbHNldmllci5jb20vY29udGVudC9hcnRpY2xlL3BpaS8wMDIyMjQ3eDg5OTAzMTd4.pdf

Optimality principles of dynamic programming in differential games

Abstract

Similar works

Full text

Available Versions

Elsevier - Publisher Connector

Elsevier - Publisher Connector