Lippman, Steven A

Elsevier - Publisher Connector 

JOURNAL OF MATHEMATICAL ANALYSIS AND APPLICATIONS 24, 440-445 (1968) 
On the Set of Optimal Policies in 
Discrete Dynamic Programming 
STEVEN A. LIPPMAN 
University of California, Los Angeles, California 
Submitted by Richard Bellman 
1. INTRODUCTION 
We consider a finite state, finite action space Markovian decision process. 
One problem is to choose a policy, termed p-optimal, that maximizes the 
total discounted expected income for an infinite number of time periods. 
It is well known [l] that the set of /?-optimal policies contains a stationary 
policy. We show that the set of p-optimal policies consists of all sequences 
of one period rules (mappings from the state space to the action space) such 
that if any rule in the sequence were to be used repeatedly as a stationary 
policy, it would be P-optimal. Apparently this result is intuitively obvious. 
However, when we seek a policy, termed optimal, to maximize the expected 
return per unit time, the analogous result is not true (see example 1). In 
this case we give a partial characterization of the set of optimal policies. 
2. NOTATION 
Following the notation of Blackwell [l], we make the following definitions: 
(1, 2,..., S} is the set of states, A, is the set of actions available in state s 
(1 < s < S), i(s, u) is the immediate income received from taking action a 
while in state s, and q(s’ 1 s, a) is the conditional probability that the system 
is in state s’ at time rz + 1, given that the system was in state s and that 
action a was taken at time TZ (rz = 1,2,...). 
Letting F = Xf==, A,, a policy TT is a sequence (fi)& from F. Using the 
policy VT means that action fn(s) is chosen if the system is in state s at time n. 
ForfEF,letfm=(f,f ,... )andletrr,,fm=(fI,f, ,..., fn,f,f ,... );f”is 
called a stationary policy. We associate with each f EF the S x 1 column 
vector r(f) and the S x S Markov matrix Q( f) whose sth and (s, s’) elements 
are i(s, f (s)) and Q(S’ j s, f (s)), respectively. If T = < fi)& , let 
Q&d = Q(fi) Q(fi) ..-Q(fn>- 
440 
OPTIMAL POLICIES IN DYNAMIC PROGRAMMING 441 
Then the vector of total discounted returns from policy 7r is 
where /3 (0 ,< /3 < 1) is the discount factor and So(~) = I, the S x S identity 
matrix. 
The vector of total expected returns in periods 1, 2,..., 1z from policy CT is 
given by 
n-1 
Denoting the sth component of a vector u by [u]~ , we write u 3 v if 
[uls > [zJ]$ for all s and u > v if u > v and u f v. Also, let 1 be the S-vector 
of 1’s. 
A policy rr* is called /3-optimal (0 < ,!3 < 1) or optimal if for all policies n 
we have 
VA,*) 3 ~&> (3) 
or 
(4) 
3. THE SET OF ~-OPTIMAL POLICIES 
Let A(/?) be the set of ,%optimal policies and define the nonempty set 
(see [ll, P. 7WW3) by 
F(P) = {f EF :f” E 4Bh O<jI<l% 
Throughout this section we assume that /3(0 < /3 < 1) is fixed and that 
g EW). 
LEMMA 1. If TT is a sequence from F(B), then m E A@). 
PROOF. We first show by induction that 
97 EA(/l) if rr=?rn,,gm where fi E F(P) for 1 <i<n. 
(5) 
For n = 1 we have 
G(4 = r(fA + B&(fd b(g”i 
= r(f1) + KYfi) Vdf I) 
= Ve(f3 = ~Bk”). 
442 LIPPMAN 
Suppose (5) holds for 1 < i < n, then 
n-1 
Vt?(r) = c PQi(4 Y(fi+1) + B”Qn(4 P(fn+d + BQ(fn+d ~&“)I 
i=O 
n-1 
= z. B"Qi(4 r(fi+d + PQ&) V&"> 
= ~dg"), 
the last equality follows from the induction hypothesis. 
Now let 7r be a sequence from F(p) and let E > 0 be given. Choose n suf- 
ficiently large so that pVB(gm) < l 1. We may assume without loss of 
generality that i(s, u) 3 0. Consequently, 
I v&4 - ~ek”) I G I V,B(4 - ~s(% > g”) I + I ~B(% 9 g”) - ~dg”) I < E 1 
by the definition of 11 and (5). Hence, v E A@) as E > 0 was arbitrary. Q.E.D. 
To avoid trivialities, we make the following policy space reduction: if 
t% [QnWlis = 0, h t en we shall assume that fn(s) = g(s), that is, if the 
probability under r of entering state s at time n is zero, then the action taken 
at time n in state s is irrelevant so we may assume that it is g(s). 
THEOREM 1. The set A(P) of fl-optimal policies is the set of sequences from 
w9 
PROOF. Suppose V,(f, g”) = V,(g”), then since 
VB(f? P) = r(f) + PO(f) Vdg”) 
we have 
~,@=9 = [I- BQ!(f)l-’ 6f) = ~,P), 
that is, f E F(p). 
Take rr E A@) and suppose fi EF(@, 1 < i < n, and fn+l $F(&, then 
Q&r) V,( fn+l , gm) < Qn(~) V,(g”) by the policy space reduction and the 
above so that 
n-1 
n-1 
< C PQi(4 r(fi+d + B”Qn(4 T/8(fn+l, g”> 
i=O 
But this contradicts x E A@), so fi EF(/~) for all i. 
The converse is provided by Lemma 1. Q.E.D. 
OPTIMAL POLICIES IN DYNAMIC PROGRAMMING 443 
Of course, if F(p) is a singleton, then g* is the unique #?-optimal policy. 
In particular, there is no /?-optimal nonstationary policy. 
Since [V,(fm)ls = [(I - j?Q(f))-l r(f& for each f EF, we have from 
Cramer’s rule that [V,(f”)ls is the ratio of two polynomials in j? and has at 
most S roots on [0, 1) (unless [F’,(jm)ls = 0). Let 1 F 1 be the number of 
distinct stationary policies. It now follows that F(p) is a singleton for all 
but perhaps S(2S + 1) (1 F ) - 1) points of [0, 1) if and only if 
(i) F’a(s(hm) - V,(fm) = 0 and ff h implies {p :fm E F@)} is a finite set. 
A condition under which (i) holds is (i’) y(f) # r(h) iff f h. 
The next theorem extends some work of Blackwell [1] and Veinott 131. 
THEOREM 2. Thee is a finite sequence of pairs ((ai , g&z1 such that 
0 = a, < a2 < *** < amil = 1 and gi is /3-optimal for all p E [ai, ai+J, 
i = 1, 2,. . . , m. 
PROOF. Since [ V,( fm)ls is a rational function, {/3 : j? is an isolated root of 
[V,(f”) - V6(hm)lS , 1 < s < S, f, h E F} is a finite set, enumerated 
0 = a, < a2 < *** < a,,,,, = 1. (We include 0 and 1 whether or not they 
are roots.) Choose 1 < i < m, 01 E (ai , a,+J, and g, OF, then using 
Lemma 1, the definition of ai , ai+l , and the continuity of [Va(fm)]s for 
0 < ,8 < 1 (1 < s < S, f E F), it follows that g: is p-optimal for all 
B E [ai , ai+J. Q.E.D. 
4. THE SET OF POLICIES WITH MAXIMAL RETURN PER UNIT TIME 
Let A be the set of optimal policies (see (4)), and define the set P by 
F={~EF:~~EA). 
By Brown’s Theorem 4.2 [2], max, P(n) - P(f”) is bounded uniformly 
in n for somef E F, say g, so it follows that F is nonempty. Of course, gm E A. 
THEOREM 3. If TI E A, then 
Qn(4 49 = &9, all 71, (6) 
where 
x(g) = $2 W’vyg”). 
PROOF. Since max, P(T) - Vn(gm) is bounded uniformly in n, we have 
for each fixed n 
43 2 Q&d 4cd (7) 
444 
since 
LIPPMAN 
Since rr E A, we have from (7) that for each fixed z 
x(g) = litn+sup N-l F(r) = Q,(V) ligiyp N-l VN( fn+l , &+a , . . .) 
G !A(4 &9 G 4d. 
Q.E.D. 
COROLLARY 1. Ifrr=(fi,fi,...,f,,,g”) andfiEPfor 1 Gifn, then 
nE A. 
PROOF. Applying (6) to fi , f2 ,..., fn in reverse order we have 
lig2f W1VN(7r) = g,(n) lili WIVN(gW) 
= !A(4 Ng) = Bn-l(4 Q( fn) x(g) 
= Qn-1(a) x(g) = *a* = x(g). 
Q.E.D. 
Unlike Theorem 1, however, it is not true that all sequences from P 
are in A. Nor is it true that each policy in A is a sequence from f;. This is 
shown in examples 1 and 2, respectively. 
EXAMPLE 1. A sequencefrom Pwhich is not in A. Let S = 3, Ai = {1,2}, 
i= 1,2,3, 
!I(1 I 19 1) = 41 12, 1) = q(2 I 3, 1) = q(3 I 3,2) = q(3 I2,2) 
=q(21 1,2) = 1, 
and 
i(s, a) = i’ 
1, 
if a=s=l or if a=2 and s=3 
otherwise. 
Iff=landg=21,thenx(f)=x(g)=l;yet, 
iz lvvy7r) = 01, 
where w = <f, g, f,  g, *** ). 
OPTIMAL POLICIES IN DYNAMIC PROGRAMMING 445 
EXAMPLE 2. A policy in A which is not a sequence from i? Let S = 1, 
A, = {1,2}, i(1, 1) = 1, i(1, 2) = 2, f = 1 and g = 2. Then 
but 
x(f) = 1 < 2 =x(g), 
lili N-lP(7r) = 2, 
where rr = (,fi)TC”=, is defined by 
fi = 1: ftherii; 2” for n = 1,2,..., 
, . 
In fact, fi = f  $ P for infinitely many i. 
bFEFU%NCES 
1. D. BLACKWELL. Discrete dynamic programming. Ann. Math. Stat., 33 (1962), 
719-726. 
2. B. W. BROWN. On the iterative method of dynamic programming on a finite space 
discrete time Markov process. Ann. Math. Stat. 36 (1965), 1279-1285. 
3. A. F. VEINOTT. JR. On finding optimal policies in discrete dynamic programming 
with no discounting. Ann. Math. Stat. 37 (1966), 1284-1294. 


On the set of optimal policies in discrete dynamic programming 

file:///data/core-remote/dit/data/elsevier/pdf/2c7/aHR0cDovL2FwaS5lbHNldmllci5jb20vY29udGVudC9hcnRpY2xlL3BpaS8wMDIyMjQ3eDY4OTAwNDI1.pdf

On the set of optimal policies in discrete dynamic programming

Abstract

Similar works

Full text

Available Versions

Elsevier - Publisher Connector