Hübner, Gerhard

Czech digital mathematics library

Estimation and adaptive control of span-contracting Markov decision processes

Institute of Mathematics AS CR, v. v. i.

KybernetikaGerhard HübnerEstimation and adaptive control of span-contracting Markov decision processesKybernetika, Vol. 27 (1991), No. 1, 66--71Persistent URL: http://dml.cz/dmlcz/125653Terms of use:© Institute of Information Theory and Automation AS CR, 1991Institute of Mathematics of the Academy of Sciences of the Czech Republic provides access to digitizeddocuments strictly for personal use. Each copy of any part of this document must contain theseTerms of use.This paper has been digitized, optimized for electronic delivery and stamped withdigital signature within the project DML-CZ: The Czech Digital Mathematics Libraryhttp://project.dml.czK Y B E R N E T I K A - V O L U M E 27 (1991), NUMBER 1 ESTIMATION AND ADAPTIVE CONTROL OF SPAN-CONTRACTING MARKOV DECISION PROCESSES* GERHARD HUBNER For undiscounted Markov decision processes with unknown parameters a policy with maximal average (expected) reward may be obtained adaptively if these parameters are estimated with increasing precision and simultaneously the relative value function for the true parameters is approximated by using the presently best knowledge. The paper presents successive approxima-tion methods in case of multi-step span-contraction. This assumption weakens one-step span-contraction (corresponding to one-step scrambling) used in earlier papers. 1. I N T R O D U C T I O N For many types of sequential decision processes there are some parameters (repre-sented by 9, say) which are known only approximately. So during the process new information on these parameters is gained. Therefore usually at each stage of the process decisions are chosen optimally with respect to the presently best knowledge of 9. This procedure is known as the method of estimation and control (cf. [ 11 , 12, 14, 15]). But as long as .9 is known only roughly it may be inefficient to calculate optimal decisions on this basis. Thus there are some approximating methods, especi-ally the so called non-stationary value iteration (cf. [1 , 2, 4, 7, 13]). Both methods and others too may be described in the following unified way (see [10]): Use any sequence un approximating the true value function vva and apply to this sequence at each stage a one-step optimal reward operator U9". Then the maximizing actions form an optimal policy for the true parameter 9. (Of course, the estimated para-meters 9n must converge to the true 9.) But unfortunately this result was derived under a very strong convergence condition, i.e. the one-step scrambling condition which is not valid for many real applications, * Presented at the "Kolloquium uber Mathematische Statistik im Rahmen der Wissenschaft-lichen Kolloquien der U.aversitat Hamburg und der Karls-Universitat Prag", Hamburg. June 1989. 66 cf. [10], Rem. 2.4. We present here the proof of the same result under the condition that for each parameter # the optimal reward operator U9 is v-step span-contracting (for some v). Even this condition may be weakened to hold only for applying U3 tt> the true value function ws and any M„out of the tail of the approximating sequence (see the Remark at the end of the paper). On the other hand to insure the v-step span contraction it is in general insufficient to have a v-step scrambling condition. But the stronger "scrambling type condition" introduced in [5] will do (see also [3], p. 55 and p. 128). In the following we summarize the model, the assumptions, and the main result of [10] together with the new conditions and then present the pertinent new proof. 2. NOTATIONS AND ASSUMPTIONS The model consists of a countable state space I, feasible (non-empty) decision sets A(i), a (topological) parameter space 0, the transition probabilities pfj(a) and the one-stage (real) rewards r9(a). The aim is to obtain policies with maximum average reward for the true parameter S. For shortness we introduce the set K : = {(i, a), i el, a e A(i)} of feasible state-action pairs, the (feasible) state-action histories up to stage n hn := (i0, a0, it, ... ..., a„- l 5 i„) e H„ and the set of (deterministic) policies H := {n = (f0,fu . . .),/„: Hn --> [jA(i),fn(h„) e A(in), neN0}. For bounded functions u: I -» R we use U9 u(i, a) : = r?(a) + £ p%(a) u(j), (i, a) e K, jel and U9 u(i) : = sup 1/ u(i, a), i el . aeA(i) g: I —> \JA(i) with g(i) e A(i) is an s-maximizer of L3u if J^u(i, g(i))^Va u(i) — s, iel. Finally we write (as usual) for u: B —» U sup u := sup u(x), infu := infM(x), xeB xeB sp u := sup M — inf M and [lull := sup |M(X)|. Note that sp u ^ 2||M|| and, if u(x) = 0 for some xeB, |[M| ^ sp u. xeB We shall use the following assumptions (B) (Boundedness): For all S e 0 the reward r* is bounded on K. (C) (Continuity): For &' -> & Ar(S', S) := sup \rf(a) - r%a)\ -> 0 (i,a)eK Ap(S', S) := sup X |ps;(a) - pl-(a)\ -> 0 . (ST) (Scrambling type condition): For all & e 0 there is a v = vs e N and a Q9 > 0 such that XminKW ,^^ ) ]^^ 67 for all i + /(_!) and all v-stage policies 7tv = (f0, . . . , L _ t ) , c\, = (g0,..., gv-i) where Pa(nv), p^i(vv) a r e the corresponding v-step transition probabilities. Related to (ST) are the conditions (MS) and (S): (MS) (Multi-step scrambling condition): (ST) holds only for 7rv = crv = (f,f, ...,.,£) for a l l / . (S) (Scrambling condition): (MS) (or equivalently (ST)) holds with v& = 1 for all Se0. Further assumptions related to (ST) (see Lemma I bslow) are (MSC) and (SC): (MSC) (Multi-step span contraction): There is a v = v8 and Q* > 0 with sp [(\jy u - (Us)v v] ^ (1 - Qa) sp (u - v) for any bounded u: I -+ U, v: I -> U . (SC) (Span contraction): (MSC) holds for v® = 1 for all Se0. Since we do not discuss estimation methods in this paper we use (E) (Existence of a strongly consistent sequence of estimates): There is a sequence (£0, 3i, ...) with Sn: Hn ~> 0 and Sn(hn) -* $ PJ-a.s. (n --> oo) for all 5 e 0, %eU where P£ is the canonical probability measure induced by % (and some starting condition) and hn is the random n-stage history. For examples of sequences (Sn) see e.g. [2] and [11] — [14]. In [10] (and other papers) assumption (ST) is replaced by (S). For relations of the assumptions (B), (C), and (S) to assumptions used elsewhere see [10], Remark 2.3. The assumptions (ST), (MS), (S), (MSC) and (SC) themselves are related as follows. Lemma 1 (cf. [5], Th. 5, and [9], Th. 2). (a) Trivially (S) => (ST) => (MS) and (SC) => (MSC). (b) (ST) => (MSC), especially (S) -=> (SC). (c) If£ a is replaced by 0 conditions (SC) and (MSC) hold without further assump-tions. The optimal average gain /c3 and the relative value function (w^(i), i e I) for a fixed (known) & are obtained as a solution of the optimality equation (Lemma 2) or as a limit of a successive approximation (Lemma 3). Especially w9, will play a central role in the main theorem. Lemma 2 (cf. [6], Th. 2.1). From (B) and (MS) or (MSC) follows: (UBS) For fixed Se 0 and i0 el there is a unique bounded solution (wa, fe*), \v3: I ^U,k*eU, of U V 9 = W* + k* , w*(i0) = 0 and for this solution ka is the-maximal gain (for 9) independent of the starting distribution. Lemma 3 (cf. [6], Sec. 3, [3], Sec. 5.4). Assume (B) and (MS) or (MSC). For any bounded u0: I —> U define vt:=(\J*)"vlneN, Bj(|) : = fti) - vl(i0) , iel, neN0. 68 Then v!n is bounded, v* --> w\ and U* ^( / 0) -> /ca (n -> oo). If (MSC) holds then in addition (with [•] denoting the integer part of a real number) sp (»; - w*) = (1 - ^ )L " / v 9 j sp (eg - wa) (» e N) . To obtain an optimal policy the unknown function wa has to be approximated by a sequence of functions w„ depending in the estimates 50 , 9 l s ...,#„ thus far obtained. In the classical case of "estimation and control" (un) will be the sequence (w**) (see Th. 2(a)). Definition. A sequence of functions («;'" (i), i eI, &{n) := (S0, ..., Sn) e 0"^{) is called an admissible approximation for (w9, & e 0) if sp (ufn) — w3) -> 0 for any sequence (#„, n e N0) with Sn —> 3 for n -> GO. Note that sp (w„ — wd) -> 0 if and only if ||vv„ — ws|[ —> 0 with w„ := w„ — w„(/0). Now we are ready to formulate the main result: Theorem 1 (cf. [10], Th. 3.1). Assume (B), (C), (E), (UBS) and (1) (ufn), n e N0) is an admissible approximation for (w*, # e 0 ) , (2) (e„, n G N0) is a sequence of errors e„ -> 0 (n —> oo), (3) rf00 is an e„-maximizer of L^ttf0 , 3("} 6 6>"+1, n e N0. Then the policy n := (gff n (in), n e N0) is average reward optimal for the true par-ameter 9. Note that in Theorem 1 the assumption (UBS) can be replaced by (MS) or (MSC), cf. Lemma. 2. Theorem 2 (cf. [10], Th. 3.2). Assume (B), (C), (MSC). Then (uf"\ n e N0) is an admissible approximation for (w9, $ e 9) in the following cases: (a) ttfn):= w \ # B ) e < 9 B + 1 , n e N 0 . (b) uf0) := «0 (tt0 bounded), uT := U*-1 M ^ V " , $ « s 0 " + 1 , n e N . Case (b) yields the "non-stationary value iteration" introduced by Federgruen, Schweitzer [4] and Baranov [2] (see also [1, 7, 13]). The right hand term of (b) is already calculated in the preceding step of the algorithm when gn2\J) is determined according to (3) of Theorem 1. So at each step the optimal reward operator Ud" has to be applied only once. This corresponds to the usual "value iteration". For modifica-tions of (b) see [10], Remarks 3.4 to 3.6. 3. PROOF OF THEOREM 2 The proof of case (a) of Theorem 2 differs only inessentially from that of [10], Theorem 3.2(a). So only case (b) is proved here. At first we need two lemmas. Lemma 4 (Disturbed contraction). For any non-negative sequences (an) and (bn) with b„ -> 0 and an + y <, can + bn for some v _• 1, 0 <. c < 1 follows an -> 0 (n -> oo). 69 Proof. By induction an + kv „ ckan + _r cl 1/3„+(fc_()v _ cfca„ + 1/(1 - c) sup bm. i = 1 m g « Setting n large enough for the second term to be small and then increasing k we obtain an+kv -> 0 (k -> oo) for all n and hence a„ -> 0. Q Lemma 5 (cf. [8], Th. 6.8, [16], Th. 6.1(b)). Using the notions of Assumption (C) we have ||U*'t> - Uh\ „ Ar(/9', S) + _• A/?(,9', 5) sp i; and hence sp (U3'y - UBv) _ _ 2 Ar(&, ,9) + A_>(S', 3) sp v. Proof of Theorem 2(b). We shall use the assumptions (B), (C), (MSC). We have to show for a fixed sequence (#„) with <9„ -> $ an := sp (ufn) - w*) -> 0 (n -> oo) . By Lemma 4 it is sufficient to show (for n _ n0) (1) a„ + v _ ca„ + b„ with 0 _ c < 1 and fe„ -> 0 . Using M„ : - wf °, U„ : = U9", U := Us, and sp (Uws - w3) = 0 (cp. Lemma 2) we obtain an + v = sp(a„ + v - w3) _ _ sp (U„ + V_x ... U„u„ - UVM„) + sp (Wun - U V ) = : (I) + (II) From (MSC) we have ( l l ) _ ( l - / ) s p ( M „ - w a ) = ( l - a 3 ) a „ . By splitting up we obtain (I) _ V S P ( UV - ' - 1 U „ + / M „ + ! - U - ' -1 Uun+l) 1 = 0 and hence by Lemma 1(c) and Lemma 5 (I) _VV_Sp(U„ + iM„ + i - UM„ + /) ^YJ[2Ar(Sn + l,S) + Ap(Sn + l,S)spun + l]. 1 = 0 1=0 Therefore with sp M„ + Z _ sp (un + l — ws) + sp ws (ws is bounded) v - 1 an + v _ vb'n + c'„ X an+l + (1 - Q9) an 1 = 0 where c„ := max (Ap(Sn + l, &), 0 _ / _ n - 1) -> 0 /3„ : = 2 max (Ar(#„+/, 5), 0 _ / _ n - 1) + cn sp wd -> 0 (n -> oo) . Analogously with (MSC) replaced by Lemma 1(c) for term (II) we have an + i _ lb'„ + c'n J_ a„+m + a„, 1 __'/ __ v — 1 -m = 0 Straightforward (recursive) calculations yield «,I + v _ « I W + l)m + (c£ + l)v a„ - c » _ m = 0 / 70 (Iteration of the simpler one-step inequality an+1 f» K + (c'n + 1) an results in the same formula without the essential term -Q9an, cp. the proof of Lemma 4.) Now, if n is large enough we have (c'n + \Y - Q* £ c < 1 (for some c) and there-fore (1) holds. • Remark. As is to be seen from the preceding proof the condition (MSC) may be slightly weakened: The contraction inequality is needed only for v = w*. In the same way the assumption (ST) is needed only for ffy — (/*, .. . ,/*) — if an optimal sta-tionary policy (/ s 9/*, ...) exists for each 5. (Received December 8, 1989.) R E F E R E N C E S [1] R. S. Acosta-Abreu and O. Hernandez-Lerma: Iterative adaptive control of denumerable state average-cost Markov systems. Control Cybernet. 14 (1985), 313 — 322. [2] V. V. Baranov: Recursive algorithms of adaptive control in stochastic systems. Cybernetics 17 (1981), 815 -824 . [3] A. Federgruen: Markovian Control Problems. Math. Centre Tracts 97, Amsterdam 1983. [4] A. Federgruen and P. J. Schweitzer: Nonstationary Markov decision problems with con-verging parameters. J. optim. Theory Appl. 34 (1981), 207—241. [5] A. Federgruen, P. J. Schweitzer and H. C Tijms: Contraction mappings underlying un-discounted Markov decision problems. J. Math. Anal. Appl. 65 (1978), 711 — 730. [6] A. Federgruen and H. C Tijms: The optimality equation in average cost denumerable state semi-Markov decision problems, recurrency conditions and algorithms. J. Appl. Probab. 15 (1978), 3 5 6 - 3 7 3 . [7] O. Hernandez-Lerma: Adaptive Control Processes. Springer-Verlag, Berlin—Heidelberg-New York 1989. [8] K. Hinderer: on approximate solutions of finite-stage dynamic programs. In: Dynamic Programming and its applications (M. L. Puterman, ed.), Academic Press, New York 1978, pp. 2 8 9 - 3 1 7 . [9] G. Hiibner: Contraction properties of Markov decision models with applications to the elimination of non-optimal actions. In: Dynamische optimierung, Bonner Math. Schriften 98 (1977), 5 7 - 6 5 . [10] G. Hiibner: A unified approach to adaptive control of average reward Markov decision processes. OR Spektrum 10 (1988), 161-166 . [11] M. Kurano: Discrete-time Markovian decision processes with an unknown parameter — average return criterion. J. oper. Res. Soc. Japan 15 (1972), 67—76. [12] M. Kurano: Adaptive policies in Markov decision processes with uncertain matrices. J. Inf. Optim. 4 (1983), 2 1 - 4 0 . [13] M. Kurano: Learning algorithms for Markov decision processes. J. Appl. Probab. 24 (1987), 2 7 0 - 2 7 6 . [14] P. Mandl: Estimation and control of Markov chains. Adv. in Appl. Probab. 6 (1974), 4 0 - 6 0 . [15] P. Mandl: On the adaptive control of countable Markov chains. In: Probability Theory, Banach Centre Publications, Warsaw 1979, pp. 159—173. [16] W. Whitt: Approximations of dynamic programs. Math. Oper. Res. 3 (1978), 231 — 243. Prof. Dr. Gerhard Hiibner, Universitdt Hamburg, Institut fur Mathematische Stochastik, Bundesstrasse 55, D-2000 Hamburg. Federal Republic of Germany. 71 

English

http://dml.cz/bitstream/handle/10338.dmlcz/125653/Kybernetika_27-1991-1_6.pdf

Estimation and adaptive control of span-contracting Markov decision processes

Abstract

Similar works

Full text

Available Versions

Czech digital mathematics library

Institute of Mathematics AS CR, v. v. i.