We prove in a dynamic programming framework that uniform convergence of the finite horizon values implies that asymptotically the average accumulated payoff is constant on optimal trajectories. We analyze and discuss several possible extensions to two-person games

Sorin, S

Venel, X

Vigeral, G

English

Sorin S

Venel X

Vigeral G

Archivio della ricerca- LUISS Libera Università Internazionale degli Studi Sociali Guido Carli di Roma

arXiv:1012.5149v1  [math.OC]  23 Dec 2010ASYMPTOTIC PROPERTIES OF OPTIMAL TRAJECTORIES IN DYNAMICPROGRAMMINGSYLVAIN SORIN, XAVIER VENEL, GUILLAUME VIGERALAbstract. We show in a dynamic programming framework that uniform convergence of thefinite horizon values implies that asymptotically the average accumulated payoff is constant onoptimal trajectories. We analyze and discuss several possible extensions to two-person games.1. PresentationConsider a dynamic programming problem as described in Lehrer and Sorin [1]. Given a set ofstates S, a correspondence Φ from S to itself with non empty values and a payoff function f fromS to [0, 1], a feasible play at s ∈ S is a sequence {sm} of states with s1 = s and sm+1 ∈ Φ(sm). Itinduces a sequence of payoffs {fm = f(sm)},m = 1, ..., n, .... Recall that starting from a standardproblem with random transitions and/or signals on the state, this presentation amounts to workon the set of probabilities on S and to consider expected payoffs.Let vn(s) (resp. vλ(s)) be the value of the n stage program Gn(s) (resp. λ discounted programGλ(s)) starting from state s. The asymptotic approach deals with asymptotic properties ofthe values vn and vλ as n goes to ∞ or λ goes to 0.The uniform approach focuses on properties of the strategies that hold uniformly in longhorizons. v∞ is the uniform value if for each ε > 0 there exists N such that for each s ∈ S:1) there is a feasible play {sm} at s with1nn∑m=1f(sm) ≥ v∞(s)− ε, ∀n ≥ N2) for any feasible play {s′m} at s and any n ≥ N1nn∑m=1f(s′m) ≤ v∞(s) + ε.Obviously the second approach is more powerful than the second (existence of a uniform valueimplies existence of an asymptotic value : the limit of vn exists) but it is also more demanding:there are problems without uniform value where the asymptotic value exists (see Section 2).Note that the condition for the existence of a uniform value implies that the average accumulatedpayoff on optimal trajectories remains close to the value.We will prove that a similar phenomenon holds true under conditions that are stronger thanthe existence of an asymptotic value but weaker than the existence of a uniform value.Say that the dynamic programming problem is regular if :i) lim vn(s) = v(s) exists for each s ∈ S.ii) the convergence is uniform.This condition was already introduced and studied in Lehrer and Sorin [1] (see Section 2).We consider the following property P:Date: October 2009.12 SYLVAIN SORIN, XAVIER VENEL, GUILLAUME VIGERALFor any ε > 0, there exists n0, such that for all n ≥ n0, for any state s and any feasible play{sm} ε-optimal for Gn(s) and for any t ∈ [0, 1]:(1) 3ε ≥1n([tn]∑m=1fm)− tv(s) ≥ −3ε.where [tn] stands for the integer part of tn.This condition says that the average payoff remains close to the value on every almost-optimaltrajectory with long duration (but the trajectory may depend on this duration).It also implies a similar property on every time interval.2. Examples and comments1) The existence of the asymptotic value is not enough to control the payoff as required inproperty P.An example is given in Lehrer and Sorin [1] (Section 2), where both lim vn and lim vλ exist onS but where the asymptotic average payoff is not constant on the unique optimal trajectory, noron ε-optimal trajectories: in G2n, an optimal play will induce n times 0 then n times 1 whilev = 1/2.Note that this example is not regular: the convergence of vn to v is not uniform.2) Recall that in the framework of dynamic programming, regularity is also equivalent touniform convergence of vλ (and with the same limit), see Lehrer and Sorin [1] (Section 3).Note also that this regularity condition is not sufficient to obtain the existence of a uniform value,see Monderer and Sorin [2] (Section 2).3) General conditions for regularity can be found in Renault [5].3. Main resultTheorem 3.1. Assume that the program is regular, then P holds.ProofLet us start with the upper bound inequality in (1).The result is clear for t ≤ ε (recall that that the payoff is in [0, 1]). Otherwise let n1 large enoughso that n ≥ n1 implies ||vn − v|| ≤ ε by uniform convergence. Then the required inequality holdsfor n ≥ n2 with [εn2] ≥ n1.Consider now the lower bound inequality in (1).The result holds for t ≥ 1 − ε by the ε-optimal property of the play, for n ≥ n1. Otherwise weuse the following lemma from Lehrer and Sorin [1] (Proposition 1).Lemma 3.1. Both lim sup vn and lim sup vλ decresase on feasible histories.In particular, starting from s[tn] the value of the program for the last n− [tn] stages is at mostv(s[tn]) + ε for n ≥ n2, by uniform convergence, hence less than the initial v(s) + ε, using theprevious Lemma. Since the play is ε-optimal in Gn(s), this implies that(2)[tn]∑m=1fm + (n− [tn])(v(s) + ε) ≥ n(vn(s)− ε) ≥ n(v(s)− 2ε)hence the required inequality.4. Extensions4.1. Discounted case.A similar result holds for the program Gλ corresponding to the evaluation∑∞m=1 λ(1−λ)m−1fm.Explicitly, one introduces the property P′:AS 3For any ε > 0, there exists λ0, such that for all λ ≤ λ0, for any state s and any feasible play {sm}ε-optimal for Gλ(s) and for any t ∈ [0, 1]:(3) 3ε ≥n(t;λ)∑m=1λ(1− λ)m−1fm)− tv(s) ≥ −3ε.where n(t;λ) = inf{p ∈ IN;∑pm=1 λ(1 − λ)m−1 ≥ t}. Stage n(t;λ) corresponds to the fraction tof the total duration of the program.Theorem 4.1. Assume that the program is regular, then P’ holds.ProofThe proof follows the same lines than the proof of Theorem 3.1.Recall that by regularity both vn and vλ converge uniformly to v. Moreover the discounted sums(1−λ)−N∑Nm=1 λ(1−λ)m−1fm belong to the convex hull of the averages1n∑nm=1 fm; 1 ≤ n ≤ N .The counterpart of equation (2) is now(4)n(t;λ)∑m=1λ(1− λ)m−1fm + (1− t)(v(s) + ε) ≥ (vλ(s)− ε) ≥ v(s)− 2ε4.2. Continuous time.Similar results holds in the following set-up: vT (x) is the value of the control problem ΓT withcontrol set U where the state variable inX is governed by a differential equation (or more generallya differential inclusion)ẋt = f(xt, ut)starting from x at time 0. The real payoff function is g(x, u) and the evaluation is given by:1T∫ T0g(xt, ut)dt.Regularity in this framework amounts to uniform convergence (on X) of VT to some V . (Sufficientconditions for regularity can be found in Quincampoix and Renault [4]). The correspondingproperty is now P”:For any ε > 0, there exists T0, such that for all T ≥ T0, for any state x and any feasible trajectoryε-optimal for ΓT (x) and for any θ ∈ [0, 1]:(5) 3ε ≥1T∫ θT0g(xt, ut)dt− θV (x) ≥ −3ε.Theorem 4.2. Assume that the optimal control problem is regular, then P” holds.ProofFollow exactly the same lines than the proof of Theorem (2).Finally the same tools can be used for an evaluation of the form λ∫ +∞0 e−λtg(xt, ut)dt.5. Two-player zero-sum gamesIn trying to extend this result to a two-person zero-sum framework, several problems occurs.4 SYLVAIN SORIN, XAVIER VENEL, GUILLAUME VIGERAL5.1. Optimal strategies on both sides.First it is necessary, to obtain good properties on the trajectory, to ask for optimality on bothsides.For example in the Big Match with no signals,α βa 1∗ 0∗b 0 1where a ∗ denotes an absorbing payoff, the optimal strategy of player 1 in the “asymptotic game”on [0, 1] is to play “a before time t” with probability t, see Sorin [6] Section 5.3.2. Obviously,if there is no restrictions on player 2’s moves the average payoff will not be constant. However,the optimal strategy of player 2 is “always (1/2, 1/2)” hence time independent on [0, 1]. It thusinduces a constant payoff and it is easy to see that the property is robust to small perturbationsin the evaluation of the payoff.5.2. Player 1 controls the transition.Consider a repeated game with finite characteristics (states, moves, signals, ...) and use therecursive formula corresponding to the canonical representation with entrance laws being consis-tent probabilities on the universal belief space, see Mertens, Sorin and Zamir [3], Chapters III.1,IV.3. This representation preserves the values but in the auxiliary game, if player 1 controls thetransition an optimal strategy of player 2 is to play a stage by stage best reply. Hence the modelreduces to the dynamic programming framework and the results of the previous sections apply.A simple example corresponds to a game with incomplete information on one side where asymp-totically an optimal strategy of the uniform player 1 is a splitting at time 0, while player 2 canobain u(pt) at time t where u is the value of the non-revealing game and pt the martingale ofposteriors at time t, see Sorin [6], 3.7.2.5.3. Example.Back to the general framework of two person zero-sum repeated games, the following exampleshows that in addition one has to strengthen the conditions on the pair of ε-optimal strategies.We exhibit a game having a uniform value v but for some state s with v(s) = 0 one can construct,for each n, optimal strategies in Γn(s) inducing essentially a constant payoff 1 during the firsthalf of the game.Starting from the initial state s, the tree representing the game Γ has countably many subgamesΓ̃2n, the transition being controlled by player 1 (with payoff 0). In Γ̃2n there are at most n stagesbefore reaching an absorbing state. At each of these stages of the form (2n,m),m = 1, ...n,the players plays a “jointly controlled” process leading either to a payoff 1 and the next stage(2n,m+ 1) (if they agree) or an absorbing payoff x2n,m with (m− 1) + (2n− (m− 1))x2n,m = 0,otherwise. Hence every feasible path of length 2n in Γ̃2n gives a total payoff 0. Obviouslythe uniform value exists since each player can stop the game at each node, inducing the sameabsorbing payoff. The representation is as follows:Notice that in the 2n + 1 stage game, after a move of player 1 to Γ̃2n, any play is compatiblewith optimal strategies, in particular those leading to the sequence of payoffs 2n times 0 or ntimes 1 then n times −1.5.4. Conjectures.A natural conjecture is that in any regular game (i.e. where vn converges uniformly to v):for any ε > 0, there exists n0, such that for all n ≥ n0, for any initial state s, there exists a couple(σn, τn) of ε-optimal strategies in Gn(s) such that for any t ∈ [0, 1]:(6) 3ε ≥1nEsσn,τn([tn]∑m=1fm)− tv(s) ≥ −3ε.AS 5•s0•(2, 1)0•(4, 1)00•(2n, 1)0Γ̃2 Γ̃4 Γ̃2nFigure 1. The game Γ starting from state sC A C A C A C A C AC 1−→ 0* C 1−→ x2n,2* C 1−→ x2n,m* C 1−→ x2n,n* C -1* -1*A 0* 0* A x2n,2* x2n,2* A x2n,m* x2n,m* A x2n,n* x2n,n* A -1* -1*(2n, 1) (2n, 2) · · · (2n,m) · · · (2n, n) −1∗Figure 2. The subgame Γ̃2n starting from state (2n, 1)where [tn] stands for the integer part of tn and fm is the payoff at stage m.A more elaborate conjecture would rely on the existence of an asymptotic game Γ∗ played in con-tinuous time on [0, 1] with value v (as in Section 5.1). We use the representation of the repeatedgame as a stochastic game trough the recursive structure as above, see Mertens, Sorin, Zamir [3],Chapter IV. The condition is now the existence of a couple of strategies (σ, τ) in the asymptoticgame that would depend only on the time t ∈ [0, 1] and on the current state s such that forany ε > 0, there exists η with the following property: in any repeated game where the (relative)weight of stage m is αm, with {αm} decreasing and less than η, thus defining a partition Π of[0, 1], the strategies (σΠ, τΠ) induced in the repeated game by (σ, τ) satisfies (6).Acknowledgment:This work was done while the three authors were members of the Equipe Combinatoire et Opti-misation.Sorin’s research was supported by grant ANR-08-BLAN-0294-01 (France).References[1] Lehrer E. and S. Sorin (1992) A Uniform tauberian theorem in dynamic programming, Mathematics of Oper-ations Research, 17, 303-307.[2] Monderer D. and S. Sorin (1993) Asymptotic properties in dynamic programming, International Journal ofGame Theory, 22, 1-11.[3] Mertens J.-F. , S. Sorin and S. Zamir (1994) Repeated Games, CORE Discussion Papers 9420, 9421, 9422.[4] Quincampoix M. and J. Renault (2009) On the existence of a limit value in some non expansive optimal controlproblems, preprint.[5] Renault J. (2007) Uniform value in dynamic programming, Cahier du CEREMADE, 2007-1.[6] Sorin S. (2002) A first course on zero-sum repeated games, Mathmatiques et Applications, 37, Springer.[7] Sorin S. (2005) New approaches and recent advances in two-person zero-sum repeated games, Advances inDynamic Games, A. Nowak and K. Szajowski (eds.), Birkhauser, 67-93.6 SYLVAIN SORIN, XAVIER VENEL, GUILLAUME VIGERALEquipe Combinatoire et Optimisation, CNRS FRE 3232, Faculté de Mathématiques, UPMC-Paris6, 175 Rue du Chevaleret, 75013 Paris, FranceGREMAQ Universit de Toulouse 1 Manufacture des Tabacs, Aile J.J. Laffont 21 alle de Brienne31000 Toulouse, FranceINRIA Saclay - Ile-de-France and CMAP, Ecole Polytechnique, route de Saclay, 91128 Palaiseaucedex, FranceE-mail address: sorin@math.jussieu.fr, xavier.venel@sip.univ-tlse1.fr, guillaumevigeral@gmail.com

Asymptotic properties of optimal trajectories in dynamic programming

https://iris.luiss.it/retrieve/handle/11385/197463/101919/Sorin_Vigeral_Venel_arxiv_last_allowed.pdf

Asymptotic properties of optimal trajectories in dynamic programming

Abstract

Similar works

Full text

Available Versions

Archivio della ricerca- LUISS Libera Università Internazionale degli Studi Sociali Guido Carli di Roma