In this report, we outline a method for approximating a Markovian (or feedback-Nash) equilibrium of a dynamic game, possibly subject to coupled-constraints. We treat such a game as a "multiple" optimal control problem. A method for approximating a solution to a given optimal control problem via backward induction on Markov chains was developed in [Kra01]. A Markovian equilibrium may be obtained numerically by adapting this backward induction approach to a stage Nikaido-Isoda function (described in [KZ06])

Azzato, Jeffrey D

Krawczyk, Jacek B

In this report, we outline a method for approximating a Markovian (or feedback-Nash) equilibrium of a dynamic game, possibly subject to coupled-constraints. We treat such a game as a "multiple" optimal control problem. A method for approximating a solution to a given optimal control problem via backward induction on Markov chains was developed in Krawczyk (2006). A Markovian equilibrium may be obtained numerically by adapting this backward induction approach to a stage Nikaido-Isoda function (described in Krawczyk & Zuccollo (2006))

Krawczyk, Jacek

Azzato, Jeffrey

Munich RePEc Personal Archive

MPRAMunich Personal RePEc ArchiveNISOCSol an algorithm forapproximating Markovian equilibria indynamic games with coupled-constraintsJacek Krawczyk and Jeffrey AzzatoVictoria University of Wellington2006Online at http://mpra.ub.uni-muenchen.de/1195/MPRA Paper No. 1195, posted 16. December 2006A REPORT ONNISOCSolAN ALGORITHM FOR APPROXIMATING MARKOVIAN EQUILIBRIA IN DYNAMICGAMES WITH COUPLED-CONSTRAINTSJACEK B. KRAWCZYK & JEFFREY D. AZZATOAbstract. In this report, we outline a method for approximating aMarkov-ian (or feedback-Nash) equilibrium of a dynamic game, possibly subjectto coupled-constraints. We treat such a game as a “multiple” optimal con-trol problem. A method for approximating a solution to a given optimalcontrol problem via backward induction on Markov chains was devel-oped in Krawczyk (2001). A Markovian equilibrium may be obtainednumerically by adapting this backward induction approach to a stageNikaidoˆ-Isoda function (described in Krawczyk & Zuccollo (2006)).2006 Working PaperSchool of Economics and FinanceJEL Classification: C63 (Computational Techniques), C72 (Noncooperative Games), C87 (EconometricSoftware), E62 (Taxation), Q25 (Water; Climate).MSC Categories: 90C39 (Dynamic Programming), 91A25 (Dynamic Games), 91A80 (Applications ofGame Theory), 91B76 (Environmental Economics).Authors’ Keywords: Computational economics, Nikaidoˆ-Isoda function, approximating Markovdecision chains.This report documents 2005 - 2006 research intoComputational Economics Methodsdirected by Jacek B. Krawczyksupported by VUW FCA FRG-05 (24644).Correspondence should be addressed to:Jacek B. Krawczyk. Commerce & Administration Faculty, Victoria University of Wellington, PO Box 600,Wellington, New Zealand; fax +64-4-4712200. Email: Jacek.Krawczykatvuw.ac.nz ;http://www.vuw.ac.nz/staff/jacek krawczyk1. IntroductionThis report draws from Krawczyk (2006). It also provides some technical remarks on the use ofthe new routines constituting NISOCSol.In Krawczyk & Tidball (2006) a finite horizon unconstrained dynamic game was solved throughbackward induction. At each stage, a Markovian (feedback-Nash) equilibrium was computedas a solution to a system of the Bellman equations where each Bellman equation characteriseda player’s value function. Due to the lack of constraints, the computations were rather straight-forward albeit sensitive to the horizon length.The backward induction technique will be also applied in this paper. However, instead ofsimultaneous maximisation of the players’ value functions (as in Krawczyk & Tidball (2006)),constrained min-maximisation of the stage Nikaidoˆ-Isoda function will be performed witheach player’s value function characterised by their Bellman equation. This method appearssatisfactory in the constrained dynamic game context.In this paper, some background on Markovian (feedback-Nash) solutions is given in Section 2.The technical remarks of Section 3 explain how a feedback solution to a dynamic game can beobtained.2. Markovian (Feedback-Nash) Equilibrium2.1. Preliminaries. Denote the game value (Bellman function) for player i from state x(τ) attime τ by the payoff-to-go1 function Fi(x(τ);u):(1) F(τ)i(x(τ);u) =T−1∑t=τρt(φi(x(t)) − c3,i(u(t)i)2)+ ρTk(x(T)i)when the players are using strategy u = [u1, u2, u3]. Here, φi(·) is the static (concave) payoff forplayer i contingent on full utilisation of production capacity, c3,i(·)2 is a quadratic investmentcost†, k(·) is the capacity’s scrap value function and ρ the discount factor.Definition 1. We say that strategies u∗1,u∗2,u∗3constitute a feedback-Nash (or Markoviansubgame-perfect) equilibrium on IR3+ if:(i) They are admissible at any x(t) ∈ IR3+, and1Or continuation payoff.†Payoff concavity and a quadratic investment cost are not crucial for the method - weak convex-concavity of theNikaidoˆ-Isoda function suffices.1(ii) For any history x(τ), x(τ+1), . . . , x(t) and for any admissible (ui,u∗−i), i = 1, 2, 3 the followinginequalities hold:F(t)i(x(t);u∗) ≥ F(t)i(x(t);ui,u∗−i), i = 1, 2, 3.(2)Here:(3) u(t)i= ui(x(t), t)and each strategy is a map:(4)uiR3+ × T =⇒ R i = 1, 2, 3∈ ∈(x(t), t) u(t)it ∈ {0, . . . ,T − 1} ≡ TThe symbol u−i will be used to represent the other players’ strategies.So, we wish to solve problem (2) for strategies (3).2.2. A solution method.2.2.1. Stage-dependent Nikaidoˆ-Isoda functions. Acoupled-constraint game is a difficult construct.A solution concept suitable for this type of game is a “generalised”Nash equilibrium (seeArrow& Debreu (1954), McKenzie (1959), Rosen (1965), Hobbs & Pang (2006), Contreras et al. (2004),Haurie & Krawczyk (1997), Krawczyk & Uryasev (2000), Krawczyk (2005b)). A numericalapproachmay be necessary to obtain such a general solution - this was the case for a static gamesolved in Krawczyk & Zuccollo (2006), and also for the open-loop game of Krawczyk (2005b).For Markovian games, a solution consists of strategies understood as maps rather than timeprofiles. The latter may also be obtained for Markovian games as realisations of strategies inform of (4).SOCSol4L (see Azzato & Krawczyk (2006); also e.g., Krawczyk (2001) and Krawczyk (2005a))is a suite of MATLABR© routines capable of approximating optimal strategies to intertemporaldecision problems having a single planner2. Hence, SOCSol4L can rather straightforwardly beused to approximate the Pareto optimal strategies.2I.e., SOCSol4L solves optimal control problems.2Here we explain the usage of SOCSol4L to compute feedback-Nash equilibrium strategies. Inthis case, the solution strategies will be maximisers of the Nikado-Isoda functions formulatedstage-by-stage from the players’ value functions Fi(x(τ);u) (see (1)) and iterated backward intime3. The equilibrium solution (also, equilibrium existence, uniqueness and the algorithmconvergence) will be contingent upon the stageNikado-Isoda functions’ weak convex-concavityand the constraint set’s convexity. These can be verified.In practical terms, to solve a dynamic game (2) in admissible feedback strategies (3) we need tocombine results of the Convergence Theorem (see e.g., Krawczyk & Uryasev (2000), Krawczyk(2005b)) with the Bellman optimality principle. The use of optimality principle implies thatstage gameswill be solved backward in time. Hence, the Convergence Theorem will be appliedat each stage (backward in time) with the role of a “one-off” (or static) utility being played bythe utility-to-go F(t)i(x(t);u).2.2.2. StageGames. LetU(t)(−i)denote the i th player’s strategy set at time t. In a coupled-constraintcontext, this set depends on the other players’ strategies (indicated by the subscript (−i)). Itfollows from Section 2.1 that:U(t)(−1)∪ U(t)(−2)∪ U(t)(−3)= U(t) ⊂ IR3Assume that f (i)(x(t),u(t)i) and k(x(T)) are concave, with:(5) f (i)(x(t),u(t)i) = φi(x(t)) − c3,i(u(t)i)2 , f(i)T(x(T)i) = ki(x(T)i)Define V(t)i(x(t)), an optimal value function for player i at stage t, by:V(t)i(x(t)) = maxu(t)i∈U(t)(−i)F(t)i(x(t);u(t)i,u−i(x(t), t)), t = T − 1, . . . , 1, 0(6)V(i)T(x(T),T) = f(i)T(xTi )(7)where:(8) F(t)i(x(t);u(t)i,u−i(x(t), t)) ≡ f (i)(x(t),u(t)i) + ̺V(t+1)i(x(t+1))3The dynamic game at hand is finite horizon.3The following theorem4 establishes a basis for using dynamic programming as a computationaltechnique for Markovian (feedback-Nash) equilibria in dynamic games.Theorem 1. If there exist value functions V(t)i(x(t)) and strategies ui(x(t), t) which satisfy equations (6),(8) and (7) for t = T − 1, . . . 1, 0, where x(t) is a vector of state variables observable at t, then the strategy:u∗ = (u∗i ,u∗−i)constitutes aMarkovian (feedback-Nash) equilibrium for the dynamic game under a feedback informationpattern5. Moreover, the value functions V(t)i(x(t)) represent player i’s optimal utility for the game startingat (x(t), t). In particular:(9) V(0)i(x(0)) = Φi(x(0);u∗)If each stage game F(·)i,F(·)−iis concave, then it makes sense to ask whether the stage games haveunique equilibria. We observe that at least the last game “played” to maximise the utility-to-gofunctions F(T−1)i,F(T−1)−iis concave. This is necessary for the game’s solution. Weak convex-concavity of the Nikaidoˆ-Isoda functions for stage games (or diagonal strict concavity of thesegames) can be established stage-wise backward in time to demonstrate solution uniqueness.Notice that in equilibrium:V(t)i(x(t)) = F(t)i(x(t);u∗), i = 1, 2, 3, t = T − 1, . . . , 1, 0Hence, if player i played strategy vi while the rest were playing u∗, then:F(t)i(x(t);vi|u∗) ≤ F(t)i(x(t);u∗), i = 1, 2, 3, t = T − 1, . . . , 1, 0However, if u is not an equilibrium, then player i might improve his or her utility-to-go byplaying the strategy vi (given that the other players play u). Consequently, the expression:(10) ψ(t)i(u,v) = F(t)i(x(t);vi|u) − F(t)i(x(t);u), t = T − 1, . . . , 04“Standard” in dynamic games, see Bas¸ar & Olsder (1982), Theorem 6.6, pp. 284-285.5Such an equilibrium is subgame perfect.4might be positive. This suggests that we can define a stage Nikaidoˆ-Isoda function whosemaximisation will reveal the equilibrium u∗ under certain concavity conditions of F(t)i(· ;u), t =T − 1, . . . , 1, 0, i = 1, 2, 3.Define the stage Nikadoˆ-Isoda function at state x(t) by:(11) Ψx(t) : U(t) ×U(t) → IR : (u,v) 7→3∑i=1(F(t)i(x(t);vi|u) − F(t)i(x(t);u)), t = T − 1, . . . , 1, 0Obviously:(12) Ψx(t)(u,u) ≡ 0, u ∈ U(t)Each summand in (11) can be thought of as the improvement in the player’s value function thathe (or she) will receive by changing his (or her) strategy from ui to vi while all other playerscontinue to playu−i. TheNikaidoˆ-Isoda function thus represents the sumof these value functionimprovements. Note that itsmaximum for a given u is always non-negative as a consequence of(12). Moreover, (11) is non-positive everywhere when either u or v is a Markovian equilibriumstrategy, for in an equilibrium situation no player can unilaterally improve their payoff, and soin this case each summand is at most zero.From here, we conclude that when the Nikaidoˆ-Isoda function (11) cannot be made (signifi-cantly) positive at each stage for a given u, we have (approximately) reached the Markovianequilibrium point. This observation is useful in constructing a termination condition for ouralgorithm: choose an ε > 0 such that the equilibrium has been reached to a sufficient degree ofprecision when maxv∈UΨx(t)(us,v) < ε, where us ∈ U is computed at the current iteration s.More formally, we shall compute a “normalised” Markovian equilibrium which will be a Nashequilibrium under weak convex-concavity ofΨx(t)(u,v).Definition 2. We call an admissible strategy u∗ ∈ U aNash normalised Markovian equilibrium if:(13) maxv∈UΨx(t)(u∗,v) = 0A Nash normalised Markovian equilibrium u∗ is a unique feedback-Nash equilibrium if:• (∀t) Ψx(t)(u,v) is weakly convex-concave, or5• (∀t)∑ni=1 F(t)i(x(t);u) is diagonally strictly concave (see Krawczyk & Tidball (2006),where this feature was examined in a dynamic game).3. Numerical results through NISOCSolThis section should be read in conjunction with Azzato & Krawczyk (2006), where the calls toSOCSol4L are explained. Here, we explain the differences that the user should be familiar within order to be able to call NISOCSol.3.1. State and time discretisation. These are organised in the same way as for SOCSol4L.3.2. Options. In this version of NISOCSol, it is assumed that eachplayer has only onedimensionof control. Consequently, the control dimension should be set to the number of players.3.3. User defined .m functions. The utility functions for NISOCSol return values for the i thplayer, where i is passed as an argument following those standard for SOCSol4L. If the game isdefined as discrete-timewith times 0,∆, 2∆, . . . ,T, where T is a multiple of∆, then the followingshould also be observed:• TimeStep = ones(1, T∆)/T∆;• The return value of the instantaneous cost function needs to be multplied by T∆.• The return value of the motion (or “delta”) function needs to be multplied by T∆.With these modifications a typical NISOCSol call has the form:NISOCSol(‘DeltaFunctionFile’, ‘InstantaneousCostFunctionFile’,‘TerminalStateFunctionFile’, StateLB, StateUB, StateStep, TimeStep,‘ProblemFile’, Options, InitialControlValue, A, b, Aeq, beq, ControlLB,ControlUB, ‘UserConstraintFunctionFile’);3.4. A test: a static equilibrium obtained as feedback-Nash. The static solution of the RiverBasin Pollution game has been cited in many publications (e.g., (Krawczyk & Uryasev (2000),Krawczyk (2005b) and Krawczyk & Zuccollo (2006)). The unique coupled-constraint equilib-rium is:x∗ = (21.14, 16.03, 2.73)We will use this problem to test NISOCSol.6Presumably, with no investment cost and no depreciation, agents should move to this equilib-rium from any “initial condition” in one step. So we set the following parameter values: T = 1,c3,i ≡ 0, ρ = 1, ki(·) ≡ φi(x). Furthermore, we assume the following system dynamics:(14) x(t+1)i= (1 − µi)x(t)i+ u(t)iAs there is no depreciation, µi ≡ 0. We assume that:(15) φi(x) = [d1 − d2(x1 + x2 + x3)]xi︸                        ︷︷                        ︸Revenue− (c1,i + c2,ixi)xi︸          ︷︷          ︸CostThe economic constants d1 and d2 determine an inverse demand law. We assign them therespective values 3 and 0.01. The values of the cost function coefficients c1,i and c2,i and theother model parameters are given in Table 1 below.Player i c1,i c2,i ei δi,1 δi,21 0.10 0.01 0.50 6.5 4.5832 0.12 0.05 0.25 5.0 6.2503 0.15 0.01 0.75 5.5 3.750Table 1. Constants for the River Basin Pollution gameThe fourth column of Table 1 gives the emission parameters and the fifth gives the pollutiontransportation & decay parameters that govern contamination levels at location ℓ = 1, 2. Thelocal authority imposes pollution constraints:(16) qℓ(x) =3∑i=1δi,ℓeixi ≤ Kℓ, ℓ = 1, 2We take K1 = K2 = 100.Given this information and recalling the comments in Subsections 3.1, 3.2 and 3.3, the followingfunctions were defined after Azzato & Krawczyk (2006):Delta(u, x, t)Costi(u, x, t, i)7Termi(x, i)Constraints(u, x, ts)The programme’s output is the null strategy for x(0) = x∗ - see Figure 1, which shows strategiesat t = 0. Players’ actions u1,u2 and u3 are each plotted (from left to right) as functions of thethird, second and first state variables. We see that the feedback strategies intersect with thehorizontal axis (i.e., no action) for the initial condition x∗.The strategy realisations are the state and action time profiles shown in Figure 2. The solidlines (which are constant in time) demonstrate that no changes are made when given x(0) = x∗.The dash-dotted lines show that the equilibrium point is achieved from an arbitrary initial state(here, (22, 15, 4)).0 5 10 15 20−505u10 5 10 15 20−505u20 5 10 15 20−505u3x3 x2x1Figure 1. One-step Markovian equilibrium strategies.82121.522x 1151617x 2234x 3−202u1−202u20 1−202u3TimeFigure 2. One-step Markovian equilibrium states and actions.Lagrange multiplier rules have also been computed. To ensure that the standards (16) areobeyed, the government needs to inform the players what taxes would be collected shouldthe pollution levels be exceeded. Under the feedback information pattern, the value of theLagrange multipliers is a function of states [x(t)1, x(t)2, x(t)3] (here, t = 0). See Figure 3.90 5 10 15 200.30.40.50.60.70.80.9λx3 x2x1Figure 3. Lagrange multipliers at t = 0 as a function of state.4. Concluding RemarksThis report outlines a set of useful machinery for the study dynamic games (possibly) subject tocoupled-constraints (typical of environmental management problems). In particular, we haveseen how the Nikaidoˆ-Isoda function can be utilised to find a coupled-constraint equilibriumthrough an optimisation approach.ReferencesK.J. Arrow & G. Debreu, 1954, “Existence of an Equilibrium for a Competitive Economy”, Econometrica,Vol. 22, No. 3, pp. 265-290.Azzato, J. & J.B. Krawczyk, 2006, “SOCSol4L An improved MATLABR© Package for Approximatingthe Solution to a Continuous-Time Stochastic Optimal Control Problem”, Working Paper, School ofEconomics and Finance, VUW, [MPRA: 1179]. Available at: http://mpra.ub.uni-muenchen.de/1179/ on2006-12-16.Bas¸ar, T. & G.J. Olsder, 1982, “Dynamic Noncooperative Game Theory”, Academic Press, New York.10Contreras, J., M. Klusch & J.B. Krawczyk, 2004, “Numerical solutions to Nash-Cournot equilibria incoupled constraint electricity markets”, IEEE Transactions on Power Systems, Vol. 19, No. 1, 195-206, [doi:10.1109/TPWRS.2003.820692].Haurie, A. & J.B. Krawczyk, 1997, “Optimal charges on river effluent from lumped and distributedsources”, Environmental Modeling and Assessment, Vol. 2, No. 3, pp. 93-106.Hobbs, B. & J.-S. Pang, 2006, “Nash-Cournot Equilibria in Electric Power Markets with Piecewise LinearDemand Functions and Joint Constraints”, Operations Research, forthcoming.Krawczyk, J.B., 2001, “A Markovian Approximated Solution to a Portfolio Management Prob-lem”, Information Technology for Economics and Management, Vol. 1, No. 1. Available at:http://www.item.woiz.polsl.pl/issue/journal1.htm on 2006-12-14.Krawczyk, J.B., 2005, “Numerical Solutions to Lump-Sum Pension Fund Problems that Can Yield Left-Skewed Fund Return Distributions”. In: C. Deissenberg and R.F. Hartl (eds.), Optimal Control and DynamicGames, Springer, pp. 155-176.Krawczyk, J.B., 2005, “Coupled constraint Nash equilibria in environmental games”, Resource and EnergyEconomics, Vol. 27, Iss. 2, pp. 157-181.Krawczyk, J.B., 2006, “Coupled Constraint Markovian Equilibria in Dynamic Games of Compliance”,Seminar at Kyoto University Institute of Economic Research, 30 November 2006.Krawczyk, J.B. & M. Tidball, 2006, “A Discrete-Time Dynamic Game of Seasonal Water Allocation”,Journal of Optimization Theory and Applications, Vol. 128, No. 2, pp. 411-429.Krawczyk, J.B. & S. Uryasev, 2000, “Relaxation algorithms to find Nash equilibria with economic appli-cations”, Environmental Modeling and Assessment, Vol. 5, No. 1, pp. 63-73.Krawczyk, J.B. & J. Zuccollo, 2006, “NIRA-3 An improved MATLAB package for finding Nash equilibriain infinite games”, Working Paper, School of Economics and Finance, VUW, [MPRA: 1119]. Available at:http://mpra.ub.uni-muenchen.de/1119/ on 2006-12-16.McKenzie, L.W., 1959, “On theExistence ofGeneral Equilibrium for aCompetitiveMarket”, Econometrica,Vol. 27, No. 1, pp. 54-71.Nikaidoˆ, H. & K. Isoda, 1955, “Note on Noncooperative Convex Games”, Pacific Journal of Mathematics,Vol. 5, Supp. 1, pp. 807-815.Rosen, J.B., 1965, “Existence and Uniqueness of Equilibrium Points for Concave N-Person Games”,Econometrica, Vol. 33, No. 3, pp. 520-534.11

English

NISOCSol an algorithm for approximating Markovian equilibria in dynamic games with coupled-constraints

Krawczyk, Jacek B.

Azzato, Jeffrey D.

MPRAMunich Personal RePEc ArchiveA report on NISOCSol: An algorithm forapproximating Markovian equilibria indynamic games with coupled-constraintsJacek B. Krawczyk and Jeffrey D. AzzatoVictoria University of WellingtonDecember 2006Online at http://mpra.ub.uni-muenchen.de/10235/MPRA Paper No. 10235, posted 30. August 2008 08:36 UTCA REPORT ONNISOCSolAN ALGORITHM FOR APPROXIMATING MARKOVIAN EQUILIBRIA INDYNAMIC GAMES WITH COUPLED-CONSTRAINTSJACEK B. KRAWCZYK & JEFFREY D. AZZATOAbstract. In this report, we outline a method for approximat-ing a Markovian (or feedback-Nash) equilibrium of a dy-namic game, possibly subject to coupled-constraints. Wetreat such a game as a “multiple” optimal control problem.A method for approximating a solution to a given optimalcontrol problem via backward induction on Markov chainswas developed in [Kra01]. A Markovian equilibrium maybe obtained numerically by adapting this backward induc-tion approach to a stage Nikaidô-Isoda function (describedin [KZ06]).2006 Working PaperSchool of Economics and FinanceJEL Classification: C63 (Computational Techniques), C72 (Noncooperative Games), C87(Econometric Software), E62 (Taxation), Q25 (Water; Climate).MSC Categories: 90C39 (Dynamic Programming), 91A25 (Dynamic Games), 91A80(Applications of Game Theory), 91B76 (Environmental Economics).Authors’ Keywords: Computational economics, Nikaidô-Isoda function, ApproximatingMarkov decision chains. This report documents 2005–2006 research intoComputational Economics Methodsdirected by Jacek B. Krawczyksupported by VUW FCA FRG-05 (24644)Correspondence should be addressed to:Jacek B. Krawczyk. Faculty of Commerce & Administration, Victoria University of Wellington,P.O. Box 600, Wellington, New Zealand. Fax: +64-4-4712200 Email: J.Krawczykatvuw.ac.nzWebpage: http://www.vuw.ac.nz/staff/jacek_krawczykContents1. Introduction 12. Markovian (feedback-Nash) equilibrium 12.1. Preliminaries 12.2. A solution method 23. Numerical results through NISOCSol 53.1. State and time discretisation 53.2. Options 53.3. User defined .m functions 53.4. A test: a static equilibrium obtained as feedback-Nash 54. Concluding remarks 8References 81. IntroductionThis report draws from [Kra06]. It also provides some technical remarks on the useof the new routines constituting NISOCSol.In [KT06] a finite horizon unconstrained dynamic game was solved through backwardinduction. At each stage, a Markovian (feedback-Nash) equilibrium was computed asa solution to a system of the Bellman equations where each Bellman equation char-acterised a player’s value function. Due to the lack of constraints, the computationswere rather straightforward albeit sensitive to the horizon length.The backward induction technique will be also applied in this paper. However, in-stead of simultaneous maximisation of the players’ value functions (as in [KT06]),constrained min-maximisation of the stage Nikaidô-Isoda function (see [NI55]) will beperformed with each player’s value function characterised by their Bellman equation.This method appears satisfactory in the constrained dynamic game context.In this paper, some background on Markovian (feedback-Nash) solutions is given inSection 2. The technical remarks of Section 3 explain how a feedback solution to adynamic game can be obtained.2. Markovian (feedback-Nash) equilibrium2.1. Preliminaries. Denote the game value (Bellman function) for player i from statex(τ) at time τ by the payoff-to-go (or continuation payoff ) functionF(τ)i(x(τ);u):=T−1∑t=τρt(φi(x(t))− c3, i(u(t)i)2)+ ρTk(x(T)i)(1)when the players are using strategy u = (u1,u2,u3). Here, φi(·) is the static (con-cave) payoff for player i contingent on full utilisation of production capacity, c3, i(·)2is a quadratic investment cost,† k(·) is the capacity’s scrap value function and ρ thediscount factor.Definition 1. We say that strategies u∗1, u∗2 and u∗3 constitute a feedback-Nash (or Mar-kovian subgame-perfect) equilibrium on R3+ if1. they are admissible at every x(t) ∈ R3+, and2. for every history x(τ), x(τ+1), . . . , x(t); and for every admissible (u1,u∗−1), (u2,u∗−2) and(u3,u∗−3), the inequalitiesF(t)i (x(t);u∗) > F(t)i (x(t);ui,u∗−i) (i = 1, 2, 3) (2)hold.Here,u(t)i = ui(x(t), t)(3)†Payoff concavity and a quadratic investment cost are not crucial for the method—weak convex-concavity of the Nikaidô-Isoda function suffices.1for each i, and each strategy is a mapuiR3+ × T −→ R (i = 1, 2, 3)∈ ∈(x(t), t)7−→ u(t)i (t ∈ T := {0, . . . , T − 1})(4)The symbol u−i will be used to represent the other players’ strategies.So, we wish to solve problem (2) for strategies (3).2.2. A solution method.2.2.1. Stage-dependent Nikaidô-Isoda functions. A coupled-constraint game is a difficultconstruct. A solution concept suitable for this type of game is a “generalised” Nashequilibrium (see [AD54, CKK04, HK97, HP07, Kra05b, KU00, McK59, Ros65]). Anumerical approach may be necessary to obtain such a general solution—this was thecase for a static game solved in [KZ06], and also for the open-loop game of [Kra05b].For Markovian games, a solution consists of strategies understood as maps rather thantime profiles. The latter may also be obtained for Markovian games as realisations ofstrategies in form of (4).SOCSol4L (see [AK06]; also [Kra01, Kra05a]) is a suite of MATLAB R© routines capableof approximating optimal strategies to intertemporal decision problems having a sin-gle planner.1 Hence, SOCSol4L can rather straightforwardly be used to approximatethe Pareto optimal strategies.Here we explain the usage of SOCSol4L to compute feedback-Nash equilibrium strate-gies. In this case, the solution strategies will be maximisers of the Nikado-Isodafunctions formulated stage-by-stage from the players’ value functions Fi(x(τ);u)(see(1)) and iterated backward in time.2 The equilibrium solution (also, equilibrium exis-tence, uniqueness and the algorithm convergence) will be contingent upon the stageNikado-Isoda functions’ weak convex-concavity and the constraint set’s convexity.These can be verified.In practical terms, to solve a dynamic game (2) in admissible feedback strategies (3)we need to combine results of the Convergence Theorem (see e.g., [Kra05b, KU00])with the Bellman optimality principle. The use of optimality principle implies thatstage games will be solved backward in time. Hence, the Convergence Theorem will beapplied at each stage (backward in time) with the role of a “one-off” (or static) utilitybeing played by the utility-to-go F(t)i(x(t);u).2.2.2. Stage Games. Let U(t)(−i)denote the i-th player’s strategy set at time t. In acoupled-constraint context, this set depends on the other players’ strategies (indicated1I.e., SOCSol4L solves optimal control problems.2The dynamic game at hand is finite horizon.2by the subscript (−i)). It follows from Section 2.1 thatU(t)(−1)∪ U(t)(−2)∪ U(t)(−3)= U(t) ⊆ R3.Assume that f (i)(x(t), u(t)i)and k(x(T)) are concave, withf (i)(x(t), u(t)i):= φi(x(t))− c3,i(u(t)i)2(5)andf(i)T(x(T)i):= ki(x(T)i). (6)Define V(t)i(x(t)), an optimal value function for player i at stage t, byV(t)i(x(t)):= maxu(t)i ∈U(t)(−i)F(t)i(x(t); u(t)i ,u−i(x(t), t))(7)for each t = 0, . . . , T − 1; andV(i)T(x(T), T):= f(i)T(xTi),whereF(t)i(x(t); u(t)i ,u−i(x(t), t)):= f (i)(x(t), u(t)i)+ ̺V(t+1)i(x(t+1)). (8)The following theorem3 establishes a basis for using dynamic programming as a com-putational technique for Markovian (feedback-Nash) equilibria in dynamic games.Theorem 1. If there exist value functions V(t)i(x(t))and strategies ui(x(t), t)which satisfyequations (7)–(8) for each t = T − 1, . . . , 1, 0; where x(t) is a vector of state variablesobservable at t, then the strategyu∗ := (u∗i ,u∗−i)constitutes a Markovian (feedback-Nash) equilibrium for the dynamic game under a feedbackinformation pattern.4 Moreover, the value functions V(t)i(x(t))represent player i’s optimalutility for the game starting at(x(t), t). In particular,V(0)i(x(0))= Φi(x(0);u∗).If each stage game(F(·)i , F(·)−i)is concave, then it makes sense to ask whether the stagegames have unique equilibria. We observe that at least the last game “played” tomaximise the utility-to-go functions F(T−1)i and F(T−1)−i is concave. This is necessaryfor the game’s solution. Weak convex-concavity of the Nikaidô-Isoda functions forstage games (or diagonal strict concavity of these games) can be established stage-wise backward in time to demonstrate solution uniqueness.Notice that in equilibrium,V(t)i(x(t))= F(t)i(x(t);u∗)3“Standard” in dynamic games, see [BO82, Theorem 6.6].4Such an equilibrium is subgame perfect.3for each i = 1, 2, 3 and each t = T − 1, . . . , 1, 0. Hence, if player i played strategy viwhile the rest were playing u∗, thenF(t)i(x(t); vi|u∗)6 F(t)i(x(t);u∗)for each i = 1, 2, 3 and each t = T − 1, . . . , 1, 0. However, if u is not an equilibrium,then player i might improve his or her utility-to-go by playing the strategy vi (giventhat the other players play u). Consequently, the expressionψ(t)i (u, v) = F(t)i(x(t); vi|u)− F(t)i(x(t);u)t = T − 1, . . . , 0might be positive. This suggests that we can define a stage Nikaidô-Isoda functionwhose maximisation will reveal the equilibrium u∗ under certain concavity conditionsof F(t)i (· ;u), where i = 1, 2, 3 and t = T − 1, . . . , 1, 0.Define the stage Nikadô-Isoda function at state x(t) byΨx(t) : U(t) ×U(t) → R : (u, v) 7→3∑i=1(F(t)i(x(t); vi|u)− F(t)i(x(t);u))(9)for each t = T − 1, . . . , 1, 0. Obviously,Ψx(t)(u,u) ≡ 0 (10)for every u ∈ U(t).Each summand in (9) can be thought of as the improvement in the player’s valuefunction that he (or she) will receive by changing his (or her) strategy from ui tovi while all other players continue to play u−i. The Nikaidô-Isoda function thusrepresents the sum of these value function improvements. Note that its maximumfor a given u is always non-negative as a consequence of (10). Moreover, (9) is non-positive everywhere when either u or v is a Markovian equilibrium strategy, for in anequilibrium situation no player can unilaterally improve their payoff, and so in thiscase each summand is at most zero.From here, we conclude that when the Nikaidô-Isoda function (9) cannot be made(significantly) positive at each stage for a given u, we have (approximately) reachedthe Markovian equilibrium point. This observation is useful in constructing a termi-nation condition for our algorithm: choose an ε > 0 such that the equilibrium hasbeen reached to a sufficient degree of precision when maxv∈U Ψx(t)(us, v) < ε, whereus ∈ U is computed at the current iteration s.More formally, we shall compute a “normalised” Markovian equilibrium which willbe a Nash equilibrium under weak convex-concavity of Ψx(t)(u, v).Definition 2. We call an admissible strategy u∗ ∈ U a Nash normalised Markovianequilibrium ifmaxv∈UΨx(t)(u∗, v) = 0.A Nash normalised Markovian equilibrium u∗ is a unique feedback-Nash equilibriumif(1) Ψx(t)(u, v) is weakly convex-concave for every t, or4(2) ∑ni=1 F(t)i(x(t);u)is diagonally strictly concave for every t (see [KT06], wherethis feature was examined in a dynamic game).3. Numerical results through NISOCSolThis section should be read in conjunction with [AK06], where the calls to SOCSol4Lare explained. Here, we explain the differences that the user should be familiar within order to be able to call NISOCSol.3.1. State and time discretisation. These are specified as for SOCSol4L.3.2. Options. In this version of NISOCSol, it is assumed that each player has onlyone dimension of control. Consequently, the control dimension should be set to thenumber of players.3.3. User defined .m functions. The utility functions for NISOCSol return values forthe i-th player, where i is passed as an argument following those standard for SOCSol4L.If the game is defined as discrete-time with times 0, ∆, 2∆, . . . , T; where T is a multipleof ∆, then the following should also be observed:• TimeStep = ones(1, T∆)/ T∆;• The return value of the instantaneous cost function needs to be multplied byT∆.• The return value of the motion (or “delta”) function needs to be multplied byT∆.With these modifications a typical NISOCSol call has the following form.NISOCSol ( ‘ Del taFunct ionFi le ’ , ‘ Ins tantaneousCostFunct ionFi le ’ ,‘ Termina lS ta teFunct ionFi le ’ , StateLB , StateUB , Sta teStep ,TimeStep , ‘ ProblemFile ’ , Options , In i t i a lCon t ro lVa lue , A,b , Aeq , beq , ControlLB , ControlUB ,‘ UserConstra intFunct ionFi le ’ ) ;3.4. A test: a static equilibrium obtained as feedback-Nash. The static solution ofthe River Basin Pollution game has been cited in many publications (e.g., [Kra05b,KU00, KZ06]). The unique coupled-constraint equilibrium isx∗ = (21.14, 16.03, 2.73).We will use this problem to test NISOCSol.Presumably, with no investment cost and no depreciation, agents should move tothis equilibrium from any “initial condition” in one step. So we set the followingparameter values: T = 1, c3, i ≡ 0, ρ = 1, ki(·) ≡ φi(x). Furthermore, we assume thesystem dynamicsx(t+1)i = (1− µi)x(t)i + u(t)i .5As there is no depreciation, µi ≡ 0. We assume thatφi(x) = [d1 − d2(x1 + x2 + x3)]xi︸ ︷︷ ︸Revenue− (c1, i + c2, ixi)xi︸ ︷︷ ︸Cost.The economic constants d1 and d2 determine an inverse demand law. We assign themthe values 3 and 0.01 respectively. The values of the cost function coefficients c1, i andc2, i and the other model parameters are given in Table 1 below.Player (i) c1, i c2, i ei δi, 1 δi, 21 0.10 0.01 0.50 6.5 4.5832 0.12 0.05 0.25 5.0 6.2503 0.15 0.01 0.75 5.5 3.750Table 1. Constants for the River Basin Pollution game.The fourth column of Table 1 gives the emission parameters, while the fifth gives thepollution transportation and decay parameters that govern contamination levels atlocation ℓ = 1, 2. The local authority imposes the pollution constraintsqℓ(x) =3∑i=1δi,ℓeixi 6 Kℓ ℓ = 1, 2. (11)We take K1 = K2 = 100.Given this information and recalling the comments in Sections 3.1, 3.2 and 3.3, thefunctions Delta(u, x, t ), Costi(u, x, t , i ), Termi(x, i ) and Constraints(u, x, ts )were defined after [AK06].The programme’s output is the null strategy for x(0) = x∗ —see Figure 1, which showsstrategies at t = 0. Players’ actions u1, u2 and u3 are each plotted (from left to right)as functions of the third, second and first state variables. We see that the feedbackstrategies intersect with the horizontal axis (i.e., no action) for the initial condition x∗.The strategy realisations are the state and action time profiles shown in Figure 2. Thesolid lines (which are constant in time) demonstrate that no changes are made whengiven x(0) = x∗. The dash-dotted lines show that the equilibrium point is achievedfrom an arbitrary initial state (here, (22, 15, 4)).Lagrange multiplier rules have also been computed. To ensure that the standards (11)are obeyed, the government needs to inform the players what taxes would be collectedshould the pollution levels be exceeded. Under the feedback information pattern, thevalue of the Lagrange multipliers is a function of states (x(t)1 , x(t)2 , x(t)3 ) (here, t = 0).See Figure 3.60 5 10 15 20−505u10 5 10 15 20−505u20 5 10 15 20−505u3x3 x2x1Figure 1. One-step Markovian equilibrium strategies.2121.522x 1151617x 2234x 3−202u1−202u20 1−202u3TimeFigure 2. One-step Markovian equilibrium states and actions.70 5 10 15 200.30.40.50.60.70.80.9λx3 x2x1Figure 3. Lagrange multipliers at t = 0 as a function of state.4. Concluding remarksThis report outlines a set of useful machinery for the study dynamic games (possibly)subject to coupled-constraints (typical of environmental management problems). Inparticular, we have seen how the Nikaidô-Isoda function can be utilised to find acoupled-constraint equilibrium through an optimisation approach.References[AD54] K. J. Arrow and G. Debreu. Existence of an equilibrium for a competitive economy. Economet-rica, 22(3):265–290, 1954.[AK06] Jeffrey D. Azzato and Jacek B. Krawczyk. SOCSol4L: An improved MATLAB R© package forapproximating the solution to a continuous-time stochastic optimal control problem. Workingpaper, School of Ecnomics and Finance, Victoria University of Wellington, Dec 2006. Availableat http://mpra.ub.uni-muenchen.de/10015/ on 26/08/2008.[BO82] T. Bas¸ar and G. J. Olsder. Dynamic Noncooperative Game Theory. Academic Press, New York,1982.[CKK04] Javier Contreras, Matthias Klusch, and Jacek B. Krawczyk. Numerical solutions to Nash–Cournot equilibria in coupled constraint electricity markets. IEEE T. Power Syst., 19(1):195–206, Feb 2004.[HK97] Alain Haurie and Jacek B. Krawczyk. Optimal charges on river effluent fromlumped and distributed sources. Environ. Model. Assess., 2(3):177–189, Oct 1997. DOI:10.1023/A:1019049008557.8[HP07] Benjamin F. Hobbs and J.-S. Pang. Nash–Cournot equilibria in electric power markets withpiecewise linear demand functions and joint constraints. 55(1):113–127, Jan/Feb 2007. DOI:10.1287/opre.1060.0342.[Kra01] Jacek B. Krawczyk. A Markovian approximated solution to a portfolio management problem.ITEM., 1(1), 2001. Available at http://www.item.woiz.polsl.pl/issue/journal1.htm on26/08/2008.[Kra05a] J. B. Krawczyk. Numerical solutions to lump-sum pension problems that can yield left-skewed fund return distributions. In Christophe Deissenburg and Richard F. Hartl, editors,Optimal Control and Dynamic Games, number 7 in Advances in Computational ManagementScience, chapter 10, pages 155–176. Springer, New York, 2005.[Kra05b] Jacek B. Krawczyk. Coupled constraint Nash equilibria in environmental games. ResourceEnergy Econ., 27(2):157–181, Jun 2005. DOI: 10.1016/j.reseneeco.2004.08.001.[Kra06] Jacek B. Krawczyk. Coupled constraint Markovian equilibria in dynamic games of compli-ance. Seminar at the Kyoto University Institute of Economic Research, 30 Nov 2006.[KT06] J. B. Krawczyk and M. Tidball. A discrete-time dynamic game of seasonal water allocation. J.Optim. Theory Appl., 128(2):411–429, Feb 2006. DOI: 10.1007/s10957-006-9020-0.[KU00] Jacek B. Krawczyk and Stanislav Uryasev. Relaxation algorithms to find Nash equi-libria with economic applications. Environ. Model. Assess., 5(1):63–73, Jan 2000. DOI:10.1023/A:1019097208499.[KZ06] Jacek B. Krawczyk and James Zuccollo. NIRA-3 An improved MATLAB package for findingNash equilibria in infinite games. Working paper, School of Economics and Finance, VictoriaUniversity of Wellington, Dec 2006. Available at http://mpra.ub.uni-muenchen.de/1119/on 26/08/2008.[McK59] Lionel W. McKenzie. On the existence of general equilibrium for a competitive market. Econo-metrica, 27(1):54–71, Jan 1959.[NI55] Hukukane Nikaidô and Kazuo Isoda. Note on noncooperative games. Pac. J. Math., 5(1):807–815, 1955.[Ros65] J. B. Rosen. Existence and uniqueness of equilibrium points for concave n-person games.Econometrica, 33(3):520–534, Jul 1965.9

A report on NISOCSol: An algorithm for approximating Markovian equilibria in dynamic games with coupled-constraints

http://mpra.ub.uni-muenchen.de/1195/1/MPRA_paper_1195.pdf

A Report on NISOCSol: an Algorithm For Approximating Markovian Equlibria in Dynamic Games with Coupled-Constraints

Abstract

Similar works

Full text

Available Versions

Munich RePEc Personal Archive

Munich RePEc Personal Archive