Finite-Step Algorithms for Single-Controller and Perfect Information Stochastic Games

Abstract. After a brief survey of iterative algorithms for general stochas-tic games, we concentrate on finite-step algorithms for two special classes of stochastic games. They are Single-Controller Stochastic Games and Per-fect Information Stochastic Games. In the case of single-controller games, the transition probabilities depend on the actions of the same player in all states. In perfect information stochastic games, one of the players has exactly one action in each state. Single-controller zero-sum games are effi-ciently solved by linear programming. Non-zero-sum single-controller stochastic games are reducible to linear complementary problems (LCP). In the discounted case they can be modified to fit into the so-called LCPs of Eave’s class L. In the undiscounted case the LCP’s are reducible to Lemke’s copositive plus class. In either case Lemke’s algorithm can be used to find a Nash equilibrium. In the case of discounted zero-sum perfect informa-tion stochastic games, a policy improvement algorithm is presented. Many other classes of stochastic games with orderfield property still await efficient finite-step algorithms. 1

Nonzero-sum Stochastic Games

This paper treats of stochastic games. We focus on nonzero-sum games and provide a detailed survey of selected recent results. In Section 1, we consider stochastic Markov games. A correlation of strategies of the players, involving ``public signals'', is described, and a correlated equilibrium theorem proved recently by Nowak and Raghavan for discounted stochastic games with general state space is presented. We also report an extension of this result to a class of undiscounted stochastic games, satisfying some uniform ergodicity condition. Stopping games are related to stochastic Markov games. In Section 2, we describe a version of Dynkin's game related to observation of a Markov process with random assignment mechanism of states to the players. Some recent contributions of the second author in this area are reported. The paper also contains a brief overview of the theory of nonzero-sum stochastic games and stopping games which is very far from being complete

Pure Equilibrium Strategies for Stochastic Games via Potential Functions

Strategic games with a potential function have quite often equilibria in pure strategies (Monderer and Shapley [4]). This is also true for stochastic games but the existence of a potential function is mostly hard to prove. For some classes of stochastic games with an additional structure, an equilibrium can be found by solving one or a finite number of finite strategic games.We call these games auxiliary games. In this paper, we investigate if we can derive the existence of equilibria in pure stationary strategies from the fact that the auxiliary games allow for a potential function. We will do this for zero-sum, two-person discounted stochastic games and non-zero-sum discounted stochastic games with additive reward functions and additive transitions (Raghavan et al. [8]) or with separable rewards and state independent transitions (Parthasarathy et al. [5])