Approachability in Stackelberg Stochastic Games with Vector Costs

The notion of approachability was introduced by Blackwell [1] in the context of vector-valued repeated games. The famous Blackwell's approachability theorem prescribes a strategy for approachability, i.e., for `steering' the average cost of a given agent towards a given target set, irrespective of the strategies of the other agents. In this paper, motivated by the multi-objective optimization/decision making problems in dynamically changing environments, we address the approachability problem in Stackelberg stochastic games with vector valued cost functions. We make two main contributions. Firstly, we give a simple and computationally tractable strategy for approachability for Stackelberg stochastic games along the lines of Blackwell's. Secondly, we give a reinforcement learning algorithm for learning the approachable strategy when the transition kernel is unknown. We also recover as a by-product Blackwell's necessary and sufficient condition for approachability for convex sets in this set up and thus a complete characterization. We also give sufficient conditions for non-convex sets.Comment: 18 Pages, Submitted to Dynamic Games and Application

Approachability in Stackelberg Stochastic Games with Vector Costs

The notion of approachability was introduced by Blackwell [1] in the context of vector-valued repeated games. The famous Blackwell's approachability theorem prescribes a strategy for approachability, i.e., for `steering' the average cost of a given agent towards a given target set, irrespective of the strategies of the other agents. In this paper, motivated by the multi-objective optimization/decision making problems in dynamically changing environments, we address the approachability problem in Stackelberg stochastic games with vector valued cost functions. We make two main contributions. Firstly, we give a simple and computationally tractable strategy for approachability for Stackelberg stochastic games along the lines of Blackwell's. Secondly, we give a reinforcement learning algorithm for learning the approachable strategy when the transition kernel is unknown. We also recover as a by-product Blackwell's necessary and sufficient condition for approachability for convex sets in this set up and thus a complete characterization. We also give sufficient conditions for non-convex sets.Comment: 18 Pages, Submitted to Dynamic Games and Application

On an unified framework for approachability in games with or without signals

We unify standard frameworks for approachability both in full or partial monitoring by defining a new abstract game, called the "purely informative game", where the outcome at each stage is the maximal information players can obtain, represented as some probability measure. Objectives of players can be rewritten as the convergence (to some given set) of sequences of averages of these probability measures. We obtain new results extending the approachability theory developed by Blackwell moreover this new abstract framework enables us to characterize approachable sets with, as usual, a remarkably simple and clear reformulation for convex sets. Translated into the original games, those results become the first necessary and sufficient condition under which an arbitrary set is approachable and they cover and extend previous known results for convex sets. We also investigate a specific class of games where, thanks to some unusual definition of averages and convexity, we again obtain a complete characterization of approachable sets along with rates of convergence

Robust approachability and regret minimization in games with partial monitoring

Approachability has become a standard tool in analyzing earning algorithms in the adversarial online learning setup. We develop a variant of approachability for games where there is ambiguity in the obtained reward that belongs to a set, rather than being a single vector. Using this variant we tackle the problem of approachability in games with partial monitoring and develop simple and efficient algorithms (i.e., with constant per-step complexity) for this setup. We finally consider external regret and internal regret in repeated games with partial monitoring and derive regret-minimizing strategies based on approachability theory

Approachability of Convex Sets in Games with Partial Monitoring

We provide a necessary and sufficient condition under which a convex set is approachable in a game with partial monitoring, i.e.\ where players do not observe their opponents' moves but receive random signals. This condition is an extension of Blackwell's Criterion in the full monitoring framework, where players observe at least their payoffs. When our condition is fulfilled, we construct explicitly an approachability strategy, derived from a strategy satisfying some internal consistency property in an auxiliary game. We also provide an example of a convex set, that is neither (weakly)-approachable nor (weakly)-excludable, a situation that cannot occur in the full monitoring case. We finally apply our result to describe an $\epsilon$-optimal strategy of the uninformed player in a zero-sum repeated game with incomplete information on one side

Attainability in Repeated Games with Vector Payoffs

We introduce the concept of attainable sets of payoffs in two-player repeated games with vector payoffs. A set of payoff vectors is called {\em attainable} if player 1 can ensure that there is a finite horizon $T$ such that after time $T$ the distance between the set and the cumulative payoff is arbitrarily small, regardless of what strategy player 2 is using. This paper focuses on the case where the attainable set consists of one payoff vector. In this case the vector is called an attainable vector. We study properties of the set of attainable vectors, and characterize when a specific vector is attainable and when every vector is attainable.Comment: 28 pages, 2 figures, conference version at NetGCoop 201

Strong and safe Nash equilibrium in some repeated 3-player games

We consider a 3-player game in the normal form, in which each player has two actions. We assume that the game is symmetric and repeated infinitely many times. At each stage players make their choices knowing only the average payoffs from previous stages of all the players. A strategy of a player in the repeated game is a function defined on the convex hull of the set of payoffs. Our aim is to construct a strong Nash equilibrium in the repeated game, i.e. a strategy profile being resistant to deviations by coalitions. Constructed equilibrium strategies are safe, i.e. the non-deviating player payoff is not smaller than the equilibrium payoff in the stage game, and deviating players' payoffs do not exceed the non-deviating player payoff more than a positive constant which can be arbitrary small and chosen by the non-deviating player. Our construction is inspired by Smale's good strategies described in \cite{smale}, where the repeated Prisoner's Dilemma was considered. In proofs we use arguments based on approachability and strong approachability type results.Comment: 19 page