7,492 research outputs found
Making the Best of Limited Memory in Multi-Player Discounted Sum Games
In this paper, we establish the existence of optimal bounded memory strategy
profiles in multi-player discounted sum games. We introduce a non-deterministic
approach to compute optimal strategy profiles with bounded memory. Our approach
can be used to obtain optimal rewards in a setting where a powerful player
selects the strategies of all players for Nash and leader equilibria, where in
leader equilibria the Nash condition is waived for the strategy of this
powerful player. The resulting strategy profiles are optimal for this player
among all strategy profiles that respect the given memory bound, and the
related decision problem is NP-complete. We also provide simple examples, which
show that having more memory will improve the optimal strategy profile, and
that sufficient memory to obtain optimal strategy profiles cannot be inferred
from the structure of the game.Comment: In Proceedings GandALF 2015, arXiv:1509.0685
Learning with Opponent-Learning Awareness
Multi-agent settings are quickly gathering importance in machine learning.
This includes a plethora of recent work on deep multi-agent reinforcement
learning, but also can be extended to hierarchical RL, generative adversarial
networks and decentralised optimisation. In all these settings the presence of
multiple learning agents renders the training problem non-stationary and often
leads to unstable training or undesired final results. We present Learning with
Opponent-Learning Awareness (LOLA), a method in which each agent shapes the
anticipated learning of the other agents in the environment. The LOLA learning
rule includes a term that accounts for the impact of one agent's policy on the
anticipated parameter update of the other agents. Results show that the
encounter of two LOLA agents leads to the emergence of tit-for-tat and
therefore cooperation in the iterated prisoners' dilemma, while independent
learning does not. In this domain, LOLA also receives higher payouts compared
to a naive learner, and is robust against exploitation by higher order
gradient-based methods. Applied to repeated matching pennies, LOLA agents
converge to the Nash equilibrium. In a round robin tournament we show that LOLA
agents successfully shape the learning of a range of multi-agent learning
algorithms from literature, resulting in the highest average returns on the
IPD. We also show that the LOLA update rule can be efficiently calculated using
an extension of the policy gradient estimator, making the method suitable for
model-free RL. The method thus scales to large parameter and input spaces and
nonlinear function approximators. We apply LOLA to a grid world task with an
embedded social dilemma using recurrent policies and opponent modelling. By
explicitly considering the learning of the other agent, LOLA agents learn to
cooperate out of self-interest. The code is at github.com/alshedivat/lola
On Partially Controlled Multi-Agent Systems
Motivated by the control theoretic distinction between controllable and
uncontrollable events, we distinguish between two types of agents within a
multi-agent system: controllable agents, which are directly controlled by the
system's designer, and uncontrollable agents, which are not under the
designer's direct control. We refer to such systems as partially controlled
multi-agent systems, and we investigate how one might influence the behavior of
the uncontrolled agents through appropriate design of the controlled agents. In
particular, we wish to understand which problems are naturally described in
these terms, what methods can be applied to influence the uncontrollable
agents, the effectiveness of such methods, and whether similar methods work
across different domains. Using a game-theoretic framework, this paper studies
the design of partially controlled multi-agent systems in two contexts: in one
context, the uncontrollable agents are expected utility maximizers, while in
the other they are reinforcement learners. We suggest different techniques for
controlling agents' behavior in each domain, assess their success, and examine
their relationship.Comment: See http://www.jair.org/ for any accompanying file
Average-energy games
Two-player quantitative zero-sum games provide a natural framework to
synthesize controllers with performance guarantees for reactive systems within
an uncontrollable environment. Classical settings include mean-payoff games,
where the objective is to optimize the long-run average gain per action, and
energy games, where the system has to avoid running out of energy.
We study average-energy games, where the goal is to optimize the long-run
average of the accumulated energy. We show that this objective arises naturally
in several applications, and that it yields interesting connections with
previous concepts in the literature. We prove that deciding the winner in such
games is in NP inter coNP and at least as hard as solving mean-payoff games,
and we establish that memoryless strategies suffice to win. We also consider
the case where the system has to minimize the average-energy while maintaining
the accumulated energy within predefined bounds at all times: this corresponds
to operating with a finite-capacity storage for energy. We give results for
one-player and two-player games, and establish complexity bounds and memory
requirements.Comment: In Proceedings GandALF 2015, arXiv:1509.0685
Non-Zero Sum Games for Reactive Synthesis
In this invited contribution, we summarize new solution concepts useful for
the synthesis of reactive systems that we have introduced in several recent
publications. These solution concepts are developed in the context of non-zero
sum games played on graphs. They are part of the contributions obtained in the
inVEST project funded by the European Research Council.Comment: LATA'16 invited pape
The Adversarial Stackelberg Value in Quantitative Games
In this paper, we study the notion of adversarial Stackelberg value for
two-player non-zero sum games played on bi-weighted graphs with the mean-payoff
and the discounted sum functions. The adversarial Stackelberg value of Player 0
is the largest value that Player 0 can obtain when announcing her strategy to
Player 1 which in turn responds with any of his best response. For the
mean-payoff function, we show that the adversarial Stackelberg value is not
always achievable but epsilon-optimal strategies exist. We show how to compute
this value and prove that the associated threshold problem is in NP. For the
discounted sum payoff function, we draw a link with the target discounted sum
problem which explains why the problem is difficult to solve for this payoff
function. We also provide solutions to related gap problems.Comment: long version of an ICALP'20 pape
The Adversarial Stackelberg Value in Quantitative Games
In this paper, we study the notion of adversarial Stackelberg value for two-player non-zero sum games played on bi-weighted graphs with the mean-payoff and the discounted sum functions. The adversarial Stackelberg value of Player 0 is the largest value that Player 0 can obtain when announcing her strategy to Player 1 which in turn responds with any of his best response. For the mean-payoff function, we show that the adversarial Stackelberg value is not always achievable but ?-optimal strategies exist. We show how to compute this value and prove that the associated threshold problem is in NP. For the discounted sum payoff function, we draw a link with the target discounted sum problem which explains why the problem is difficult to solve for this payoff function. We also provide solutions to related gap problems
- …