990 research outputs found
Competitive Gradient Descent
We introduce a new algorithm for the numerical computation of Nash equilibria
of competitive two-player games. Our method is a natural generalization of
gradient descent to the two-player setting where the update is given by the
Nash equilibrium of a regularized bilinear local approximation of the
underlying game. It avoids oscillatory and divergent behaviors seen in
alternating gradient descent. Using numerical experiments and rigorous
analysis, we provide a detailed comparison to methods based on \emph{optimism}
and \emph{consensus} and show that our method avoids making any unnecessary
changes to the gradient dynamics while achieving exponential (local)
convergence for (locally) convex-concave zero sum games. Convergence and
stability properties of our method are robust to strong interactions between
the players, without adapting the stepsize, which is not the case with previous
methods. In our numerical experiments on non-convex-concave problems, existing
methods are prone to divergence and instability due to their sensitivity to
interactions among the players, whereas we never observe divergence of our
algorithm. The ability to choose larger stepsizes furthermore allows our
algorithm to achieve faster convergence, as measured by the number of model
evaluations.Comment: Appeared in NeurIPS 2019. This version corrects an error in theorem
2.2. Source code used for the numerical experiments can be found under
http://github.com/f-t-s/CGD. A high-level overview of this work can be found
under http://f-t-s.github.io/projects/cgd
Ergodic Mean Field Games with H\"ormander diffusions
We prove existence of solutions for a class of systems of subelliptic PDEs
arising from Mean Field Game systems with H\"ormander diffusion. These results
are motivated by the feedback synthesis Mean Field Game solutions and the Nash
equilibria of a large class of -player differential games
Competitive Policy Optimization
A core challenge in policy optimization in competitive Markov decision processes is the design of efficient optimization methods with desirable convergence and stability properties. To tackle this, we propose competitive policy optimization (CoPO), a novel policy gradient approach that exploits the game-theoretic nature of competitive games to derive policy updates. Motivated by the competitive gradient optimization method, we derive a bilinear approximation of the game objective. In contrast, off-the-shelf policy gradient methods utilize only linear approximations, and hence do not capture interactions among the players. We instantiate CoPO in two ways:(i) competitive policy gradient, and (ii) trust-region competitive policy optimization. We theoretically study these methods, and empirically investigate their behavior on a set of comprehensive, yet challenging, competitive games. We observe that they provide stable optimization, convergence to sophisticated strategies, and higher scores when played against baseline policy gradient methods
Competitive Policy Optimization
A core challenge in policy optimization in competitive Markov decision
processes is the design of efficient optimization methods with desirable
convergence and stability properties. To tackle this, we propose competitive
policy optimization (CoPO), a novel policy gradient approach that exploits the
game-theoretic nature of competitive games to derive policy updates. Motivated
by the competitive gradient optimization method, we derive a bilinear
approximation of the game objective. In contrast, off-the-shelf policy gradient
methods utilize only linear approximations, and hence do not capture
interactions among the players. We instantiate CoPO in two ways:(i) competitive
policy gradient, and (ii) trust-region competitive policy optimization. We
theoretically study these methods, and empirically investigate their behavior
on a set of comprehensive, yet challenging, competitive games. We observe that
they provide stable optimization, convergence to sophisticated strategies, and
higher scores when played against baseline policy gradient methods.Comment: 11 pages main paper, 6 pages references, and 31 pages appendix. 14
figure
- …