417,205 research outputs found
Learning Sparse Graphon Mean Field Games
Although the field of multi-agent reinforcement learning (MARL) has made
considerable progress in the last years, solving systems with a large number of
agents remains a hard challenge. Graphon mean field games (GMFGs) enable the
scalable analysis of MARL problems that are otherwise intractable. By the
mathematical structure of graphons, this approach is limited to dense graphs
which are insufficient to describe many real-world networks such as power law
graphs. Our paper introduces a novel formulation of GMFGs, called LPGMFGs,
which leverages the graph theoretical concept of graphons and provides a
machine learning tool to efficiently and accurately approximate solutions for
sparse network problems. This especially includes power law networks which are
empirically observed in various application areas and cannot be captured by
standard graphons. We derive theoretical existence and convergence guarantees
and give empirical examples that demonstrate the accuracy of our learning
approach for systems with many agents. Furthermore, we extend the Online Mirror
Descent (OMD) learning algorithm to our setup to accelerate learning speed,
empirically show its capabilities, and conduct a theoretical analysis using the
novel concept of smoothed step graphons. In general, we provide a scalable,
mathematically well-founded machine learning approach to a large class of
otherwise intractable problems of great relevance in numerous research fields.Comment: accepted for publication at the International Conference on
Artificial Intelligence and Statistics (AISTATS) 2023; code available at:
https://github.com/ChrFabian/Learning_sparse_GMFG
Fitted Q-Learning in Mean-field Games
In the literature, existence of equilibria for discrete-time mean field games
has been in general established via Kakutani's Fixed Point Theorem. However,
this fixed point theorem does not entail any iterative scheme for computing
equilibria. In this paper, we first propose a Q-iteration algorithm to compute
equilibria for mean-field games with known model using Banach Fixed Point
Theorem. Then, we generalize this algorithm to model-free setting using fitted
Q-iteration algorithm and establish the probabilistic convergence of the
proposed iteration. Then, using the output of this learning algorithm, we
construct an approximate Nash equilibrium for finite-agent stochastic game with
mean-field interaction between agents.Comment: 22 page
A General Framework for Learning Mean-Field Games
This paper presents a general mean-field game (GMFG) framework for
simultaneous learning and decision-making in stochastic games with a large
population. It first establishes the existence of a unique Nash Equilibrium to
this GMFG, and demonstrates that naively combining reinforcement learning with
the fixed-point approach in classical MFGs yields unstable algorithms. It then
proposes value-based and policy-based reinforcement learning algorithms (GMF-V
and GMF-P, respectively) with smoothed policies, with analysis of their
convergence properties and computational complexities. Experiments on an
equilibrium product pricing problem demonstrate that GMF-V-Q and GMF-P-TRPO,
two specific instantiations of GMF-V and GMF-P, respectively, with Q-learning
and TRPO, are both efficient and robust in the GMFG setting. Moreover, their
performance is superior in convergence speed, accuracy, and stability when
compared with existing algorithms for multi-agent reinforcement learning in the
-player setting.Comment: 43 pages, 7 figures. arXiv admin note: substantial text overlap with
arXiv:1901.0958
Reinforcement Learning Algorithm for Mixed Mean Field Control Games
We present a new combined \textit{mean field control game} (MFCG) problem
which can be interpreted as a competitive game between collaborating groups and
its solution as a Nash equilibrium between groups. Players coordinate their
strategies within each group. An example is a modification of the classical
trader's problem. Groups of traders maximize their wealth. They face cost for
their transactions, for their own terminal positions, and for the average
holding within their group. The asset price is impacted by the trades of all
agents. We propose a three-timescale reinforcement learning algorithm to
approximate the solution of such MFCG problems. We test the algorithm on
benchmark linear-quadratic specifications for which we provide analytic
solutions
- …