Optimization frameworks and sensitivity analysis of Stackelberg mean-field games

This paper proposes and studies a class of discrete-time finite-time-horizon Stackelberg mean-field games, with one leader and an infinite number of identical and indistinguishable followers. In this game, the objective of the leader is to maximize her reward considering the worst-case cost over all possible $\epsilon$-Nash equilibria among followers. A new analytical paradigm is established by showing the equivalence between this Stackelberg mean-field game and a minimax optimization problem. This optimization framework facilitates studying both analytically and numerically the set of Nash equilibria for the game; and leads to the sensitivity and the robustness analysis of the game value. In particular, when there is model uncertainty, the game value for the leader suffers non-vanishing sub-optimality as the perturbed model converges to the true model. In order to obtain a near-optimal solution, the leader needs to be more pessimistic with anticipation of model errors and adopts a relaxed version of the original Stackelberg game

Linear Quadratic Reinforcement Learning: Sublinear Regret in the Episodic Continuous-Time Framework

In this paper we study a continuous-time linear quadratic reinforcement learning problem in an episodic setting. We first show that na\"ive discretization and piecewise approximation with discrete-time RL algorithms yields a linear regret with respect to the number of learning episodes $N$. We then propose an algorithm with continuous-time controls based on a regularized least-squares estimation, and establish a sublinear regret bound in the order of $\tilde{O}(\sqrt{N})$. The analysis consists of two parts: parameter estimation error, which relies on properties of sub-exponential random variables and double stochastic integrals; and perturbation analysis, which establishes the robustness of the associated continuous-time Riccati equation by exploiting its regularity property.Comment: 25 page

A General Framework for Learning Mean-Field Games

This paper presents a general mean-field game (GMFG) framework for simultaneous learning and decision-making in stochastic games with a large population. It first establishes the existence of a unique Nash Equilibrium to this GMFG, and demonstrates that naively combining reinforcement learning with the fixed-point approach in classical MFGs yields unstable algorithms. It then proposes value-based and policy-based reinforcement learning algorithms (GMF-V and GMF-P, respectively) with smoothed policies, with analysis of their convergence properties and computational complexities. Experiments on an equilibrium product pricing problem demonstrate that GMF-V-Q and GMF-P-TRPO, two specific instantiations of GMF-V and GMF-P, respectively, with Q-learning and TRPO, are both efficient and robust in the GMFG setting. Moreover, their performance is superior in convergence speed, accuracy, and stability when compared with existing algorithms for multi-agent reinforcement learning in the $N$-player setting.Comment: 43 pages, 7 figures. arXiv admin note: substantial text overlap with arXiv:1901.0958

MFGLib: A Library for Mean-Field Games

Mean-field games (MFGs) are limiting models to approximate $N$-player games, with a number of applications. Despite the ever-growing numerical literature on computation of MFGs, there is no library that allows researchers and practitioners to easily create and solve their own MFG problems. The purpose of this document is to introduce MFGLib, an open-source Python library for solving general MFGs with a user-friendly and customizable interface. It serves as a handy tool for creating and analyzing generic MFG environments, along with embedded auto-tuners for all implemented algorithms. The package is distributed under the MIT license and the source code and documentation can be found at https://github.com/radar-research-lab/MFGLib/

Preparation and Mechanical Properties of Continuous Carbon Nanotube Networks Modified C f

Continuous carbon nanotube (CNT) networks were formed in Cf/SiC composites via freeze-drying method. Composites were fabricated by precursor infiltration and pyrolysis (PIP) process afterwards. The different distribution morphologies of CNTs in the preforms originating from the different CNT contents were analyzed while the influence of the distribution of CNTs was discussed in detail. Compared to composites without CNTs, the interfacial shear strength (ILSS) and the flexural strength of Cf/1%CNTs/SiC were increased by 31% and 27%, respectively, but the values of Cf/2.5%CNTs/SiC decreased as a result of lots of defects caused by excess CNTs. With the analysis of ILSS, the flexural strengths, and the fracture morphologies, CNTs effectively improved the weak interfacial strength between T700SC carbon fibers and SiC matrix

Chromosome-Level Genome Assembly for Acer pseudosieboldianum and Highlights to Mechanisms for Leaf Color and Shape Change

Acer pseudosieboldianum (Pax) Komarov is an ornamental plant with prominent potential and is naturally distributed in Northeast China. Here, we obtained a chromosome-scale genome assembly of A. pseudosieboldianum combining HiFi and Hi-C data, and the final assembled genome size was 690.24 Mb and consisted of 287 contigs, with a contig N50 value of 5.7 Mb and a BUSCO complete gene percentage of 98.4%. Genome evolution analysis showed that an ancient duplication occurred in A. pseudosieboldianum. Phylogenetic analyses revealed that Aceraceae family could be incorporated into Sapindaceae, consistent with the present Angiosperm Phylogeny Group system. We further construct a gene-to-metabolite correlation network and identified key genes and metabolites that might be involved in anthocyanin biosynthesis pathways during leaf color change. Additionally, we identified crucial teosinte branched1, cycloidea, and proliferating cell factors (TCP) transcription factors that might be involved in leaf morphology regulation of A. pseudosieboldianum, Acer yangbiense and Acer truncatum. Overall, this reference genome is a valuable resource for evolutionary history studies of A. pseudosieboldianum and lays a fundamental foundation for its molecular breeding