Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions

Generative Adversarial Networks (GANs) is a novel class of deep generative models which has recently gained significant attention. GANs learns complex and high-dimensional distributions implicitly over images, audio, and data. However, there exists major challenges in training of GANs, i.e., mode collapse, non-convergence and instability, due to inappropriate design of network architecture, use of objective function and selection of optimization algorithm. Recently, to address these challenges, several solutions for better design and optimization of GANs have been investigated based on techniques of re-engineered network architectures, new objective functions and alternative optimization algorithms. To the best of our knowledge, there is no existing survey that has particularly focused on broad and systematic developments of these solutions. In this study, we perform a comprehensive survey of the advancements in GANs design and optimization solutions proposed to handle GANs challenges. We first identify key research issues within each design and optimization technique and then propose a new taxonomy to structure solutions by key research issues. In accordance with the taxonomy, we provide a detailed discussion on different GANs variants proposed within each solution and their relationships. Finally, based on the insights gained, we present the promising research directions in this rapidly growing field.Comment: 42 pages, Figure 13, Table

Modeling Lane-Changing Behavior in a Connected Environment: A Game Theory Approach

AbstractVehicle-to-Vehicle communications provide the opportunity to create an internet of cars through the recent advances in communication technologies, processing power, and sensing technologies. Aconnected vehicle receives real-time information from surrounding vehicles; such information can improve drivers’ awareness about their surrounding traffic condition and lead to safer and more efficient driving maneuvers. Lane-changing behavior,as one of the most challenging driving maneuvers to understand and to predict, and a major source of congestion and collisions, can benefit from this additional information.This paper presents a lane-changing model based on a game-theoretical approach that endogenously accounts for the flow of information in a connected vehicular environment.A calibration approach based on the method of simulated moments is presented and a simplified version of the proposed framework is calibrated against NGSIM data. The prediction capability of the simplified model is validated. It is concluded the presented framework is capable of predicting lane-changing behavior with limitations that still need to be addressed.Finally, a simulation framework based on the fictitious play is proposed. The simulation results revealed that the presented lane-changing model provides a greater level of realism than a basic gap-acceptance model

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Multi-agent Reinforcement Learning as Applied to Autonomous Systems

Multi-agent reinforcement learning (MARL) is a relatively unexplored area. Existing MARL methods and algorithms often scale poorly to the number of agents or suffer from the issue of non-stationary learning. This thesis aims to develop distributed training methods and algorithms for RL of the multi-agent autonomous systems that ensure scalability and stabilized learning.Specifically, we consider three common paradigms of multi-agent interaction: cooperative/ team, strategic/competitive, and leader-follower settings. We first study game-theoretic RL for the last two settings. Game-theoretic models are inherently distributed, and each agent takes other agents’ responses into account in the resulting game-theoretical equilibrium. Therefore, learning based on the game-theoretic equilibrium can effectively address the issue of non-stationary learning. We investigate Nash Q-learning in the strategic/competitive setting and Stackelberg Q-learning in the leader-follower setting, and apply them successfully to several applications that involve simple yet basic multi-agent interactions. We then propose a distributed deep Q-leaning for the cooperative/team setting, where each agent updates the estimate of her Q-value based on her own reward and her neighbors’ Q-values. We analyze the convergence of the proposed algorithm, characterize its performance gap to the centralized Q-learning, and evaluate it with a cooperative multi-agent navigation task

Coordination problems on networks revisited: statics and dynamics

Simple binary-state coordination models are widely used to study collective socio-economic phenomena such as the spread of innovations or the adoption of products on social networks. The common trait of these systems is the occurrence of large-scale coordination events taking place abruptly, in the form of a cascade process, as a consequence of small perturbations of an apparently stable state. The conditions for the occurrence of cascade instabilities have been largely analysed in the literature, however for the same coordination models no sufficient attention was given to the relation between structural properties of (Nash) equilibria and possible outcomes of dynamical equilibrium selection. Using methods from the statistical physics of disordered systems, the present work investigates both analytically and numerically, the statistical properties of such Nash equilibria on networks, focusing mostly on random graphs. We provide an accurate description of these properties, which is then exploited to shed light on the mechanisms behind the onset of coordination/miscoordination on large networks. This is done studying the most common processes of dynamical equilibrium selection, such as best response, bounded-rational dynamics and learning processes. In particular, we show that well beyond the instability region, full coordination is still globally stochastically stable, however equilibrium selection processes with low stochasticity (e.g. best response) or strong memory effects (e.g. reinforcement learning) can be prevented from achieving full coordination by being trapped into a large (exponentially in number of agents) set of locally stable Nash equilibria at low/medium coordination (inefficient equilibria). These results should be useful to allow a better understanding of general coordination problems on complex networks.Comment: Revtex style, 56 pages, 21 figure