Inverse Factorized Q-Learning for Cooperative Multi-agent Imitation Learning

This paper concerns imitation learning (IL) (i.e, the problem of learning to mimic expert behaviors from demonstrations) in cooperative multi-agent systems. The learning problem under consideration poses several challenges, characterized by high-dimensional state and action spaces and intricate inter-agent dependencies. In a single-agent setting, IL has proven to be done efficiently through an inverse soft-Q learning process given expert demonstrations. However, extending this framework to a multi-agent context introduces the need to simultaneously learn both local value functions to capture local observations and individual actions, and a joint value function for exploiting centralized learning. In this work, we introduce a novel multi-agent IL algorithm designed to address these challenges. Our approach enables the centralized learning by leveraging mixing networks to aggregate decentralized Q functions. A main advantage of this approach is that the weights of the mixing networks can be trained using information derived from global states. We further establish conditions for the mixing networks under which the multi-agent objective function exhibits convexity within the Q function space. We present extensive experiments conducted on some challenging competitive and cooperative multi-agent game environments, including an advanced version of the Star-Craft multi-agent challenge (i.e., SMACv2), which demonstrates the effectiveness of our proposed algorithm compared to existing state-of-the-art multi-agent IL algorithms

A Survey on Causal Reinforcement Learning

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR). Moreover, we summarize the evaluation matrices and open sources while we discuss emerging applications, along with promising prospects for the future development of CRL.Comment: 29 pages, 20 figure

Enabling imitation-based cooperation in dynamic social networks

The emergence of cooperation among self-interested agents has been a key concern of the multi-agent systems community for decades. With the increased importance of network-mediated interaction, researchers have shifted the attention to the impact of social networks and their dynamics in promoting or hindering cooperation, drawing various context-dependent conclusions. For example, some lines of research, theoretical and experimental, suggest the existence of a threshold effect in the ratio of timescales of network evolution, after which cooperation will emerge, whereas other lines dispute this, suggesting instead a Goldilocks zone. In this paper we provide an evolutionary game theory framework to understand coevolutionary processes from a bottom up perspective - in particular the emergence of a cooperator-core and defector-periphery - clarifying the impact of partner selection and imitation strategies in promoting cooperative behaviour, without assuming underlying communication or reputation mechanisms. In doing so we provide a unifying framework to study imitation-based cooperation in dynamic social networks and show that disputes in the literature can in fact coexist in so far as the results stem from different equally valid assumptions

Law and Culture in China and Japan: A Framework for Analysis

This Comment is divided into two parts. The first sets forth a series of definitional propositions intended for a more general analysis of the interrelationships of law and culture. The second comprises an introduction to the evolution of legal institutions that enables us to understand better the reception and development of Western legal institutions in East Asia and provides context for the four articles and their individual and collective insights

Technological development, strategic behavior and government policy in information technology industries

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Political Science, 1989.Includes bibliographical references.by Charles H. Ferguson.Ph.D