Nash Equilibria in Multi-Agent Motor Interactions

Social interactions in classic cognitive games like the ultimatum game or the prisoner's dilemma typically lead to Nash equilibria when multiple competitive decision makers with perfect knowledge select optimal strategies. However, in evolutionary game theory it has been shown that Nash equilibria can also arise as attractors in dynamical systems that can describe, for example, the population dynamics of microorganisms. Similar to such evolutionary dynamics, we find that Nash equilibria arise naturally in motor interactions in which players vie for control and try to minimize effort. When confronted with sensorimotor interaction tasks that correspond to the classical prisoner's dilemma and the rope-pulling game, two-player motor interactions led predominantly to Nash solutions. In contrast, when a single player took both roles, playing the sensorimotor game bimanually, cooperative solutions were found. Our methodology opens up a new avenue for the study of human motor interactions within a game theoretic framework, suggesting that the coupling of motor systems can lead to game theoretic solutions

Motor coordination: when two have to act as one

Trying to pass someone walking toward you in a narrow corridor is a familiar example of a two-person motor game that requires coordination. In this study, we investigate coordination in sensorimotor tasks that correspond to classic coordination games with multiple Nash equilibria, such as “choosing sides,” “stag hunt,” “chicken,” and “battle of sexes”. In these tasks, subjects made reaching movements reflecting their continuously evolving “decisions” while they received a continuous payoff in the form of a resistive force counteracting their movements. Successful coordination required two subjects to “choose” the same Nash equilibrium in this force-payoff landscape within a single reach. We found that on the majority of trials coordination was achieved. Compared to the proportion of trials in which miscoordination occurred, successful coordination was characterized by several distinct features: an increased mutual information between the players’ movement endpoints, an increased joint entropy during the movements, and by differences in the timing of the players’ responses. Moreover, we found that the probability of successful coordination depends on the players’ initial distance from the Nash equilibria. Our results suggest that two-person coordination arises naturally in motor interactions and is facilitated by favorable initial positions, stereotypical motor pattern, and differences in response times

Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments

The Game Theory & Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.Comment: Published in AI Communications 202

Development of collaborative strategies in joint action

Many tasks in daily life involve coordinating movements between two or more individuals. A couple of dancers, a team of players, two workers carrying a load or a therapist interacting with a patient are just a few examples. Acting in collaboration or joint action is a crucial human ability, and our sensorimotor system is shaped to support this capability efficiently. When two partners have different goals but may benefit from collaborating, they face the challenge of negotiating a joint strategy. To do this, first and foremost both subjects need to know their partner\u2019s state and current strategy. It is unclear how the collaboration would be affected if information about the partner is unreliable or incomplete. This work intends to investigate the development of collaborative strategies in joint action. To this purpose, I developed a dedicated experimental apparatus and task. I also developed a general computational framework \u2013 based on differential game theory \u2013 for the description and implementation of interactive behaviours of two subjects performing a joint motor task. The model allows to simulate any joint sensorimotor action in which the joint dynamics can be represented as a linear dynamical system and each agent\u2019s task is formulated in terms of a quadratic cost functional. The model also accounts for imperfect information about dyad dynamics and partner\u2019s actions, and can predict the development of joint action through repeated performance. A first experimental study, focused on how the development of joint action is affected by incomplete and unreliable information. We found that information about the partner not only affects the speed at which a collaborative strategy is achieved (less information, slower learning) but also optimality of the collaboration. In particular, when information about the partner is reduced, the learned strategy is characterised by the development of alternating patterns of leader-follower roles, whereas greater information leads to a more synchronous behaviour. Simulations with a computational model based on game theory suggest that synchronous behaviours are close to optimal in a game theoretic sense (Nash equilibrium). The emergence of roles is a compensation strategy which minimises the need to estimate partner\u2019s intentions and is, therefore, more robust to incomplete information. A second study addresses how physical interaction develops between adults with Autism spectrum disorder (ASD) and typically developing subjects. ASD remains mostly a mystery and has therefore generated some theories trying to explain their cognitive disabilities, which involve an impaired ability to interact with other human partners. Although preliminary due to the small number of subjects, our results suggest that ASD subjects display heterogeneity in establishing a collaboration, which can be only partly explained with their ability to perceive haptic force. This work is a first attempt to establish a sensorimotor theory of joint action. It may provide new insights into the development of robots that are capable of establishing optimal collaborations with human partners, for instance in the context of robot-assisted rehabilitation

A framework of human–robot coordination based on game theory and policy iteration

In this paper, we propose a framework to analyze the interactive behaviors of human and robot in physical interactions. Game theory is employed to describe the system under study, and policy iteration is adopted to provide a solution of Nash equilibrium. The human’s control objective is estimated based on the measured interaction force, and it is used to adapt the robot’s objective such that human-robot coordination can be achieved. The validity of the proposed method is verified through a rigorous proof and experimental studies