Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence

Learning agents that are not only capable of taking tests, but also innovating is becoming a hot topic in AI. One of the most promising paths towards this vision is multi-agent learning, where agents act as the environment for each other, and improving each agent means proposing new problems for others. However, existing evaluation platforms are either not compatible with multi-agent settings, or limited to a specific game. That is, there is not yet a general evaluation platform for research on multi-agent intelligence. To this end, we introduce Arena, a general evaluation platform for multi-agent intelligence with 35 games of diverse logics and representations. Furthermore, multi-agent intelligence is still at the stage where many problems remain unexplored. Therefore, we provide a building toolkit for researchers to easily invent and build novel multi-agent problems from the provided game set based on a GUI-configurable social tree and five basic multi-agent reward schemes. Finally, we provide Python implementations of five state-of-the-art deep multi-agent reinforcement learning baselines. Along with the baseline implementations, we release a set of 100 best agents/teams that we can train with different training schemes for each game, as the base for evaluating agents with population performance. As such, the research community can perform comparisons under a stable and uniform standard. All the implementations and accompanied tutorials have been open-sourced for the community at https://sites.google.com/view/arena-unity/

Evaluating Singleplayer and Multiplayer in Human Computation Games

Human computation games (HCGs) can provide novel solutions to intractable computational problems, help enable scientific breakthroughs, and provide datasets for artificial intelligence. However, our knowledge about how to design and deploy HCGs that appeal to players and solve problems effectively is incomplete. We present an investigatory HCG based on Super Mario Bros. We used this game in a human subjects study to investigate how different social conditions---singleplayer and multiplayer---and scoring mechanics---collaborative and competitive---affect players' subjective experiences, accuracy at the task, and the completion rate. In doing so, we demonstrate a novel design approach for HCGs, and discuss the benefits and tradeoffs of these mechanics in HCG design.Comment: 10 pages, 4 figures, 2 table

Putting theory oriented evaluation into practice

Evaluations of gaming simulations and business games as teaching devices are typically end-state driven. This emphasis fails to detect how the simulation being evaluated does or does not bring about its desired consequences. This paper advances the use of a logic model approach which possesses a holistic perspective that aims at including all elements associated with the situation created by a game. The use of the logic model approach is illustrated as applied to Simgame, a board game created for secondary school level business education in six European Union countries

Evaluating Visual Conversational Agents via Cooperative Human-AI Games

As AI continues to advance, human-AI teams are inevitable. However, progress in AI is routinely measured in isolation, without a human in the loop. It is crucial to benchmark progress in AI, not just in isolation, but also in terms of how it translates to helping humans perform certain tasks, i.e., the performance of human-AI teams. In this work, we design a cooperative game - GuessWhich - to measure human-AI team performance in the specific context of the AI being a visual conversational agent. GuessWhich involves live interaction between the human and the AI. The AI, which we call ALICE, is provided an image which is unseen by the human. Following a brief description of the image, the human questions ALICE about this secret image to identify it from a fixed pool of images. We measure performance of the human-ALICE team by the number of guesses it takes the human to correctly identify the secret image after a fixed number of dialog rounds with ALICE. We compare performance of the human-ALICE teams for two versions of ALICE. Our human studies suggest a counterintuitive trend - that while AI literature shows that one version outperforms the other when paired with an AI questioner bot, we find that this improvement in AI-AI performance does not translate to improved human-AI performance. This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.Comment: HCOMP 201

Online Dispute Resolution Through the Lens of Bargaining and Negotiation Theory: Toward an Integrated Model

[Excerpt] In this article we apply negotiation and bargaining theory to the analysis of online dispute resolution. Our principal objective is to develop testable hypotheses based on negotiation theory that can be used in ODR research. We have not conducted the research necessary to test the hypotheses we develop; however, in a later section of the article we suggest a possible methodology for doing so. There is a vast literature on negotiation and bargaining theory. For the purposes of this article, we realized at the outset that we could only use a small part of that literature in developing a model that might be suitable for empirical testing. We decided to use the behavioral theory of negotiation developed by Richard Walton and Robert McKersie, which was initially formulated in the 1960s. This theory has stood the test of time. Initially developed to explain union-management negotiations, it has proven useful in analyzing a wide variety of disputes and conflict situations. In constructing their theory, Walton and McKersie built on the contributions and work of many previous bargaining theorists including economists, sociologists, game theorists, and industrial relations scholars. In this article, we have incorporated a consideration of the foundations on which their theory was based. In the concluding section of the article we discuss briefly how other negotiation and bargaining theories might be applied to the analysis of ODR

The Present and Future of Game Theory

A broad nontechnical coverage of many of the developments in game theory since the 1950s is given together with some comments on important open problems and where some of the developments may take place. The nearly 90 references given serve only as a minimal guide to the many thousands of books and articles that have been written. The purpose here is to present a broad brush picture of the many areas of study and application that have come into being. The use of deep techniques flourishes best when it stays in touch with application. There is a vital symbiotic relationship between good theory and practice. The breakneck speed of development of game theory calls for an appreciation of both the many realities of conflict, coordination and cooperation and the abstract investigation of all of them.Game theory, Application and theory, Social sciences, Law, Experimental gaming, conflict, Coordination and cooperation