Making friends on the fly : advances in ad hoc teamwork

textGiven the continuing improvements in design and manufacturing processes in addition to improvements in artificial intelligence, robots are being deployed in an increasing variety of environments for longer periods of time. As the number of robots grows, it is expected that they will encounter and interact with other robots. Additionally, the number of companies and research laboratories producing these robots is increasing, leading to the situation where these robots may not share a common communication or coordination protocol. While standards for coordination and communication may be created, we expect that any standards will lag behind the state-of-the-art protocols and robots will need to additionally reason intelligently about their teammates with limited information. This problem motivates the area of ad hoc teamwork in which an agent may potentially cooperate with a variety of teammates in order to achieve a shared goal. We argue that agents that effectively reason about ad hoc teamwork need to exhibit three capabilities: 1) robustness to teammate variety, 2) robustness to diverse tasks, and 3) fast adaptation. This thesis focuses on addressing all three of these challenges. In particular, this thesis introduces algorithms for quickly adapting to unknown teammates that enable agents to react to new teammates without extensive observations. The majority of existing multiagent algorithms focus on scenarios where all agents share coordination and communication protocols. While previous research on ad hoc teamwork considers some of these three challenges, this thesis introduces a new algorithm, PLASTIC, that is the first to address all three challenges in a single algorithm. PLASTIC adapts quickly to unknown teammates by reusing knowledge it learns about previous teammates and exploiting any expert knowledge available. Given this knowledge, PLASTIC selects which previous teammates are most similar to the current ones online and uses this information to adapt to their behaviors. This thesis introduces two instantiations of PLASTIC. The first is a model-based approach, PLASTIC-Model, that builds models of previous teammates' behaviors and plans online to determine the best course of action. The second uses a policy-based approach, PLASTIC-Policy, in which it learns policies for cooperating with past teammates and selects from among these policies online. Furthermore, we introduce a new transfer learning algorithm, TwoStageTransfer, that allows transferring knowledge from many past teammates while considering how similar each teammate is to the current ones. We theoretically analyze the computational tractability of PLASTIC-Model in a number of scenarios with unknown teammates. Additionally, we empirically evaluate PLASTIC in three domains that cover a spread of possible settings. Our evaluations show that PLASTIC can learn to communicate with unknown teammates using a limited set of messages, coordinate with externally-created teammates that do not reason about ad hoc teams, and act intelligently in domains with continuous states and actions. Furthermore, these evaluations show that TwoStageTransfer outperforms existing transfer learning algorithms and enables PLASTIC to adapt even better to new teammates. We also identify three dimensions that we argue best describe ad hoc teamwork scenarios. We hypothesize that these dimensions are useful for analyzing similarities among domains and determining which can be tackled by similar algorithms in addition to identifying avenues for future research. The work presented in this thesis represents an important step towards enabling agents to adapt to unknown teammates in the real world. PLASTIC significantly broadens the robustness of robots to their teammates and allows them to quickly adapt to new teammates by reusing previously learned knowledge.Computer Science

Benchmarking robot cooperation without pre-coordination in the RoboCup Standard Platform League drop-in player competition

Abstract — The Standard Platform League is one of the main competitions of the annual RoboCup world championships. In this competition, teams of five humanoid robots play soccer against each other. In 2014, the league added a new sub-competition which serves as a testbed for cooperation without pre-coordination: the Drop-in Player Competition. Instead of homogeneous robot teams that are each programmed by the same people and hence implicitly pre-coordinated, this competition features ad hoc teams, i. e. teams that consist of robots originating from different RoboCup teams and that are each running different software. In this paper, we provide an overview of this competition, including its motivation and rules. We then present and analyze the results of the 2014 competition, which gathered robots from 23 teams, involved at least 50 human participants, and consisted of fifteen 20-minute games for a total playing time of 300 minutes. We also suggest improvements for future iterations, many of which will be evaluated at RoboCup 2015. I

The Hanabi Challenge: A New Frontier for AI Research

From the early days of computing, games have been important testbeds for studying how well machines can do sophisticated decision making. In recent years, machine learning has made dramatic advances with artificial agents reaching superhuman performance in challenge domains like Go, Atari, and some variants of poker. As with their predecessors of chess, checkers, and backgammon, these game domains have driven research by providing sophisticated yet well-defined challenges for artificial intelligence practitioners. We continue this tradition by proposing the game of Hanabi as a new challenge domain with novel problems that arise from its combination of purely cooperative gameplay with two to five players and imperfect information. In particular, we argue that Hanabi elevates reasoning about the beliefs and intentions of other agents to the foreground. We believe developing novel techniques for such theory of mind reasoning will not only be crucial for success in Hanabi, but also in broader collaborative efforts, especially those with human partners. To facilitate future research, we introduce the open-source Hanabi Learning Environment, propose an experimental framework for the research community to evaluate algorithmic advances, and assess the performance of current state-of-the-art techniques.Comment: 32 pages, 5 figures, In Press (Artificial Intelligence

RoboCup 2D Soccer Simulation League: Evaluation Challenges

We summarise the results of RoboCup 2D Soccer Simulation League in 2016 (Leipzig), including the main competition and the evaluation round. The evaluation round held in Leipzig confirmed the strength of RoboCup-2015 champion (WrightEagle, i.e. WE2015) in the League, with only eventual finalists of 2016 competition capable of defeating WE2015. An extended, post-Leipzig, round-robin tournament which included the top 8 teams of 2016, as well as WE2015, with over 1000 games played for each pair, placed WE2015 third behind the champion team (Gliders2016) and the runner-up (HELIOS2016). This establishes WE2015 as a stable benchmark for the 2D Simulation League. We then contrast two ranking methods and suggest two options for future evaluation challenges. The first one, "The Champions Simulation League", is proposed to include 6 previous champions, directly competing against each other in a round-robin tournament, with the view to systematically trace the advancements in the League. The second proposal, "The Global Challenge", is aimed to increase the realism of the environmental conditions during the simulated games, by simulating specific features of different participating countries.Comment: 12 pages, RoboCup-2017, Nagoya, Japan, July 201

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications

Ad-hoc teamwork with behavior-switching agents

As autonomous AI agents proliferate in the real world, they will increasingly need to cooperate with each other to achieve complex goals without always being able to coordinate in advance. This kind of cooperation, in which agents have to learn to cooperate on the fly, is called ad hoc teamwork. Many previous works investigating this setting assumed that teammates behave according to one of many predefined types that is fixed throughout the task. This assumption of stationarity in behaviors, is a strong assumption which cannot be guaranteed in many real-world settings. In this work, we relax this assumption and investigate settings in which teammates can change their types during the course of the task. This adds complexity to the planning problem as now an agent needs to recognize that a change has occurred in addition to figuring out what is the new type of the teammate it is interacting with. In this paper, we present a novel Convolutional-Neural-Network-based Change Point Detection (CPD) algorithm for ad hoc teamwork. When evaluating our algorithm on the modified predator prey domain, we find that it outperforms existing Bayesian CPD algorithms.Electrical and Computer Engineerin

Generating Diverse Teammates to Train Robust Agents For Ad Hoc Teamwork

Ad hoc teamwork (AHT) is the challenge of designing a learner that effectively collaborates with unknown teammates without prior coordination mechanisms. Early approaches address the AHT challenge by training the learner with a diverse set of handcrafted teammate policies, usually designed based on an expert's domain knowledge about the policies the learner may encounter. However, implementing teammate policies for training based on domain knowledge is not always feasible. In such cases, recent approaches attempted to improve the robustness of the learner by training it with teammate policies generated by optimising information-theoretic diversity metrics. However, optimising information-theoretic diversity metrics may generate teammates with superficially different behaviours, which does not necessarily result in a robust learner that can effectively collaborate with unknown teammates. In this paper, we present an automated teammate policy generation method optimising the Best-Response Diversity (BRDiv) metric, which measures diversity based on the compatibility of teammate policies in terms of returns. We evaluate our approach in environments with multiple valid coordination strategies, comparing against methods optimising information-theoretic diversity metrics and an ablation not optimising any diversity metric. Our experiments indicate that optimising BRDiv yields a diverse set of training teammate policies that improve the learner's performance relative to previous teammate generation approaches when collaborating with near-optimal previously unseen teammate policies

Peli Alkaa : fantasia-aiheinen pakohuonepeli oppimisympäristönä Englanti vieraana kielenä opetuksessa

My thesis discusses the use of a fantasy-themed escape room as a learning environment for English as a foreign language classroom. Games, especially roleplaying games, have been the target of many studies and they have been shown to increase motivation, create new language scenarios and contexts as well as promote interaction between students and the gaming environment. However, the digital educational gaming industry in Finland is in its infancy. My study centers around piloting a digital fantasy-themed escape room called “The Mage’s Hut”, and this material package functions as the revision material for the course. I set to answer four research questions. The main goal is to find out how well a fantasy themed escape room functions as a learning environment. Secondly, I examine the attitudes of students and the teacher towards the game. Lastly, the fourth research question intends to answer how students’ previous gaming habits affect the gaming experience. I use relevant contemporary literature to examine gaming trends among Finnish upper secondary school students as well as delve deeper into the benefits of gamification used in formal contexts. This includes literature on roleplaying games as well as escape rooms and their utilization in formal educational contexts. The research design includes a digital fantasy-themed escape room that I have created based on the course material provided by Insights 5 -book by Otava for ENA5 course. The participants of the study included 26 Finnish upper secondary schoolers and their teacher. To gather the data, two questionnaires were constructed – one for students and one for the teacher. Twenty of the students responded to the questionnaire. The method for the study includes both qualitative and quantitative aspects. The questionnaires were analyzed by close-reading and were transformed into numerical values providing an easy way to compare results with each other. The results indicate that the attitudes of the students as well as the teacher towards the game are positive. The teacher and most of the students enjoyed playing the game and would like to see more similar games utilized in education. The game was deemed to be quite challenging, but it did not seem to negatively affect the game’s enjoyability or reported development of skills. These skills included vocabulary, grammar, reading comprehension, teamwork, information retrieval, logical and critical thinking. On the contrary, the students who deemed the game to be quite challenging reported more skills developed overall. The gaming habits of the students influenced the results partly. The students were categorized to non-gamers, casual gamers, active gamers, and hardcore gamers. Casual gamers, those with one to five hours of game time per week, reported developing the most skills out of all other groups. On the other hand, casual gamers reported the game as quite challenging the most. Therefore, it is not certain whether it is the difficulty or the game time that facilitates the development of the skills. Even though the majority of all groups reported enjoying the game, casual gamers reported the highest enjoyment rate. What we can gather from this study is that digital fantasy-themed escape rooms function well as a learning environment. The game corresponds to the learning objectives set by the Finnish curriculum and the teacher. Furthermore, the students felt that learning took place while playing the game. Both aspects of the game, the fantasy theme and the mechanics of an escape room were enjoyed, further reinforcing the previous knowledge on the functionality of roleplaying games in formal educational contexts

Fly with me : algorithms and methods for influencing a flock

As robots become more affordable, they will begin to exist in the world in greater quantities. Some of these robots will likely be designed to act as components in specific teams. These teams could work on tasks that are too large or complex for a single robot - or that are merely more efficiently accomplished by a team - such as surveillance in a large building or product delivery to packers in a warehouse. Multiagent systems research studies how these teams are formed and how they work together. Ad hoc teamwork, a newer area of multiagent systems research, studies how new robots can join these pre-existing teams and assist the team in accomplishing its goal. This dissertation extends and applies research in ad hoc teamwork towards the general area of flocking, which is an emergent swarm behavior. In particular, the work in this dissertation considers how ad hoc agents - called influencing agents in this dissertation - can join a flock, be recognized by the rest of the flock as part of the flock, influence the flock towards particular behaviors through their own behavior, and then separate from the flock. Specifically, the primary research question addressed in this dissertation is How can influencing agents be utilized in various types of flocks to influence the flock towards a particular behavior? In order to address this research question, this dissertation makes six main types of contributions. First, this dissertation formalizes the problem of using influencing agents to influence a flock. Second, this dissertation contributes and analyzes algorithms for influencing a flock to a desired orientation. Third, this dissertation presents methods for determining how to best add influencing agents to a flock. Fourth, this dissertation provides methods by which influencing agents can join and then leave a flock in motion. Fifth, this dissertation evaluates some of the influencing agent algorithms on a robot platform. Sixth, although the majority of this dissertation assumes the influencing agents will join a flock that behaves similarly to European starlings, this dissertation also provides insight into when and how its algorithms are generalizable to other types of flocks as well as to general teamwork and coordination research. All of the methods presented in this dissertation are empirically evaluated using a simulator that can support large flocks.Computer Science

The SATIN component system - a metamodel for engineering adaptable mobile systems

Mobile computing devices, such as personal digital assistants and mobile phones, are becoming increasingly popular, smaller, and more capable. We argue that mobile systems should be able to adapt to changing requirements and execution environments. Adaptation requires the ability-to reconfigure the deployed code base on a mobile device. Such reconfiguration is considerably simplified if mobile applications are component-oriented rather than monolithic blocks of code. We present the SATIN (system adaptation targeting integrated networks) component metamodel, a lightweight local component metamodel that offers the flexible use of logical mobility primitives to reconfigure the software system by dynamically transferring code. The metamodel is implemented in the SATIN middleware system, a component-based mobile computing middleware that uses the mobility primitives defined in the metamodel to reconfigure both itself and applications that it hosts. We demonstrate the suitability of SATIN in terms of lightweightedness, flexibility, and reusability for the creation of adaptable mobile systems by using it to implement, port, and evaluate a number of existing and new applications, including an active network platform developed for satellite communication at the European space agency. These applications exhibit different aspects of adaptation and demonstrate the flexibility of the approach and the advantages gaine