Grounding Language for Transfer in Deep Reinforcement Learning

In this paper, we explore the utilization of natural language to drive transfer for reinforcement learning (RL). Despite the wide-spread application of deep RL techniques, learning generalized policy representations that work across domains remains a challenging problem. We demonstrate that textual descriptions of environments provide a compact intermediate channel to facilitate effective policy transfer. Specifically, by learning to ground the meaning of text to the dynamics of the environment such as transitions and rewards, an autonomous agent can effectively bootstrap policy learning on a new domain given its description. We employ a model-based RL approach consisting of a differentiable planning module, a model-free component and a factorized state representation to effectively use entity descriptions. Our model outperforms prior work on both transfer and multi-task scenarios in a variety of different environments. For instance, we achieve up to 14% and 11.5% absolute improvement over previously existing models in terms of average and initial rewards, respectively.Comment: JAIR 201

Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog

A number of recent works have proposed techniques for end-to-end learning of communication protocols among cooperative multi-agent populations, and have simultaneously found the emergence of grounded human-interpretable language in the protocols developed by the agents, all learned without any human supervision! In this paper, using a Task and Tell reference game between two agents as a testbed, we present a sequence of 'negative' results culminating in a 'positive' one -- showing that while most agent-invented languages are effective (i.e. achieve near-perfect task rewards), they are decidedly not interpretable or compositional. In essence, we find that natural language does not emerge 'naturally', despite the semblance of ease of natural-language-emergence that one may gather from recent literature. We discuss how it is possible to coax the invented languages to become more and more human-like and compositional by increasing restrictions on how two agents may communicate.Comment: 9 pages, 7 figures, 2 tables, accepted at EMNLP 2017 as short pape

Learning a Policy for Opportunistic Active Learning

Active learning identifies data points to label that are expected to be the most useful in improving a supervised model. Opportunistic active learning incorporates active learning into interactive tasks that constrain possible queries during interactions. Prior work has shown that opportunistic active learning can be used to improve grounding of natural language descriptions in an interactive object retrieval task. In this work, we use reinforcement learning for such an object retrieval task, to learn a policy that effectively trades off task completion with model improvement that would benefit future tasks.Comment: EMNLP 2018 Camera Read

In-Game Social Interactions to Facilitate ESL Students\u27 Morphological Awareness, Language and Literacy Skills

Video games that require players to utilize a target or second language to complete tasks have emerged as alternative pedagogical tools for Second Language Acquisition (SLA). With the exception of vocabulary acquisition, much of the prior research in game-based SLA fails to gauge students\u27 literacy skills, specifically their morphological awareness or understanding of the smallest meaningful linguistic units (e.g., prefixes, suffixes, and roots). Given this shortcoming, we utilize a two-player online game to facilitate social interactions between Native English Speakers (NES) and English as a Second Language (ESL) students as a mechanism to generate ESL students\u27 written output in the targeted language and draw attention to their morphological awareness. Analysis of chat logs demonstrates the game\u27s potential to enhance ESL students\u27 morphological awareness and other important L2 literacy skills such as word reading accuracy. Both NES and ESL students\u27 reflections of their gameplay experiences suggest game design modifications that promote ESL students\u27 willingness to communicate with NES while developing their morphological awareness and practicing their L2 communication and literacy skills

Improving Grounded Natural Language Understanding through Human-Robot Dialog

Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically---continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world

Grounding Symbols in Multi-Modal Instructions

As robots begin to cohabit with humans in semi-structured environments, the need arises to understand instructions involving rich variability---for instance, learning to ground symbols in the physical world. Realistically, this task must cope with small datasets consisting of a particular users' contextual assignment of meaning to terms. We present a method for processing a raw stream of cross-modal input---i.e., linguistic instructions, visual perception of a scene and a concurrent trace of 3D eye tracking fixations---to produce the segmentation of objects with a correspondent association to high-level concepts. To test our framework we present experiments in a table-top object manipulation scenario. Our results show our model learns the user's notion of colour and shape from a small number of physical demonstrations, generalising to identifying physical referents for novel combinations of the words.Comment: 9 pages, 8 figures, To appear in the Proceedings of the ACL workshop Language Grounding for Robotics, Vancouver, Canad

“Welcome to the world of Pokémon!”: music and the player’s experience in Chunsoft’s Pokémon Mystery Dungeon

Most scholarship on video game music tends to focus on either their interactive or non-interactive elements, known as ‘gameplay’ and ‘story’. The music of Chunsoft’s Pokémon Mystery Dungeon series unites gameplay and story through the use of motives, silence, and shared modes and keys. This blending has important ramifications for the player’s gaming experience. The recurrence of musical elements links discrete tracks and scenes within the games, making the audio crucial for understanding the full meaning of the games

Games for a new climate: experiencing the complexity of future risks

This repository item contains a single issue of the Pardee Center Task Force Reports, a publication series that began publishing in 2009 by the Boston University Frederick S. Pardee Center for the Study of the Longer-Range Future.This report is a product of the Pardee Center Task Force on Games for a New Climate, which met at Pardee House at Boston University in March 2012. The 12-member Task Force was convened on behalf of the Pardee Center by Visiting Research Fellow Pablo Suarez in collaboration with the Red Cross/Red Crescent Climate Centre to “explore the potential of participatory, game-based processes for accelerating learning, fostering dialogue, and promoting action through real-world decisions affecting the longer-range future, with an emphasis on humanitarian and development work, particularly involving climate risk management.” Compiled and edited by Janot Mendler de Suarez, Pablo Suarez and Carina Bachofen, the report includes contributions from all of the Task Force members and provides a detailed exploration of the current and potential ways in which games can be used to help a variety of stakeholders – including subsistence farmers, humanitarian workers, scientists, policymakers, and donors – to both understand and experience the difficulty and risks involved related to decision-making in a complex and uncertain future. The dozen Task Force experts who contributed to the report represent academic institutions, humanitarian organization, other non-governmental organizations, and game design firms with backgrounds ranging from climate modeling and anthropology to community-level disaster management and national and global policymaking as well as game design.Red Cross/Red Crescent Climate Centr

Language Understanding for Text-based Games Using Deep Reinforcement Learning

In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This framework enables us to map text descriptions into vector representations that capture the semantics of the game states. We evaluate our approach on two game worlds, comparing against baselines using bag-of-words and bag-of-bigrams for state representations. Our algorithm outperforms the baselines on both worlds demonstrating the importance of learning expressive representations.Comment: 11 pages, Appearing at EMNLP, 201