VirtualHome: Simulating Household Activities via Programs

In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.Comment: CVPR 2018 (Oral

Kaleidoscope JEIRP on Learning Patterns for the Design and Deployment of Mathematical Games: Final Report

Project deliverable (D40.05.01-F)Over the last few years have witnessed a growing recognition of the educational potential of computer games. However, it is generally agreed that the process of designing and deploying TEL resources generally and games for mathematical learning specifically is a difficult task. The Kaleidoscope project, "Learning patterns for the design and deployment of mathematical games", aims to investigate this problem. We work from the premise that designing and deploying games for mathematical learning requires the assimilation and integration of deep knowledge from diverse domains of expertise including mathematics, games development, software engineering, learning and teaching. We promote the use of a design patterns approach to address this problem. This deliverable reports on the project by presenting both a connected account of the prior deliverables and also a detailed description of the methodology involved in producing those deliverables. In terms of conducting the future work which this report envisages, the setting out of our methodology is seen by us as very significant. The central deliverable includes reference to a large set of learning patterns for use by educators, researchers, practitioners, designers and software developers when designing and deploying TEL-based mathematical games. Our pattern language is suggested as an enabling tool for good practice, by facilitating pattern-specific communication and knowledge sharing between participants. We provide a set of trails as a "way-in" to using the learning pattern language. We report in this methodology how the project has enabled the synergistic collaboration of what started out as two distinct strands: design and deployment, even to the extent that it is now difficult to identify those strands within the processes and deliverables of the project. The tools and outcomes from the project can be found at: http://lp.noe-kaleidoscope.org

Text-based Adventures of the Golovin AI Agent

The domain of text-based adventure games has been recently established as a new challenge of creating the agent that is both able to understand natural language, and acts intelligently in text-described environments. In this paper, we present our approach to tackle the problem. Our agent, named Golovin, takes advantage of the limited game domain. We use genre-related corpora (including fantasy books and decompiled games) to create language models suitable to this domain. Moreover, we embed mechanisms that allow us to specify, and separately handle, important tasks as fighting opponents, managing inventory, and navigating on the game map. We validated usefulness of these mechanisms, measuring agent's performance on the set of 50 interactive fiction games. Finally, we show that our agent plays on a level comparable to the winner of the last year Text-Based Adventure AI Competition

AI Researchers, Video Games Are Your Friends!

If you are an artificial intelligence researcher, you should look to video games as ideal testbeds for the work you do. If you are a video game developer, you should look to AI for the technology that makes completely new types of games possible. This chapter lays out the case for both of these propositions. It asks the question "what can video games do for AI", and discusses how in particular general video game playing is the ideal testbed for artificial general intelligence research. It then asks the question "what can AI do for video games", and lays out a vision for what video games might look like if we had significantly more advanced AI at our disposal. The chapter is based on my keynote at IJCCI 2015, and is written in an attempt to be accessible to a broad audience.Comment: in Studies in Computational Intelligence Studies in Computational Intelligence, Volume 669 2017. Springe

Generating Levels That Teach Mechanics

The automatic generation of game tutorials is a challenging AI problem. While it is possible to generate annotations and instructions that explain to the player how the game is played, this paper focuses on generating a gameplay experience that introduces the player to a game mechanic. It evolves small levels for the Mario AI Framework that can only be beaten by an agent that knows how to perform specific actions in the game. It uses variations of a perfect A* agent that are limited in various ways, such as not being able to jump high or see enemies, to test how failing to do certain actions can stop the player from beating the level.Comment: 8 pages, 7 figures, PCG Workshop at FDG 2018, 9th International Workshop on Procedural Content Generation (PCG2018

SAGA: A DSL for Story Management

Video game development is currently a very labour-intensive endeavour. Furthermore it involves multi-disciplinary teams of artistic content creators and programmers, whose typical working patterns are not easily meshed. SAGA is our first effort at augmenting the productivity of such teams. Already convinced of the benefits of DSLs, we set out to analyze the domains present in games in order to find out which would be most amenable to the DSL approach. Based on previous work, we thus sought those sub-parts that already had a partially established vocabulary and at the same time could be well modeled using classical computer science structures. We settled on the 'story' aspect of video games as the best candidate domain, which can be modeled using state transition systems. As we are working with a specific company as the ultimate customer for this work, an additional requirement was that our DSL should produce code that can be used within a pre-existing framework. We developed a full system (SAGA) comprised of a parser for a human-friendly language for 'story events', an internal representation of design patterns for implementing object-oriented state-transitions systems, an instantiator for these patterns for a specific 'story', and three renderers (for C++, C# and Java) for the instantiated abstract code.Comment: In Proceedings DSL 2011, arXiv:1109.032

Tweeting your Destiny: Profiling Users in the Twitter Landscape around an Online Game

Social media has become a major communication channel for communities centered around video games. Consequently, social media offers a rich data source to study online communities and the discussions evolving around games. Towards this end, we explore a large-scale dataset consisting of over 1 million tweets related to the online multiplayer shooter Destiny and spanning a time period of about 14 months using unsupervised clustering and topic modelling. Furthermore, we correlate Twitter activity of over 3,000 players with their playtime. Our results contribute to the understanding of online player communities by identifying distinct player groups with respect to their Twitter characteristics, describing subgroups within the Destiny community, and uncovering broad topics of community interest.Comment: Accepted at IEEE Conference on Games 201