35 research outputs found

    NetHack is Hard to Hack

    Full text link
    Neural policy learning methods have achieved remarkable results in various control problems, ranging from Atari games to simulated locomotion. However, these methods struggle in long-horizon tasks, especially in open-ended environments with multi-modal observations, such as the popular dungeon-crawler game, NetHack. Intriguingly, the NeurIPS 2021 NetHack Challenge revealed that symbolic agents outperformed neural approaches by over four times in median game score. In this paper, we delve into the reasons behind this performance gap and present an extensive study on neural policy learning for NetHack. To conduct this study, we analyze the winning symbolic agent, extending its codebase to track internal strategy selection in order to generate one of the largest available demonstration datasets. Utilizing this dataset, we examine (i) the advantages of an action hierarchy; (ii) enhancements in neural architecture; and (iii) the integration of reinforcement learning with imitation learning. Our investigations produce a state-of-the-art neural agent that surpasses previous fully neural policies by 127% in offline settings and 25% in online settings on median game score. However, we also demonstrate that mere scaling is insufficient to bridge the performance gap with the best symbolic models or even the top human players

    Probabilistic Modeling for Game Content Creation and Adaption

    Get PDF
    Dynamic Difficulty Adjustment studies how games can adapt content totheir users’ skill level, aiming to keep them in flow. Most of these methodsmaximize engagement or minimize churn by adapting factors like the opponentAI or the availability of resources. However, such methods do notmaintain a model of the player, and use technologies that are highly specificto the games in which they are tested (e.g. requiring forward modelsfor enemy AIs based on planning agents). Designers may also intend tofind content that is more difficult/easier on purpose, and current methodsdo not allow for such targeting.This thesis proposes and tests a framework for adapting game content tousers based on Bayesian Optimization, giving designers flexibility whenchoosing which skill level to target. Starting with a design space, a metricto be measured, a prior over this metric, and a target value, our frameworkquickly searches possible levels/tasks for one with ideal difficulty (i.e. closeto the specified target). In the process, our framework maintains a simpledata-driven model of the player, which could be used for further decisionmakingand analysis.We test this framework in two settings: adapting content to planning agentsbased on search algorithms likeMonte Carlo Tree Search and Rolling HorizonEvolution in a dungeon crawler-type game, and adapting both Sudokupuzzles and dungeon crawler levels to players. Our framework successfullyadapts content to planning agents as long as their skill level is not extreme,and takes roughly 7 iterations to find an appropriate Sudoku puzzle.Additionally, instead of relying on designers to specify a real-valued encodingof the content (e.g. the number of pre-filled cells in a Sudoku puzzle),we investigate learning this encoding automatically usingDeep GenerativeModels. In other words, we explore design spaces learned as latent spacesof Variational Autoencoders using tile-based representations of games likeSuperMario Bros and The Legend of Zelda.Our final contribution is a novel way of interpolating, sampling and optimizingin the playable regions of latent spaces of Variational Autoencoders,and addresses the challenge that generative models are not always guaranteedto decode playable content. This contribution, based on differentialgeometry, is inspired by recent advancements in domains like robotics andproteinmodeling. We combine these ideas of safe generation with contentoptimization and propose a restricted version of Bayesian Optimization,which optimizes content inside playable regions. We see a clear trade-off:restricting the latent space to playable regions decreases the diversity ofthe generated content, as well as the quality of the optimal values in theoptimization.In summary, this thesis studies applications of Bayesian Optimization andDeep Generative Models to the problem of creating and adapting gamecontent to users. We develop a framework that quickly finds relevant levelsin settings varying from corpora of levels to the latent spaces of generativemodels, and we show in experiments involving both human and artificialplayers that this framework finds appropriate game content in a few iterations.This framework is readily applicable, and could be used to creategames that learn and adapt to their players.<br/

    Dungeons and Data: A Large-Scale NetHack Dataset

    Get PDF
    Recent breakthroughs in the development of agents to solve challenging sequential decision making problems such as Go [50], StarCraft [58], or DOTA [3], have relied on both simulated environments and large-scale datasets. However, progress on this research has been hindered by the scarcity of open-sourced datasets and the prohibitive computational cost to work with them. Here we present the NetHack Learning Dataset (NLD), a large and highly-scalable dataset of trajectories from the popular game of NetHack, which is both extremely challenging for current methods and very fast to run [23]. NLD consists of three parts: 10 billion state transitions from 1.5 million human trajectories collected on the NAO public NetHack server from 2009 to 2020; 3 billion state-action-score transitions from 100,000 trajectories collected from the symbolic bot winner of the NetHack Challenge 2021; and, accompanying code for users to record, load and stream any collection of such trajectories in a highly compressed form. We evaluate a wide range of existing algorithms including online and offline RL, as well as learning from demonstrations, showing that significant research advances are needed to fully leverage large-scale datasets for challenging sequential decision making tasks

    Generation and Analysis of Content for Physics-Based Video Games

    Get PDF
    The development of artificial intelligence (AI) techniques that can assist with the creation and analysis of digital content is a broad and challenging task for researchers. This topic has been most prevalent in the field of game AI research, where games are used as a testbed for solving more complex real-world problems. One of the major issues with prior AI-assisted content creation methods for games has been a lack of direct comparability to real-world environments, particularly those with realistic physical properties to consider. Creating content for such environments typically requires physics-based reasoning, which imposes many additional complications and restrictions that must be considered. Addressing and developing methods that can deal with these physical constraints, even if they are only within simulated game environments, is an important and challenging task for AI techniques that intend to be used in real-world situations. The research presented in this thesis describes several approaches to creating and analysing levels for the physics-based puzzle game Angry Birds, which features a realistic 2D environment. This research was multidisciplinary in nature and covers a wide variety of different AI fields, leading to this thesis being presented as a compilation of published work. The central part of this thesis consists of procedurally generating levels for physics-based games similar to those in Angry Birds. This predominantly involves creating and placing stable structures made up of many smaller blocks, as well as other level elements. Multiple approaches are presented, including both fully autonomous and human-AI collaborative methodologies. In addition, several analyses of Angry Birds levels were carried out using current state-of-the-art agents. A hyper-agent was developed that uses machine learning to estimate the performance of each agent in a portfolio for an unknown level, allowing it to select the one most likely to succeed. Agent performance on levels that contain deceptive or creative properties was also investigated, allowing determination of the current strengths and weaknesses of different AI techniques. The observed variability in performance across levels for different AI techniques led to the development of an adaptive level generation system, allowing for the dynamic creation of increasingly challenging levels over time based on agent performance analysis. An additional study also investigated the theoretical complexity of Angry Birds levels from a computational perspective. While this research is predominately applied to video games with physics-based simulated environments, the challenges and problems solved by the proposed methods also have significant real-world potential and applications

    Procedural Constraint-based Generation for Game Development

    Get PDF

    Language Learning in Interactive Environments

    Get PDF
    Natural language communication has long been considered a defining characteristic of human intelligence. I am motivated by the question of how learning agents can understand and generate contextually relevant natural language in service of achieving a goal. In pursuit of this objective, I have been studying Interactive Narratives, or text-adventures: simulations in which an agent interacts with the world purely through natural language—"seeing” and “acting upon” the world using textual descriptions and commands. These games are usually structured as puzzles or quests in which a player must complete a sequence of actions to succeed. My work studies two closely related aspects of Interactive Narratives: operating in these environments and creating them in addition to their intersection—each presenting its own set of unique challenges. Operating in these environments presents three challenges: (1) Knowledge representation—an agent must maintain a persistent memory of what it has learned through its experiences with a partially observable world; (2) Commonsense reasoning to endow the agent with priors on how to interact with the world around it; and (3) Scaling to effectively explore sparse-reward, combinatorially-sized natural language state-action spaces. On the other hand, creating these environments can be split into two complementary considerations: (1) World generation, or the problem of creating a world that defines the limits of the actions an agent can perform; and (2) Quest generation, i.e. defining actionable objectives grounded in a given world. I will present my work thus far—showcasing how structured, interpretable data representations in the form of knowledge graphs aid in each of these tasks—in addition to proposing how exactly these two aspects of Interactive Narratives can be combined to improve language learning and generalization across this board of challenges.Ph.D

    Languages of games and play: A systematic mapping study

    Get PDF
    Digital games are a powerful means for creating enticing, beautiful, educational, and often highly addictive interactive experiences that impact the lives of billions of players worldwide. We explore what informs the design and construction of good games to learn how to speed-up game development. In particular, we study to what extent languages, notations, patterns, and tools, can offer experts theoretical foundations, systematic techniques, and practical solutions they need to raise their productivity and improve the quality of games and play. Despite the growing number of publications on this topic there is currently no overview describing the state-of-the-art that relates research areas, goals, and applications. As a result, efforts and successes are often one-off, lessons learned go overlooked, language reuse remains minimal, and opportunities for collaboration and synergy are lost. We present a systematic map that identifies relevant publications and gives an overview of research areas and publication venues. In addition, we categorize research perspectives along common objectives, techniques, and approaches, illustrated by summaries of selected languages. Finally, we distill challenges and opportunities for future research and development

    Anxiety reducing through a neurofeedback serious game with dynamic difficulty adjustment

    Get PDF
    Presently, society has to deal with a large number of mental issues. Anxiety disorder is a serious concern, affecting millions of people’s lives and, although methods to tackle the problem currently exist, these main treatments are being linked to some issues and improvements must be found. One of the alternatives is Neurofeedback, a biofeedback treatment, completely non-invasive and showing impressive results so far. It uses a neuroheadset equipment to read the neural activity of the brain, giving the user visual feedback about it. The purpose this, is to train the users’ brain in specific regions and frequencies, allowing the subjects to learn how to voluntarily control its neural activity, even outside of the session. Current applications using this method might be too simple, which can become tedious and disengaging. Serious games can help with these issues, since it can bring enjoyment and engagement while doing this type of treatment. The interest in games’ capabilities in education has been increasing over the past years, since it has been proved that games are an excellent tool for education and skill learning. Joining these concepts of game and neurofeedback, this project aims to create a serious game prototype, applying the current treatment knowledge. The development process of a new game with neuroheadset integration, capable of reading the neural activity of the user while playing and giving the appropriate feedback, will be described in the present document. Since studies proved that a good balance between challenge and skill increases the learning performance, a dynamic difficulty adjustment system is implemented within the game, allowing the game to adapt itself to each user’s skill individually, and keeping the user in a challenging, motivating zone. At the end of the document, the results of pilot test on a few subjects are shown.Na sociedade actual o número de problemas relacionados com perturbações mentais tem sido cada vez mais relevante, sendo esse o caso da ansiedade. O distúrbio de ansiedade é um problema que atinge milhões de pessoas e, embora existam métodos para combater este problema, estudos comprovam que estes têm algumas lacunas que podem trazer outros problemas associados, sendo portanto necessário procurar melhorias aos métodos actuais. Uma das alternativas tem apresentado excelentes resultados e denomina-se Neurofeedback. Este é um tratamento de biofeedback, nãoinvasivo e que utiliza um equipamento neuroheadset para capturar a actividade neuronal, apresentando indicações visuais sobre o comportamento do utilizador. Isto é feito com o objectivo de treinar o cérebro do utilizador, em regiões e frequências específicas, para que este seja capaz de controlar voluntariamente a sua actividade neuronal. As aplicações actualmente utilizadas com este intuito podem se tornar aborrecidas e monótonas devido à sua simplicidade. Um jogo sério pode ajudar com estes problemas, uma vez que é capaz de trazer divertimento e motivação para este tipo de tratamento. O crescente interesse nas capacidades educativas dos jogos sérios, tem identificado estes como excelentes ferramentas para a educação. Este projecto pretende portanto criar um protótipo de um jogo sério, aplicando os conceitos de neurofeedback. Neste documento, é apresentado o processo de desenvolvimento de um novo jogo com integração de um neuroheadset, capaz de identificar a actividade neuronal do jogador dando respostas adequadas. Uma vez que estudos comprovam que um bom balanço entre desafio apresentado e técnica do utilizador aumenta a capacidade de aprendizagem, foi implementado também um sistema de ajuste de dificuldade dinâmica, permitindo uma adaptação do jogo a cada indivíduo e mantendo este numa zona motivante de equilíbrio entre desafio e proficiência. No final serão apresentados os resultados de um teste piloto efectuado em alguns indivíduos
    corecore