51 research outputs found
Insights from the NeurIPS 2021 NetHack Challenge
In this report, we summarize the takeaways from the first NeurIPS 2021 NetHack Challenge. Participants were tasked with developing a program or agent that can win (i.e., ‘ascend’ in) the popular dungeon-crawler game of NetHack by interacting with the NetHack Learning Environment (NLE), a scalable, procedurally generated, and challenging Gym environment for reinforcement learning (RL). The challenge showcased community-driven progress in AI with many diverse approaches significantly beating the previously best results on NetHack. Furthermore, it served as a direct comparison between neural (e.g., deep RL) and symbolic AI, as well as hybrid systems, demonstrating that on NetHack symbolic bots currently outperform deep RL by a large margin. Lastly, no agent got close to winning the game, illustrating NetHack’s suitability as a long-term benchmark for AI research
The NetHack learning environment
Progress in Reinforcement Learning (RL) algorithms goes hand-in-hand with the development of challenging environments that test the limits of current methods. While existing RL environments are either sufficiently complex or based on fast simulation, they are rarely both. Here, we present the NetHack Learning Environment (NLE), a scalable, procedurally generated, stochastic, rich, and challenging environment for RL research based on the popular single-player terminal-based roguelike game, NetHack. We argue that NetHack is sufficiently complex to drive long-term research on problems such as exploration, planning, skill acquisition, and language-conditioned RL, while dramatically reducing the computational resources required to gather a large amount of experience. We compare NLE and its task suite to existing alternatives, and discuss why it is an ideal medium for testing the robustness and systematic generalization of RL agents. We demonstrate empirical success for early stages of the game using a distributed Deep RL baseline and Random Network Distillation exploration, alongside qualitative analysis of various agents trained in the environment. NLE is open source and available at https://github.com/facebookresearch/nle
Playing NetHack with LLMs: Potential & Limitations as Zero-Shot Agents
Large Language Models (LLMs) have shown great success as high-level planners
for zero-shot game-playing agents. However, these agents are primarily
evaluated on Minecraft, where long-term planning is relatively straightforward.
In contrast, agents tested in dynamic robot environments face limitations due
to simplistic environments with only a few objects and interactions. To fill
this gap in the literature, we present NetPlay, the first LLM-powered zero-shot
agent for the challenging roguelike NetHack. NetHack is a particularly
challenging environment due to its diverse set of items and monsters, complex
interactions, and many ways to die.
NetPlay uses an architecture designed for dynamic robot environments,
modified for NetHack. Like previous approaches, it prompts the LLM to choose
from predefined skills and tracks past interactions to enhance decision-making.
Given NetHack's unpredictable nature, NetPlay detects important game events to
interrupt running skills, enabling it to react to unforeseen circumstances.
While NetPlay demonstrates considerable flexibility and proficiency in
interacting with NetHack's mechanics, it struggles with ambiguous task
descriptions and a lack of explicit feedback. Our findings demonstrate that
NetPlay performs best with detailed context information, indicating the
necessity for dynamic methods in supplying context information for complex
games such as NetHack
MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research
Progress in deep reinforcement learning (RL) is heavily driven by the availability of challenging benchmarks used for training agents. However, benchmarks that are widely adopted by the community are not explicitly designed for evaluating specific capabilities of RL methods. While there exist environments for assessing particular open problems in RL (such as exploration, transfer learning, unsupervised environment design, or even language-assisted RL), it is generally difficult to extend these to richer, more complex environments once research goes beyond proof-of-concept results. We present MiniHack, a powerful sandbox framework for easily designing novel RL environments. MiniHack is a one-stop shop for RL experiments with environments ranging from small rooms to complex, procedurally generated worlds. By leveraging the full set of entities and environment dynamics from NetHack, one of the richest grid-based video games, MiniHack allows designing custom RL testbeds that are fast and convenient to use. With this sandbox framework, novel environments can be designed easily, either using a human-readable description language or a simple Python interface. In addition to a variety of RL tasks and baselines, MiniHack can wrap existing RL benchmarks and provide ways to seamlessly add additional complexity
Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning
Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards
Improving Intrinsic Exploration with Language Abstractions
Reinforcement learning (RL) agents are particularly hard to train when
rewards are sparse. One common solution is to use intrinsic rewards to
encourage agents to explore their environment. However, recent intrinsic
exploration methods often use state-based novelty measures which reward
low-level exploration and may not scale to domains requiring more abstract
skills. Instead, we explore natural language as a general medium for
highlighting relevant abstractions in an environment. Unlike previous work, we
evaluate whether language can improve over existing exploration methods by
directly extending (and comparing to) competitive intrinsic exploration
baselines: AMIGo (Campero et al., 2021) and NovelD (Zhang et al., 2021). These
language-based variants outperform their non-linguistic forms by 47-85% across
13 challenging tasks from the MiniGrid and MiniHack environment suites.Comment: NeurIPS 202
Discussion of Game Design and Construction of a Videogame Utilizing PCG, CA, and ABM
Over time, video games have evolved and new methods for game design have allowed infinite possibilities and creativity. Some of these methods are Procedurally Content Generation and Cellular Automata. The use of CA-PCG has allowed immersive worlds for users to explore, creating an infinite amount of content to enjoy while providing challenging and unexpected gameplay. This senior project seeks to utilize these concepts along with Agent Based Modeling to create a fun dynamic game. The results of this project will be the discussion in how game design affects the use of these algorithms
Game-based Platforms for Artificial Intelligence Research
Games have been the perfect test-beds for artificial intelligence research
for the characteristics that widely exist in real-world scenarios. Learning and
optimisation, decision making in dynamic and uncertain environments, game
theory, planning and scheduling, design and education are common research areas
shared between games and real-world problems. Numerous open-sourced games or
game-based environments have been implemented for studying artificial
intelligence. In addition to single- or multi-player, collaborative or
adversarial games, there has also been growing interest in implementing
platforms for creative design in recent years. Those platforms provide ideal
benchmarks for exploring and comparing artificial intelligence ideas and
techniques. This paper reviews the game-based platforms for artificial
intelligence research, discusses the research trend induced by the evolution of
those platforms, and gives an outlook
- …