18 research outputs found
RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
Exploration in sparse reward environments remains one of the key challenges
of model-free reinforcement learning. Instead of solely relying on extrinsic
rewards provided by the environment, many state-of-the-art methods use
intrinsic rewards to encourage exploration. However, we show that existing
methods fall short in procedurally-generated environments where an agent is
unlikely to visit a state more than once. We propose a novel type of intrinsic
reward which encourages the agent to take actions that lead to significant
changes in its learned state representation. We evaluate our method on multiple
challenging procedurally-generated tasks in MiniGrid, as well as on tasks with
high-dimensional observations used in prior work. Our experiments demonstrate
that this approach is more sample efficient than existing exploration methods,
particularly for procedurally-generated MiniGrid environments. Furthermore, we
analyze the learned behavior as well as the intrinsic reward received by our
agent. In contrast to previous approaches, our intrinsic reward does not
diminish during the course of training and it rewards the agent substantially
more for interacting with objects that it can control
Hyperparameters in Reinforcement Learning and How To Tune Them
In order to improve reproducibility, deep reinforcement learning (RL) has
been adopting better scientific practices such as standardized evaluation
metrics and reporting. However, the process of hyperparameter optimization
still varies widely across papers, which makes it challenging to compare RL
algorithms fairly. In this paper, we show that hyperparameter choices in RL can
significantly affect the agent's final performance and sample efficiency, and
that the hyperparameter landscape can strongly depend on the tuning seed which
may lead to overfitting. We therefore propose adopting established best
practices from AutoML, such as the separation of tuning and testing seeds, as
well as principled hyperparameter optimization (HPO) across a broad search
space. We support this by comparing multiple state-of-the-art HPO tools on a
range of RL algorithms and environments to their hand-tuned counterparts,
demonstrating that HPO approaches often have higher performance and lower
compute overhead. As a result of our findings, we recommend a set of best
practices for the RL community, which should result in stronger empirical
results with fewer computational costs, better reproducibility, and thus faster
progress. In order to encourage the adoption of these practices, we provide
plug-and-play implementations of the tuning algorithms used in this paper at
https://github.com/facebookresearch/how-to-autorl
On the Importance of Exploration for Generalization in Reinforcement Learning
Existing approaches for improving generalization in deep reinforcement
learning (RL) have mostly focused on representation learning, neglecting
RL-specific aspects such as exploration. We hypothesize that the agent's
exploration strategy plays a key role in its ability to generalize to new
environments. Through a series of experiments in a tabular contextual MDP, we
show that exploration is helpful not only for efficiently finding the optimal
policy for the training environments but also for acquiring knowledge that
helps decision making in unseen environments. Based on these observations, we
propose EDE: Exploration via Distributional Ensemble, a method that encourages
exploration of states with high epistemic uncertainty through an ensemble of
Q-value distributions. Our algorithm is the first value-based approach to
achieve state-of-the-art on both Procgen and Crafter, two benchmarks for
generalization in RL with high-dimensional observations. The open-sourced
implementation can be found at https://github.com/facebookresearch/ede
Learning with AMIGo: Adversarially Motivated Intrinsic Goals
A key challenge for reinforcement learning (RL) consists of learning in
environments with sparse extrinsic rewards. In contrast to current RL methods,
humans are able to learn new skills with little or no reward by using various
forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating --
as form of meta-learning -- a goal-generating teacher that proposes
Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student"
policy in the absence of (or alongside) environment reward. Specifically,
through a simple but effective "constructively adversarial" objective, the
teacher learns to propose increasingly challenging -- yet achievable -- goals
that allow the student to learn general skills for acting in a new environment,
independent of the task to be solved. We show that our method generates a
natural curriculum of self-proposed goals which ultimately allows the agent to
solve challenging procedurally-generated tasks where other forms of intrinsic
motivation and state-of-the-art RL methods fail.Comment: 18 pages, 6 figures, published at The Ninth International Conference
on Learning Representations (2021
Chain-of-Verification Reduces Hallucination in Large Language Models
Generation of plausible yet incorrect factual information, termed
hallucination, is an unsolved issue in large language models. We study the
ability of language models to deliberate on the responses they give in order to
correct their mistakes. We develop the Chain-of-Verification (CoVe) method
whereby the model first (i) drafts an initial response; then (ii) plans
verification questions to fact-check its draft; (iii) answers those questions
independently so the answers are not biased by other responses; and (iv)
generates its final verified response. In experiments, we show CoVe decreases
hallucinations across a variety of tasks, from list-based questions from
Wikidata, closed book MultiSpanQA and longform text generation