1,932 research outputs found
Provably Efficient Maximum Entropy Exploration
Suppose an agent is in a (possibly unknown) Markov Decision Process in the
absence of a reward signal, what might we hope that an agent can efficiently
learn to do? This work studies a broad class of objectives that are defined
solely as functions of the state-visitation frequencies that are induced by how
the agent behaves. For example, one natural, intrinsically defined, objective
problem is for the agent to learn a policy which induces a distribution over
state space that is as uniform as possible, which can be measured in an
entropic sense. We provide an efficient algorithm to optimize such such
intrinsically defined objectives, when given access to a black box planning
oracle (which is robust to function approximation). Furthermore, when
restricted to the tabular setting where we have sample based access to the MDP,
our proposed algorithm is provably efficient, both in terms of its sample and
computational complexities. Key to our algorithmic methodology is utilizing the
conditional gradient method (a.k.a. the Frank-Wolfe algorithm) which utilizes
an approximate MDP solver.Comment: Updated experiment results; minor revisions in writin
Verifiable Reinforcement Learning via Policy Extraction
While deep reinforcement learning has successfully solved many challenging
control tasks, its real-world applicability has been limited by the inability
to ensure the safety of learned policies. We propose an approach to verifiable
reinforcement learning by training decision tree policies, which can represent
complex policies (since they are nonparametric), yet can be efficiently
verified using existing techniques (since they are highly structured). The
challenge is that decision tree policies are difficult to train. We propose
VIPER, an algorithm that combines ideas from model compression and imitation
learning to learn decision tree policies guided by a DNN policy (called the
oracle) and its Q-function, and show that it substantially outperforms two
baselines. We use VIPER to (i) learn a provably robust decision tree policy for
a variant of Atari Pong with a symbolic state space, (ii) learn a decision tree
policy for a toy game based on Pong that provably never loses, and (iii) learn
a provably stable decision tree policy for cart-pole. In each case, the
decision tree policy achieves performance equal to that of the original DNN
policy
Functional Bandits
We introduce the functional bandit problem, where the objective is to find an
arm that optimises a known functional of the unknown arm-reward distributions.
These problems arise in many settings such as maximum entropy methods in
natural language processing, and risk-averse decision-making, but current
best-arm identification techniques fail in these domains. We propose a new
approach, that combines functional estimation and arm elimination, to tackle
this problem. This method achieves provably efficient performance guarantees.
In addition, we illustrate this method on a number of important functionals in
risk management and information theory, and refine our generic theoretical
results in those cases
Composable Deep Reinforcement Learning for Robotic Manipulation
Model-free deep reinforcement learning has been shown to exhibit good
performance in domains ranging from video games to simulated robotic
manipulation and locomotion. However, model-free methods are known to perform
poorly when the interaction time with the environment is limited, as is the
case for most real-world robotic tasks. In this paper, we study how maximum
entropy policies trained using soft Q-learning can be applied to real-world
robotic manipulation. The application of this method to real-world manipulation
is facilitated by two important features of soft Q-learning. First, soft
Q-learning can learn multimodal exploration strategies by learning policies
represented by expressive energy-based models. Second, we show that policies
learned with soft Q-learning can be composed to create new policies, and that
the optimality of the resulting policy can be bounded in terms of the
divergence between the composed policies. This compositionality provides an
especially valuable tool for real-world manipulation, where constructing new
policies by composing existing skills can provide a large gain in efficiency
over training from scratch. Our experimental evaluation demonstrates that soft
Q-learning is substantially more sample efficient than prior model-free deep
reinforcement learning methods, and that compositionality can be performed for
both simulated and real-world tasks.Comment: Videos: https://sites.google.com/view/composing-real-world-policies
Recommended from our members
Towards Informed Exploration for Deep Reinforcement Learning
In this thesis, we discuss various techniques for improving exploration for deep reinforcement learning. We begin with a brief review of reinforcement learning (RL) and the fundamental v.s. exploitation trade-off. Then we review how deep RL has improved upon classical and summarize six categories of the latest exploration methods for deep RL, in the order increasing usage of prior information. We then explore representative works in three categories discuss their strengths and weaknesses. The first category, represented by Soft Q-learning, uses regularization to encourage exploration. The second category, represented by count-based via hashing, maps states to hash codes for counting and assigns higher exploration to less-encountered states. The third category utilizes hierarchy and is represented by modular architecture for RL agents to play StarCraft II. Finally, we conclude that exploration by prior knowledge is a promising research direction and suggest topics of potentially impact
Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
In real-world tasks, reinforcement learning (RL) agents frequently encounter
situations that are not present during training time. To ensure reliable
performance, the RL agents need to exhibit robustness against worst-case
situations. The robust RL framework addresses this challenge via a worst-case
optimization between an agent and an adversary. Previous robust RL algorithms
are either sample inefficient, lack robustness guarantees, or do not scale to
large problems. We propose the Robust Hallucinated Upper-Confidence RL
(RH-UCRL) algorithm to provably solve this problem while attaining near-optimal
sample complexity guarantees. RH-UCRL is a model-based reinforcement learning
(MBRL) algorithm that effectively distinguishes between epistemic and aleatoric
uncertainty and efficiently explores both the agent and adversary decision
spaces during policy learning. We scale RH-UCRL to complex tasks via neural
networks ensemble models as well as neural network policies. Experimentally, we
demonstrate that RH-UCRL outperforms other robust deep RL algorithms in a
variety of adversarial environments
- …