379 research outputs found
Log-Distributional Approach for Learning Covariate Shift Ratios
Distributional Reinforcement Learning theory suggests that distributional fixed points could play a fundamental role to learning non additive value functions. In particular, we propose a distributional approach for learning Covariate Shift Ratios, whose update rule is originally multiplicative
Primal Wasserstein Imitation Learning
Imitation Learning (IL) methods seek to match the behavior of an agent with
that of an expert. In the present work, we propose a new IL method based on a
conceptually simple algorithm: Primal Wasserstein Imitation Learning (PWIL),
which ties to the primal form of the Wasserstein distance between the expert
and the agent state-action distributions. We present a reward function which is
derived offline, as opposed to recent adversarial IL algorithms that learn a
reward function through interactions with the environment, and which requires
little fine-tuning. We show that we can recover expert behavior on a variety of
continuous control tasks of the MuJoCo domain in a sample efficient manner in
terms of agent interactions and of expert interactions with the environment.
Finally, we show that the behavior of the agent we train matches the behavior
of the expert with the Wasserstein distance, rather than the commonly used
proxy of performance.Comment: Published in International Conference on Learning Representations
(ICLR 2021
A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning
Model-based Reinforcement Learning (MBRL) aims to make agents more
sample-efficient, adaptive, and explainable by learning an explicit model of
the environment. While the capabilities of MBRL agents have significantly
improved in recent years, how to best learn the model is still an unresolved
question. The majority of MBRL algorithms aim at training the model to make
accurate predictions about the environment and subsequently using the model to
determine the most rewarding actions. However, recent research has shown that
model predictive accuracy is often not correlated with action quality, tracing
the root cause to the \emph{objective mismatch} between accurate dynamics model
learning and policy optimization of rewards. A number of interrelated solution
categories to the objective mismatch problem have emerged as MBRL continues to
mature as a research area. In this work, we provide an in-depth survey of these
solution categories and propose a taxonomy to foster future research
- …