15 research outputs found
Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination
The learned policy of model-free offline reinforcement learning (RL) methods
is often constrained to stay within the support of datasets to avoid possible
dangerous out-of-distribution actions or states, making it challenging to
handle out-of-support region. Model-based RL methods offer a richer dataset and
benefit generalization by generating imaginary trajectories with either trained
forward or reverse dynamics model. However, the imagined transitions may be
inaccurate, thus downgrading the performance of the underlying offline RL
method. In this paper, we propose to augment the offline dataset by using
trained bidirectional dynamics models and rollout policies with double check.
We introduce conservatism by trusting samples that the forward model and
backward model agree on. Our method, confidence-aware bidirectional offline
model-based imagination, generates reliable samples and can be combined with
any model-free offline RL method. Experimental results on the D4RL benchmarks
demonstrate that our method significantly boosts the performance of existing
model-free offline RL algorithms and achieves competitive or better scores
against baseline methods.Comment: NeurIPS 202
The primacy bias in Model-based RL
The primacy bias in deep reinforcement learning (DRL), which refers to the
agent's tendency to overfit early data and lose the ability to learn from new
data, can significantly decrease the performance of DRL algorithms. Previous
studies have shown that employing simple techniques, such as resetting the
agent's parameters, can substantially alleviate the primacy bias. However, we
observe that resetting the agent's parameters harms its performance in the
context of model-based reinforcement learning (MBRL). In fact, on further
investigation, we find that the primacy bias in MBRL differs from that in
model-free RL. In this work, we focus on investigating the primacy bias in MBRL
and propose world model resetting, which works in MBRL. We apply our method to
two different MBRL algorithms, MBPO and DreamerV2. We validate the
effectiveness of our method on multiple continuous control tasks on MuJoCo and
DeepMind Control Suite, as well as discrete control tasks on Atari 100k
benchmark. The results show that world model resetting can significantly
alleviate the primacy bias in model-based setting and improve algorithm's
performance. We also give a guide on how to perform world model resetting
effectively
Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse
Sample efficiency is one of the most critical issues for online reinforcement
learning (RL). Existing methods achieve higher sample efficiency by adopting
model-based methods, Q-ensemble, or better exploration mechanisms. We, instead,
propose to train an off-policy RL agent via updating on a fixed sampled batch
multiple times, thus reusing these samples and better exploiting them within a
single optimization loop. We name our method sample multiple reuse (SMR). We
theoretically show the properties of Q-learning with SMR, e.g., convergence.
Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and
conduct experiments on a variety of continuous control benchmarks. Empirical
results show that SMR significantly boosts the sample efficiency of the base
methods across most of the evaluated tasks without any hyperparameter tuning or
additional tricks.Comment: 37 page
Zero-shot Preference Learning for Offline RL via Optimal Transport
Preference-based Reinforcement Learning (PbRL) has demonstrated remarkable
efficacy in aligning rewards with human intentions. However, a significant
challenge lies in the need of substantial human labels, which is costly and
time-consuming. Additionally, the expensive preference data obtained from prior
tasks is not typically reusable for subsequent task learning, leading to
extensive labeling for each new task. In this paper, we propose a novel
zero-shot preference-based RL algorithm that leverages labeled preference data
from source tasks to infer labels for target tasks, eliminating the requirement
for human queries. Our approach utilizes Gromov-Wasserstein distance to align
trajectory distributions between source and target tasks. The solved optimal
transport matrix serves as a correspondence between trajectories of two tasks,
making it possible to identify corresponding trajectory pairs between tasks and
transfer the preference labels. However, learning directly from inferred labels
that contains a fraction of noisy labels will result in an inaccurate reward
function, subsequently affecting policy performance. To this end, we introduce
Robust Preference Transformer, which models the rewards as Gaussian
distributions and incorporates reward uncertainty in addition to reward mean.
The empirical results on robotic manipulation tasks of Meta-World and Robomimic
show that our method has strong capabilities of transferring preferences
between tasks and learns reward functions from noisy labels robustly.
Furthermore, we reveal that our method attains near-oracle performance with a
small proportion of scripted labels
State Advantage Weighting for Offline RL
We present state advantage weighting for offline reinforcement learning (RL).
In contrast to action advantage that we commonly adopt in QSA
learning, we leverage state advantage and QSS learning for
offline RL, hence decoupling the action from values. We expect the agent can
get to the high-reward state and the action is determined by how the agent can
get to that corresponding state. Experiments on D4RL datasets show that our
proposed method can achieve remarkable performance against the common
baselines. Furthermore, our method shows good generalization capability when
transferring from offline to online.Comment: 3rd Offline RL workshop at NeurIPS 2022. arXiv admin note: text
overlap with arXiv:2206.0798
Uncertainty-driven Trajectory Truncation for Model-based Offline Reinforcement Learning
Equipped with the trained environmental dynamics, model-based offline
reinforcement learning (RL) algorithms can often successfully learn good
policies from fixed-sized datasets, even some datasets with poor quality.
Unfortunately, however, it can not be guaranteed that the generated samples
from the trained dynamics model are reliable (e.g., some synthetic samples may
lie outside of the support region of the static dataset). To address this
issue, we propose Trajectory Truncation with Uncertainty (TATU), which
adaptively truncates the synthetic trajectory if the accumulated uncertainty
along the trajectory is too large. We theoretically show the performance bound
of TATU to justify its benefits. To empirically show the advantages of TATU, we
first combine it with two classical model-based offline RL algorithms, MOPO and
COMBO. Furthermore, we integrate TATU with several off-the-shelf model-free
offline RL algorithms, e.g., BCQ. Experimental results on the D4RL benchmark
show that TATU significantly improves their performance, often by a large
margin
Normalization Enhances Generalization in Visual Reinforcement Learning
Recent advances in visual reinforcement learning (RL) have led to impressive
success in handling complex tasks. However, these methods have demonstrated
limited generalization capability to visual disturbances, which poses a
significant challenge for their real-world application and adaptability. Though
normalization techniques have demonstrated huge success in supervised and
unsupervised learning, their applications in visual RL are still scarce. In
this paper, we explore the potential benefits of integrating normalization into
visual RL methods with respect to generalization performance. We find that,
perhaps surprisingly, incorporating suitable normalization techniques is
sufficient to enhance the generalization capabilities, without any additional
special design. We utilize the combination of two normalization techniques,
CrossNorm and SelfNorm, for generalizable visual RL. Extensive experiments are
conducted on DMControl Generalization Benchmark and CARLA to validate the
effectiveness of our method. We show that our method significantly improves
generalization capability while only marginally affecting sample efficiency. In
particular, when integrated with DrQ-v2, our method enhances the test
performance of DrQ-v2 on CARLA across various scenarios, from 14% of the
training performance to 97%
Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
Multi-goal reinforcement learning is widely applied in planning and robot
manipulation. Two main challenges in multi-goal reinforcement learning are
sparse rewards and sample inefficiency. Hindsight Experience Replay (HER) aims
to tackle the two challenges via goal relabeling. However, HER-related works
still need millions of samples and a huge computation. In this paper, we
propose Multi-step Hindsight Experience Replay (MHER), incorporating multi-step
relabeled returns based on -step relabeling to improve sample efficiency.
Despite the advantages of -step relabeling, we theoretically and
experimentally prove the off-policy -step bias introduced by -step
relabeling may lead to poor performance in many environments. To address the
above issue, two bias-reduced MHER algorithms, MHER() and Model-based
MHER (MMHER) are presented. MHER() exploits the return while
MMHER benefits from model-based value expansions. Experimental results on
numerous multi-goal robotic tasks show that our solutions can successfully
alleviate off-policy -step bias and achieve significantly higher sample
efficiency than HER and Curriculum-guided HER with little additional
computation beyond HER.Comment: 20pages, 8 figure
Mildly Conservative Q-Learning for Offline Reinforcement Learning
Offline reinforcement learning (RL) defines the task of learning from a
static logged dataset without continually interacting with the environment. The
distribution shift between the learned policy and the behavior policy makes it
necessary for the value function to stay conservative such that
out-of-distribution (OOD) actions will not be severely overestimated. However,
existing approaches, penalizing the unseen actions or regularizing with the
behavior policy, are too pessimistic, which suppresses the generalization of
the value function and hinders the performance improvement. This paper explores
mild but enough conservatism for offline learning while not harming
generalization. We propose Mildly Conservative Q-learning (MCQ), where OOD
actions are actively trained by assigning them proper pseudo Q values. We
theoretically show that MCQ induces a policy that behaves at least as well as
the behavior policy and no erroneous overestimation will occur for OOD actions.
Experimental results on the D4RL benchmarks demonstrate that MCQ achieves
remarkable performance compared with prior work. Furthermore, MCQ shows
superior generalization ability when transferring from offline to online, and
significantly outperforms baselines