14,861 research outputs found
Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
Reinforcement learning (RL) has shown great promise with algorithms learning
in environments with large state and action spaces purely from scalar reward
signals. A crucial challenge for current deep RL algorithms is that they
require a tremendous amount of environment interactions for learning. This can
be infeasible in situations where such interactions are expensive; such as in
robotics. Offline RL algorithms try to address this issue by bootstrapping the
learning process from existing logged data without needing to interact with the
environment from the very beginning. While online RL algorithms are typically
evaluated as a function of the number of environment interactions, there exists
no single established protocol for evaluating offline RL methods.In this paper,
we propose a sequential approach to evaluate offline RL algorithms as a
function of the training set size and thus by their data efficiency. Sequential
evaluation provides valuable insights into the data efficiency of the learning
process and the robustness of algorithms to distribution changes in the dataset
while also harmonizing the visualization of the offline and online learning
phases. Our approach is generally applicable and easy to implement. We compare
several existing offline RL algorithms using this approach and present insights
from a variety of tasks and offline datasets.Comment: TMLR 202
Least-squares methods for policy iteration
Approximate reinforcement learning deals with the essential problem of applying reinforcement learning in large and continuous state-action spaces, by using function approximators to represent the solution. This chapter reviews least-squares methods for policy iteration, an important class of algorithms for approximate reinforcement learning. We discuss three techniques for solving the core, policy evaluation component of policy iteration, called: least-squares temporal difference, least-squares policy evaluation, and Bellman residual minimization. We introduce these techniques starting from their general mathematical principles and detailing them down to fully specified algorithms. We pay attention to online variants of policy iteration, and provide a numerical example highlighting the behavior of representative offline and online methods. For the policy evaluation component as well as for the overall resulting approximate policy iteration, we provide guarantees on the performance obtained asymptotically, as the number of samples processed and iterations executed grows to infinity. We also provide finite-sample results, which apply when a finite number of samples and iterations are considered. Finally, we outline several extensions and improvements to the techniques and methods reviewed
Automating Vehicles by Deep Reinforcement Learning using Task Separation with Hill Climbing
Within the context of autonomous driving a model-based reinforcement learning
algorithm is proposed for the design of neural network-parameterized
controllers. Classical model-based control methods, which include sampling- and
lattice-based algorithms and model predictive control, suffer from the
trade-off between model complexity and computational burden required for the
online solution of expensive optimization or search problems at every short
sampling time. To circumvent this trade-off, a 2-step procedure is motivated:
first learning of a controller during offline training based on an arbitrarily
complicated mathematical system model, before online fast feedforward
evaluation of the trained controller. The contribution of this paper is the
proposition of a simple gradient-free and model-based algorithm for deep
reinforcement learning using task separation with hill climbing (TSHC). In
particular, (i) simultaneous training on separate deterministic tasks with the
purpose of encoding many motion primitives in a neural network, and (ii) the
employment of maximally sparse rewards in combination with virtual velocity
constraints (VVCs) in setpoint proximity are advocated.Comment: 10 pages, 6 figures, 1 tabl
Tracking-by-Trackers with a Distilled and Reinforced Model
Visual object tracking was generally tackled by reasoning independently on fast processing algorithms, accurate online adaptation methods, and fusion of trackers. In this paper, we unify such goals by proposing a novel tracking methodology that takes advantage of other visual trackers, offline and online. A compact student model is trained via the marriage of knowledge distillation and reinforcement learning. The first allows to transfer and compress tracking knowledge of other trackers. The second enables the learning of evaluation measures which are then exploited online. After learning, the student can be ultimately used to build (i) a very fast single-shot tracker, (ii) a tracker with a simple and effective online adaptation mechanism, (iii) a tracker that performs fusion of other trackers. Extensive validation shows that the proposed algorithms compete with real-time state-of-the-art trackers
Reinforced Self-Training (ReST) for Language Modeling
Reinforcement learning from human feedback (RLHF) can improve the quality of
large language model's (LLM) outputs by aligning them with human preferences.
We propose a simple algorithm for aligning LLMs with human preferences inspired
by growing batch reinforcement learning (RL), which we call Reinforced
Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by
generating samples from the policy, which are then used to improve the LLM
policy using offline RL algorithms. ReST is more efficient than typical online
RLHF methods because the training dataset is produced offline, which allows
data reuse. While ReST is a general approach applicable to all generative
learning settings, we focus on its application to machine translation. Our
results show that ReST can substantially improve translation quality, as
measured by automated metrics and human evaluation on machine translation
benchmarks in a compute and sample-efficient manner.Comment: 23 pages, 16 figure
Boosting Offline Reinforcement Learning with Action Preference Query
Training practical agents usually involve offline and online reinforcement
learning (RL) to balance the policy's performance and interaction costs. In
particular, online fine-tuning has become a commonly used method to correct the
erroneous estimates of out-of-distribution data learned in the offline training
phase. However, even limited online interactions can be inaccessible or
catastrophic for high-stake scenarios like healthcare and autonomous driving.
In this work, we introduce an interaction-free training scheme dubbed
Offline-with-Action-Preferences (OAP). The main insight is that, compared to
online fine-tuning, querying the preferences between pre-collected and learned
actions can be equally or even more helpful to the erroneous estimate problem.
By adaptively encouraging or suppressing policy constraint according to action
preferences, OAP could distinguish overestimation from beneficial policy
improvement and thus attains a more accurate evaluation of unseen data.
Theoretically, we prove a lower bound of the behavior policy's performance
improvement brought by OAP. Moreover, comprehensive experiments on the D4RL
benchmark and state-of-the-art algorithms demonstrate that OAP yields higher
(29% on average) scores, especially on challenging AntMaze tasks (98% higher).Comment: International Conference on Machine Learning 202
CORL: Research-oriented Deep Offline Reinforcement Learning Library
CORL is an open-source library that provides thoroughly benchmarked
single-file implementations of both deep offline and offline-to-online
reinforcement learning algorithms. It emphasizes a simple developing experience
with a straightforward codebase and a modern analysis tracking tool. In CORL,
we isolate methods implementation into separate single files, making
performance-relevant details easier to recognize. Additionally, an experiment
tracking feature is available to help log metrics, hyperparameters,
dependencies, and more to the cloud. Finally, we have ensured the reliability
of the implementations by benchmarking commonly employed D4RL datasets
providing a transparent source of results that can be reused for robust
evaluation tools such as performance profiles, probability of improvement, or
expected online performance.Comment: Conference on Neural Information Processing Systems (NeurIPS 2023)
Track on Datasets and Benchmarks. Source code at
https://github.com/corl-team/COR
Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
Contextual bandit algorithms have become popular for online recommendation
systems such as Digg, Yahoo! Buzz, and news recommendation in general.
\emph{Offline} evaluation of the effectiveness of new algorithms in these
applications is critical for protecting online user experiences but very
challenging due to their "partial-label" nature. Common practice is to create a
simulator which simulates the online environment for the problem at hand and
then run an algorithm against this simulator. However, creating simulator
itself is often difficult and modeling bias is usually unavoidably introduced.
In this paper, we introduce a \emph{replay} methodology for contextual bandit
algorithm evaluation. Different from simulator-based approaches, our method is
completely data-driven and very easy to adapt to different applications. More
importantly, our method can provide provably unbiased evaluations. Our
empirical results on a large-scale news article recommendation dataset
collected from Yahoo! Front Page conform well with our theoretical results.
Furthermore, comparisons between our offline replay and online bucket
evaluation of several contextual bandit algorithms show accuracy and
effectiveness of our offline evaluation method.Comment: 10 pages, 7 figures, revised from the published version at the WSDM
2011 conferenc
- …