24,028 research outputs found
Guarded Policy Optimization with Imperfect Online Demonstrations
The Teacher-Student Framework (TSF) is a reinforcement learning setting where
a teacher agent guards the training of a student agent by intervening and
providing online demonstrations. Assuming optimal, the teacher policy has the
perfect timing and capability to intervene in the learning process of the
student agent, providing safety guarantee and exploration guidance.
Nevertheless, in many real-world settings it is expensive or even impossible to
obtain a well-performing teacher policy. In this work, we relax the assumption
of a well-performing teacher and develop a new method that can incorporate
arbitrary teacher policies with modest or inferior performance. We instantiate
an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared
Control (TS2C), which incorporates teacher intervention based on
trajectory-based value estimation. Theoretical analysis validates that the
proposed TS2C algorithm attains efficient exploration and substantial safety
guarantee without being affected by the teacher's own performance. Experiments
on various continuous control tasks show that our method can exploit teacher
policies at different performance levels while maintaining a low training cost.
Moreover, the student policy surpasses the imperfect teacher policy in terms of
higher accumulated reward in held-out testing environments. Code is available
at https://metadriverse.github.io/TS2C.Comment: Accepted at ICLR 2023 (top 25%
Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
Adversarial Imitation Learning alternates between learning a discriminator --
which tells apart expert's demonstrations from generated ones -- and a
generator's policy to produce trajectories that can fool this discriminator.
This alternated optimization is known to be delicate in practice since it
compounds unstable adversarial training with brittle and sample-inefficient
reinforcement learning. We propose to remove the burden of the policy
optimization steps by leveraging a novel discriminator formulation.
Specifically, our discriminator is explicitly conditioned on two policies: the
one from the previous generator's iteration and a learnable policy. When
optimized, this discriminator directly learns the optimal generator's policy.
Consequently, our discriminator's update solves the generator's optimization
problem for free: learning a policy that imitates the expert does not require
an additional optimization loop. This formulation effectively cuts by half the
implementation and computational burden of Adversarial Imitation Learning
algorithms by removing the Reinforcement Learning phase altogether. We show on
a variety of tasks that our simpler approach is competitive to prevalent
Imitation Learning methods
Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets
Imitation learning has traditionally been applied to learn a single task from
demonstrations thereof. The requirement of structured and isolated
demonstrations limits the scalability of imitation learning approaches as they
are difficult to apply to real-world scenarios, where robots have to be able to
execute a multitude of tasks. In this paper, we propose a multi-modal imitation
learning framework that is able to segment and imitate skills from unlabelled
and unstructured demonstrations by learning skill segmentation and imitation
learning jointly. The extensive simulation results indicate that our method can
efficiently separate the demonstrations into individual skills and learn to
imitate them using a single multi-modal policy. The video of our experiments is
available at http://sites.google.com/view/nips17intentionganComment: Paper accepted to NIPS 201
- …