9,186 research outputs found
Efficiently Combining Human Demonstrations and Interventions for Safe Training of Autonomous Systems in Real-Time
This paper investigates how to utilize different forms of human interaction
to safely train autonomous systems in real-time by learning from both human
demonstrations and interventions. We implement two components of the
Cycle-of-Learning for Autonomous Systems, which is our framework for combining
multiple modalities of human interaction. The current effort employs human
demonstrations to teach a desired behavior via imitation learning, then
leverages intervention data to correct for undesired behaviors produced by the
imitation learner to teach novel tasks to an autonomous agent safely, after
only minutes of training. We demonstrate this method in an autonomous perching
task using a quadrotor with continuous roll, pitch, yaw, and throttle commands
and imagery captured from a downward-facing camera in a high-fidelity simulated
environment. Our method improves task completion performance for the same
amount of human interaction when compared to learning from demonstrations
alone, while also requiring on average 32% less data to achieve that
performance. This provides evidence that combining multiple modes of human
interaction can increase both the training speed and overall performance of
policies for autonomous systems.Comment: 9 pages, 6 figure
Interactive Imitation Learning in Robotics: A Survey
Interactive Imitation Learning (IIL) is a branch of Imitation Learning (IL)
where human feedback is provided intermittently during robot execution allowing
an online improvement of the robot's behavior. In recent years, IIL has
increasingly started to carve out its own space as a promising data-driven
alternative for solving complex robotic tasks. The advantages of IIL are its
data-efficient, as the human feedback guides the robot directly towards an
improved behavior, and its robustness, as the distribution mismatch between the
teacher and learner trajectories is minimized by providing feedback directly
over the learner's trajectories. Nevertheless, despite the opportunities that
IIL presents, its terminology, structure, and applicability are not clear nor
unified in the literature, slowing down its development and, therefore, the
research of innovative formulations and discoveries. In this article, we
attempt to facilitate research in IIL and lower entry barriers for new
practitioners by providing a survey of the field that unifies and structures
it. In addition, we aim to raise awareness of its potential, what has been
accomplished and what are still open research questions. We organize the most
relevant works in IIL in terms of human-robot interaction (i.e., types of
feedback), interfaces (i.e., means of providing feedback), learning (i.e.,
models learned from feedback and function approximators), user experience
(i.e., human perception about the learning process), applications, and
benchmarks. Furthermore, we analyze similarities and differences between IIL
and RL, providing a discussion on how the concepts offline, online, off-policy
and on-policy learning should be transferred to IIL from the RL literature. We
particularly focus on robotic applications in the real world and discuss their
implications, limitations, and promising future areas of research
Human Engagement Providing Evaluative and Informative Advice for Interactive Reinforcement Learning
Reinforcement learning is an approach used by intelligent agents to
autonomously learn new skills. Although reinforcement learning has been
demonstrated to be an effective learning approach in several different
contexts, a common drawback exhibited is the time needed in order to
satisfactorily learn a task, especially in large state-action spaces. To
address this issue, interactive reinforcement learning proposes the use of
externally-sourced information in order to speed up the learning process. Up to
now, different information sources have been used to give advice to the learner
agent, among them human-sourced advice. When interacting with a learner agent,
humans may provide either evaluative or informative advice. From the agent's
perspective these styles of interaction are commonly referred to as
reward-shaping and policy-shaping respectively. Evaluation requires the human
to provide feedback on the prior action performed, while informative advice
they provide advice on the best action to select for a given situation. Prior
research has focused on the effect of human-sourced advice on the interactive
reinforcement learning process, specifically aiming to improve the learning
speed of the agent, while reducing the engagement with the human. This work
presents an experimental setup for a human-trial designed to compare the
methods people use to deliver advice in term of human engagement. Obtained
results show that users giving informative advice to the learner agents provide
more accurate advice, are willing to assist the learner agent for a longer
time, and provide more advice per episode. Additionally, self-evaluation from
participants using the informative approach has indicated that the agent's
ability to follow the advice is higher, and therefore, they feel their own
advice to be of higher accuracy when compared to people providing evaluative
advice.Comment: 33 pages, 15 figure
- …