4,324 research outputs found
Reactive Reinforcement Learning in Asynchronous Environments
The relationship between a reinforcement learning (RL) agent and an
asynchronous environment is often ignored. Frequently used models of the
interaction between an agent and its environment, such as Markov Decision
Processes (MDP) or Semi-Markov Decision Processes (SMDP), do not capture the
fact that, in an asynchronous environment, the state of the environment may
change during computation performed by the agent. In an asynchronous
environment, minimizing reaction time---the time it takes for an agent to react
to an observation---also minimizes the time in which the state of the
environment may change following observation. In many environments, the
reaction time of an agent directly impacts task performance by permitting the
environment to transition into either an undesirable terminal state or a state
where performing the chosen action is inappropriate. We propose a class of
reactive reinforcement learning algorithms that address this problem of
asynchronous environments by immediately acting after observing new state
information. We compare a reactive SARSA learning algorithm with the
conventional SARSA learning algorithm on two asynchronous robotic tasks
(emergency stopping and impact prevention), and show that the reactive RL
algorithm reduces the reaction time of the agent by approximately the duration
of the algorithm's learning update. This new class of reactive algorithms may
facilitate safer control and faster decision making without any change to
standard learning guarantees.Comment: 11 pages, 7 figures, currently under journal peer revie
Reactive Reinforcement Learning in Asynchronous Environments
The relationship between a reinforcement learning (RL) agent and an asynchronous environment is often ignored. Frequently used models of the interaction between an agent and its environment, such as Markov Decision Processes (MDP) or Semi-Markov Decision Processes (SMDP), do not capture the fact that, in an asynchronous environment, the state of the environment may change during computation performed by the agent. In an asynchronous environment, minimizing reaction time—the time it takes for an agent to react to an observation—also minimizes the time in which the state of the environment may change following observation. In many environments, the reaction time of an agent directly impacts task performance by permitting the environment to transition into either an undesirable terminal state or a state where performing the chosen action is inappropriate. We propose a class of reactive reinforcement learning algorithms that address this problem of asynchronous environments by immediately acting after observing new state information. We compare a reactive SARSA learning algorithm with the conventional SARSA learning algorithm on two asynchronous robotic tasks (emergency stopping and impact prevention), and show that the reactive RL algorithm reduces the reaction time of the agent by approximately the duration of the algorithm's learning update. This new class of reactive algorithms may facilitate safer control and faster decision making without any change to standard learning guarantees
Neural Network Memory Architectures for Autonomous Robot Navigation
This paper highlights the significance of including memory structures in
neural networks when the latter are used to learn perception-action loops for
autonomous robot navigation. Traditional navigation approaches rely on global
maps of the environment to overcome cul-de-sacs and plan feasible motions. Yet,
maintaining an accurate global map may be challenging in real-world settings. A
possible way to mitigate this limitation is to use learning techniques that
forgo hand-engineered map representations and infer appropriate control
responses directly from sensed information. An important but unexplored aspect
of such approaches is the effect of memory on their performance. This work is a
first thorough study of memory structures for deep-neural-network-based robot
navigation, and offers novel tools to train such networks from supervision and
quantify their ability to generalize to unseen scenarios. We analyze the
separation and generalization abilities of feedforward, long short-term memory,
and differentiable neural computer networks. We introduce a new method to
evaluate the generalization ability by estimating the VC-dimension of networks
with a final linear readout layer. We validate that the VC estimates are good
predictors of actual test performance. The reported method can be applied to
deep learning problems beyond robotics
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems
This paper was motivated by the problem of how to make robots fuse and
transfer their experience so that they can effectively use prior knowledge and
quickly adapt to new environments. To address the problem, we present a
learning architecture for navigation in cloud robotic systems: Lifelong
Federated Reinforcement Learning (LFRL). In the work, We propose a knowledge
fusion algorithm for upgrading a shared model deployed on the cloud. Then,
effective transfer learning methods in LFRL are introduced. LFRL is consistent
with human cognitive science and fits well in cloud robotic systems.
Experiments show that LFRL greatly improves the efficiency of reinforcement
learning for robot navigation. The cloud robotic system deployment also shows
that LFRL is capable of fusing prior knowledge. In addition, we release a cloud
robotic navigation-learning website based on LFRL
Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learning
We present an approach for mobile robots to learn to navigate in dynamic
environments with pedestrians via raw depth inputs, in a socially compliant
manner. To achieve this, we adopt a generative adversarial imitation learning
(GAIL) strategy, which improves upon a pre-trained behavior cloning policy. Our
approach overcomes the disadvantages of previous methods, as they heavily
depend on the full knowledge of the location and velocity information of nearby
pedestrians, which not only requires specific sensors, but also the extraction
of such state information from raw sensory input could consume much computation
time. In this paper, our proposed GAIL-based model performs directly on raw
depth inputs and plans in real-time. Experiments show that our GAIL-based
approach greatly improves the safety and efficiency of the behavior of mobile
robots from pure behavior cloning. The real-world deployment also shows that
our method is capable of guiding autonomous vehicles to navigate in a socially
compliant manner directly through raw depth inputs. In addition, we release a
simulation plugin for modeling pedestrian behaviors based on the social force
model.Comment: ICRA 2018 camera-ready version. 7 pages, video link:
https://www.youtube.com/watch?v=0hw0GD3lkA
- …