14,163 research outputs found
Bayesian Optimization with Automatic Prior Selection for Data-Efficient Direct Policy Search
One of the most interesting features of Bayesian optimization for direct
policy search is that it can leverage priors (e.g., from simulation or from
previous tasks) to accelerate learning on a robot. In this paper, we are
interested in situations for which several priors exist but we do not know in
advance which one fits best the current situation. We tackle this problem by
introducing a novel acquisition function, called Most Likely Expected
Improvement (MLEI), that combines the likelihood of the priors and the expected
improvement. We evaluate this new acquisition function on a transfer learning
task for a 5-DOF planar arm and on a possibly damaged, 6-legged robot that has
to learn to walk on flat ground and on stairs, with priors corresponding to
different stairs and different kinds of damages. Our results show that MLEI
effectively identifies and exploits the priors, even when there is no obvious
match between the current situations and the priors.Comment: Accepted at ICRA 2018; 8 pages, 4 figures, 1 algorithm; Video at
https://youtu.be/xo8mUIZTvNE ; Spotlight ICRA presentation
https://youtu.be/iiVaV-U6Kq
Evolutionary Algorithms for Reinforcement Learning
There are two distinct approaches to solving reinforcement learning problems,
namely, searching in value function space and searching in policy space.
Temporal difference methods and evolutionary algorithms are well-known examples
of these approaches. Kaelbling, Littman and Moore recently provided an
informative survey of temporal difference methods. This article focuses on the
application of evolutionary algorithms to the reinforcement learning problem,
emphasizing alternative policy representations, credit assignment methods, and
problem-specific genetic operators. Strengths and weaknesses of the
evolutionary approach to reinforcement learning are presented, along with a
survey of representative applications
Fast Model Identification via Physics Engines for Data-Efficient Policy Search
This paper presents a method for identifying mechanical parameters of robots
or objects, such as their mass and friction coefficients. Key features are the
use of off-the-shelf physics engines and the adaptation of a Bayesian
optimization technique towards minimizing the number of real-world experiments
needed for model-based reinforcement learning. The proposed framework
reproduces in a physics engine experiments performed on a real robot and
optimizes the model's mechanical parameters so as to match real-world
trajectories. The optimized model is then used for learning a policy in
simulation, before real-world deployment. It is well understood, however, that
it is hard to exactly reproduce real trajectories in simulation. Moreover, a
near-optimal policy can be frequently found with an imperfect model. Therefore,
this work proposes a strategy for identifying a model that is just good enough
to approximate the value of a locally optimal policy with a certain confidence,
instead of wasting effort on identifying the most accurate model. Evaluations,
performed both in simulation and on a real robotic manipulation task, indicate
that the proposed strategy results in an overall time-efficient, integrated
model identification and learning solution, which significantly improves the
data-efficiency of existing policy search algorithms.Comment: IJCAI 1
Black-Box Data-efficient Policy Search for Robotics
The most data-efficient algorithms for reinforcement learning (RL) in
robotics are based on uncertain dynamical models: after each episode, they
first learn a dynamical model of the robot, then they use an optimization
algorithm to find a policy that maximizes the expected return given the model
and its uncertainties. It is often believed that this optimization can be
tractable only if analytical, gradient-based algorithms are used; however,
these algorithms require using specific families of reward functions and
policies, which greatly limits the flexibility of the overall approach. In this
paper, we introduce a novel model-based RL algorithm, called Black-DROPS
(Black-box Data-efficient RObot Policy Search) that: (1) does not impose any
constraint on the reward function or the policy (they are treated as
black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for
data-efficient RL in robotics, and (3) is as fast (or faster) than analytical
approaches when several cores are available. The key idea is to replace the
gradient-based optimization algorithm with a parallel, black-box algorithm that
takes into account the model uncertainties. We demonstrate the performance of
our new algorithm on two standard control benchmark problems (in simulation)
and a low-cost robotic manipulator (with a real robot).Comment: Accepted at the IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS) 2017; Code at
http://github.com/resibots/blackdrops; Video at http://youtu.be/kTEyYiIFGP
Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder
In this paper, we present a hierarchical path planning framework called SG-RL
(subgoal graphs-reinforcement learning), to plan rational paths for agents
maneuvering in continuous and uncertain environments. By "rational", we mean
(1) efficient path planning to eliminate first-move lags; (2) collision-free
and smooth for agents with kinematic constraints satisfied. SG-RL works in a
two-level manner. At the first level, SG-RL uses a geometric path-planning
method, i.e., Simple Subgoal Graphs (SSG), to efficiently find optimal abstract
paths, also called subgoal sequences. At the second level, SG-RL uses an RL
method, i.e., Least-Squares Policy Iteration (LSPI), to learn near-optimal
motion-planning policies which can generate kinematically feasible and
collision-free trajectories between adjacent subgoals. The first advantage of
the proposed method is that SSG can solve the limitations of sparse reward and
local minima trap for RL agents; thus, LSPI can be used to generate paths in
complex environments. The second advantage is that, when the environment
changes slightly (i.e., unexpected obstacles appearing), SG-RL does not need to
reconstruct subgoal graphs and replan subgoal sequences using SSG, since LSPI
can deal with uncertainties by exploiting its generalization ability to handle
changes in environments. Simulation experiments in representative scenarios
demonstrate that, compared with existing methods, SG-RL can work well on
large-scale maps with relatively low action-switching frequencies and shorter
path lengths, and SG-RL can deal with small changes in environments. We further
demonstrate that the design of reward functions and the types of training
environments are important factors for learning feasible policies.Comment: 20 page
- …