933 research outputs found
Deriving and improving CMA-ES with Information geometric trust regions
CMA-ES is one of the most popular stochastic search algorithms.
It performs favourably in many tasks without the need of extensive
parameter tuning. The algorithm has many beneficial properties,
including automatic step-size adaptation, efficient covariance updates
that incorporates the current samples as well as the evolution
path and its invariance properties. Its update rules are composed
of well established heuristics where the theoretical foundations of
some of these rules are also well understood. In this paper we
will fully derive all CMA-ES update rules within the framework of
expectation-maximisation-based stochastic search algorithms using
information-geometric trust regions. We show that the use of the trust
region results in similar updates to CMA-ES for the mean and the
covariance matrix while it allows for the derivation of an improved
update rule for the step-size. Our new algorithm, Trust-Region Covariance
Matrix Adaptation Evolution Strategy (TR-CMA-ES) is
fully derived from first order optimization principles and performs
favourably in compare to standard CMA-ES algorithm
On Entropy Regularized Path Integral Control for Trajectory Optimization
In this article we present a generalised view on Path Integral Control (PIC)
methods. PIC refers to a particular class of policy search methods that are
closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a
restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems.
This class is unique in the sense that it can be solved explicitly to yield a
formal optimal state trajectory distribution. In this contribution we first
review the PIC theory and discuss related algorithms tailored to policy search
in general. We are able to identify a generic design strategy that relies on
the existence of an optimal state trajectory distribution and finds a
parametric policy by minimizing the cross entropy between the optimal and a
state trajectory distribution parametrized through its policy. Inspired by this
observation we then aim to formulate a SOC problem that shares traits with the
LSOC setting yet that covers a less restrictive class of problem formulations.
We refer to this SOC problem as Entropy Regularized Trajectory Optimization.
The problem is closely related to the Entropy Regularized Stochastic Optimal
Control setting which is lately often addressed by the Reinforcement Learning
(RL) community. We analyse the theoretical convergence behaviour of the
theoretical state trajectory distribution sequence and draw connections with
stochastic search methods tailored to classic optimization problems. Finally we
derive explicit updates and compare the implied Entropy Regularized PIC with
earlier work in the context of both PIC and RL for derivative-free trajectory
optimization
On entropy regularized Path Integral Control for trajectory optimization
In this article, we present a generalized view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly yielding a formal optimal state trajectory distribution. In this contribution, we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross-entropy between the optimal and a state trajectory distribution parametrized by a parametric stochastic policy. Inspired by this observation, we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is often addressed lately by the Reinforcement Learning (RL) community. We analyze the theoretical convergence behavior of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization
- …