25,806 research outputs found
Momentum-Based Policy Gradient with Second-Order Information
Variance-reduced gradient estimators for policy gradient methods have been
one of the main focus of research in the reinforcement learning in recent years
as they allow acceleration of the estimation process. We propose a
variance-reduced policy-gradient method, called SHARP, which incorporates
second-order information into stochastic gradient descent (SGD) using momentum
with a time-varying learning rate. SHARP algorithm is parameter-free, achieving
-approximate first-order stationary point with
number of trajectories, while using a batch size of at each iteration.
Unlike most previous work, our proposed algorithm does not require importance
sampling which can compromise the advantage of variance reduction process.
Moreover, the variance of estimation error decays with the fast rate of
where is the number of iterations. Our extensive
experimental evaluations show the effectiveness of the proposed algorithm on
various control tasks and its advantage over the state of the art in practice
Policy Gradients for CVaR-Constrained MDPs
We study a risk-constrained version of the stochastic shortest path (SSP)
problem, where the risk measure considered is Conditional Value-at-Risk (CVaR).
We propose two algorithms that obtain a locally risk-optimal policy by
employing four tools: stochastic approximation, mini batches, policy gradients
and importance sampling. Both the algorithms incorporate a CVaR estimation
procedure, along the lines of Bardou et al. [2009], which in turn is based on
Rockafellar-Uryasev's representation for CVaR and utilize the likelihood ratio
principle for estimating the gradient of the sum of one cost function
(objective of the SSP) and the gradient of the CVaR of the sum of another cost
function (in the constraint of SSP). The algorithms differ in the manner in
which they approximate the CVaR estimates/necessary gradients - the first
algorithm uses stochastic approximation, while the second employ mini-batches
in the spirit of Monte Carlo methods. We establish asymptotic convergence of
both the algorithms. Further, since estimating CVaR is related to rare-event
simulation, we incorporate an importance sampling based variance reduction
scheme into our proposed algorithms
- …