Geometry and convergence of natural policy gradient methods

We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the trajectories in state-action space are solutions of gradient flows with respect to Hessian geometries, based on which we obtain global convergence guarantees and convergence rates. In particular, we show linear convergence for unregularized and regularized NPG flows with the metrics proposed by Kakade and Morimura and co-authors by observing that these arise from the Hessian geometries of conditional entropy and entropy respectively. Further, we obtain sublinear convergence rates for Hessian geometries arising from other convex functions like log-barriers. Finally, we interpret the discrete-time NPG methods with regularized rewards as inexact Newton methods if the NPG is defined with respect to the Hessian geometry of the regularizer. This yields local quadratic convergence rates of these methods for step size equal to the penalization strength.Comment: 33 pages, 5 figures, under revie

Design and implementation of symbolic algorithms for the computation of generalized asymptotes

In this paper we present two algorithms for computing the g-asymptotes or generalized asymptotes, of a plane algebraic curve, C , implicitly or parametrically defined. The asymptotes of a curve C reflect the status of C at points with sufficiently large coordinates. It is well known that an asymptote of a curve C is a line such that the distance between C and the line approaches zero as they tend to infinity. However, a curve C may have more general curves than lines describing the status of C at infinity. These curves are known as g-asymptotes or generalized asymptotes. The pseudocodes of these algorithms are presented, as well as the corresponding implementations. For this purpose, we use the algebra software Maple. A comparative analysis of the algorithms is carried out, based on some properties of the input curves and their results to analyze the efficiency of the algorithms and to establish comparative criteria. The results presented in this paper are a starting point to generalize this study to surfaces or to curves defined by a non-rational parametrization, as well as to improve the efficiency of the algorithms. Additionally, the methods developed can provide a new and different approach in prediction (regression) or classification algorithms in the machine learning field.Agencia Estatal de Investigació