861 research outputs found
Non-Asymptotic Convergence Analysis of Inexact Gradient Methods for Machine Learning Without Strong Convexity
Many recent applications in machine learning and data fitting call for the
algorithmic solution of structured smooth convex optimization problems.
Although the gradient descent method is a natural choice for this task, it
requires exact gradient computations and hence can be inefficient when the
problem size is large or the gradient is difficult to evaluate. Therefore,
there has been much interest in inexact gradient methods (IGMs), in which an
efficiently computable approximate gradient is used to perform the update in
each iteration. Currently, non-asymptotic linear convergence results for IGMs
are typically established under the assumption that the objective function is
strongly convex, which is not satisfied in many applications of interest; while
linear convergence results that do not require the strong convexity assumption
are usually asymptotic in nature. In this paper, we combine the best of these
two types of results and establish---under the standard assumption that the
gradient approximation errors decrease linearly to zero---the non-asymptotic
linear convergence of IGMs when applied to a class of structured convex
optimization problems. Such a class covers settings where the objective
function is not necessarily strongly convex and includes the least squares and
logistic regression problems. We believe that our techniques will find further
applications in the non-asymptotic convergence analysis of other first-order
methods
Implicit Langevin Algorithms for Sampling From Log-concave Densities
For sampling from a log-concave density, we study implicit integrators
resulting from -method discretization of the overdamped Langevin
diffusion stochastic differential equation. Theoretical and algorithmic
properties of the resulting sampling methods for and a
range of step sizes are established. Our results generalize and extend prior
works in several directions. In particular, for , we prove
geometric ergodicity and stability of the resulting methods for all step sizes.
We show that obtaining subsequent samples amounts to solving a strongly-convex
optimization problem, which is readily achievable using one of numerous
existing methods. Numerical examples supporting our theoretical analysis are
also presented
Accelerating Incremental Gradient Optimization with Curvature Information
This paper studies an acceleration technique for incremental aggregated
gradient ({\sf IAG}) method through the use of \emph{curvature} information for
solving strongly convex finite sum optimization problems. These optimization
problems of interest arise in large-scale learning applications. Our technique
utilizes a curvature-aided gradient tracking step to produce accurate gradient
estimates incrementally using Hessian information. We propose and analyze two
methods utilizing the new technique, the curvature-aided IAG ({\sf CIAG})
method and the accelerated CIAG ({\sf A-CIAG}) method, which are analogous to
gradient method and Nesterov's accelerated gradient method, respectively.
Setting to be the condition number of the objective function, we prove
the linear convergence rates of for
the {\sf CIAG} method, and for the {\sf
A-CIAG} method, where are constants inversely proportional to
the distance between the initial point and the optimal solution. When the
initial iterate is close to the optimal solution, the linear convergence
rates match with the gradient and accelerated gradient method, albeit {\sf
CIAG} and {\sf A-CIAG} operate in an incremental setting with strictly lower
computation complexity. Numerical experiments confirm our findings. The source
codes used for this paper can be found on
\url{http://github.com/hoitowai/ciag/}.Comment: 22 pages, 3 figures, 3 tables. Accepted by Computational Optimization
and Applications, to appea
- …