272 research outputs found
Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm
We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning (L/µ) 2 (where L is a bound on the smoothness and µ on the strong convexity) to a linear dependence on L/µ. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results. We also discuss importance sampling for SGD more broadly and show how it can improve convergence also in other scenarios. Our results are based on a connection we make between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods. In particular, we recast the randomized Kaczmarz algorithm as an instance of SGD, and apply our results to prove its exponential convergence, but to the solution of a weighted least squares problem rather than the original least squares problem. We then present a modified Kaczmarz algorithm with partially biased sampling which does converge to the original least squares solution with the same exponential convergence rate
Preasymptotic Convergence of Randomized Kaczmarz Method
Kaczmarz method is one popular iterative method for solving inverse problems,
especially in computed tomography. Recently, it was established that a
randomized version of the method enjoys an exponential convergence for
well-posed problems, and the convergence rate is determined by a variant of the
condition number. In this work, we analyze the preasymptotic convergence
behavior of the randomized Kaczmarz method, and show that the low-frequency
error (with respect to the right singular vectors) decays faster during first
iterations than the high-frequency error. Under the assumption that the inverse
solution is smooth (e.g., sourcewise representation), the result explains the
fast empirical convergence behavior, thereby shedding new insights into the
excellent performance of the randomized Kaczmarz method in practice. Further,
we propose a simple strategy to stabilize the asymptotic convergence of the
iteration by means of variance reduction. We provide extensive numerical
experiments to confirm the analysis and to elucidate the behavior of the
algorithms.Comment: 20 page
On the linear convergence of the stochastic gradient method with constant step-size
The strong growth condition (SGC) is known to be a sufficient condition for
linear convergence of the stochastic gradient method using a constant step-size
(SGM-CS). In this paper, we provide a necessary condition, for the
linear convergence of SGM-CS, that is weaker than SGC. Moreover, when this
necessary is violated up to a additive perturbation , we show that both
the projected stochastic gradient method using a constant step-size (PSGM-CS)
and the proximal stochastic gradient method exhibit linear convergence to a
noise dominated region, whose distance to the optimal solution is proportional
to
- …