140 research outputs found
A continuous-time analysis of distributed stochastic gradient
We analyze the effect of synchronization on distributed stochastic gradient
algorithms. By exploiting an analogy with dynamical models of biological quorum
sensing -- where synchronization between agents is induced through
communication with a common signal -- we quantify how synchronization can
significantly reduce the magnitude of the noise felt by the individual
distributed agents and by their spatial mean. This noise reduction is in turn
associated with a reduction in the smoothing of the loss function imposed by
the stochastic gradient approximation. Through simulations on model non-convex
objectives, we demonstrate that coupling can stabilize higher noise levels and
improve convergence. We provide a convergence analysis for strongly convex
functions by deriving a bound on the expected deviation of the spatial mean of
the agents from the global minimizer for an algorithm based on quorum sensing,
the same algorithm with momentum, and the Elastic Averaging SGD (EASGD)
algorithm. We discuss extensions to new algorithms which allow each agent to
broadcast its current measure of success and shape the collective computation
accordingly. We supplement our theoretical analysis with numerical experiments
on convolutional neural networks trained on the CIFAR-10 dataset, where we note
a surprising regularizing property of EASGD even when applied to the
non-distributed case. This observation suggests alternative second-order
in-time algorithms for non-distributed optimization that are competitive with
momentum methods.Comment: 9/14/19 : Final version, accepted for publication in Neural
Computation. 4/7/19 : Significant edits: addition of simulations, deep
network results, and revisions throughout. 12/28/18: Initial submissio
Musings on Deep Learning: Properties of SGD
[previously titled "Theory of Deep Learning III: Generalization Properties of SGD"] In Theory III we characterize with a mix of theory and experiments the generalization properties of Stochastic Gradient Descent in overparametrized deep convolutional networks. We show that Stochastic Gradient Descent (SGD) selects with high probability solutions that 1) have zero (or small) empirical error, 2) are degenerate as shown in Theory II and 3) have maximum generalization.This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF - 1231216. H.M. is supported in part by ARO Grant W911NF-15-1- 0385
Convergence and concentration properties of constant step-size SGD through Markov chains
We consider the optimization of a smooth and strongly convex objective using
constant step-size stochastic gradient descent (SGD) and study its properties
through the prism of Markov chains. We show that, for unbiased gradient
estimates with mildly controlled variance, the iteration converges to an
invariant distribution in total variation distance. We also establish this
convergence in Wasserstein-2 distance in a more general setting compared to
previous work. Thanks to the invariance property of the limit distribution, our
analysis shows that the latter inherits sub-Gaussian or sub-exponential
concentration properties when these hold true for the gradient. This allows the
derivation of high-confidence bounds for the final estimate. Finally, under
such conditions in the linear case, we obtain a dimension-free deviation bound
for the Polyak-Ruppert average of a tail sequence. All our results are
non-asymptotic and their consequences are discussed through a few applications
- …