155,678 research outputs found
Momentum-based variance reduction in non-convex SGD
Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent
in non-convex problems, providing the first algorithms to improve upon the converge rate of stochastic
gradient descent for finding first-order critical points. However, variance reduction techniques typically require carefully tuned learning rates and willingness to use excessively large “mega-batches” in order to achieve their improved results. We present a new algorithm, Storm, that does not require any batches and makes use of adaptive learning rates, enabling simpler implementation and less hyperparameter tuning. Our technique for removing the batches uses a variant of momentum to achieve variance reduction in non-convex optimization. On smooth losses F, Storm finds a point x with E[k∇F(x)k] ≤ O(1 /√ T + σ^1/3 /T^1/3) in T iterations with σ^2 variance in the gradients, matching the optimal rate and without requiring knowledge of σ.https://arxiv.org/pdf/1905.10018.pdfPublished versio
Natural selection. II. Developmental variability and evolutionary rate
In classical evolutionary theory, genetic variation provides the source of
heritable phenotypic variation on which natural selection acts. Against this
classical view, several theories have emphasized that developmental variability
and learning enhance nonheritable phenotypic variation, which in turn can
accelerate evolutionary response. In this paper, I show how developmental
variability alters evolutionary dynamics by smoothing the landscape that
relates genotype to fitness. In a fitness landscape with multiple peaks and
valleys, developmental variability can smooth the landscape to provide a
directly increasing path of fitness to the highest peak. Developmental
variability also allows initial survival of a genotype in response to novel or
extreme environmental challenge, providing an opportunity for subsequent
adaptation. This initial survival advantage arises from the way in which
developmental variability smooths and broadens the fitness landscape.
Ultimately, the synergism between developmental processes and genetic variation
sets evolutionary rate
- …