1,159 research outputs found
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Stochastic Gradient Descent (SGD) has played a central role in machine
learning. However, it requires a carefully hand-picked stepsize for fast
convergence, which is notoriously tedious and time-consuming to tune. Over the
last several years, a plethora of adaptive gradient-based algorithms have
emerged to ameliorate this problem. They have proved efficient in reducing the
labor of tuning in practice, but many of them lack theoretic guarantees even in
the convex setting. In this paper, we propose new surrogate losses to cast the
problem of learning the optimal stepsizes for the stochastic optimization of a
non-convex smooth objective function onto an online convex optimization
problem. This allows the use of no-regret online algorithms to compute optimal
stepsizes on the fly. In turn, this results in a SGD algorithm with self-tuned
stepsizes that guarantees convergence rates that are automatically adaptive to
the level of noise
Adaptive Stochastic Mirror Descent for Constrained Optimization
Mirror Descent (MD) is a well-known method of solving non-smooth convex
optimization problems. This paper analyzes the stochastic variant of MD with
adaptive stepsizes. Its convergence on average is shown to be faster than with
the fixed stepsizes and optimal in terms of lower bounds
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
Effective training of deep neural networks suffers from two main issues. The
first is that the parameter spaces of these models exhibit pathological
curvature. Recent methods address this problem by using adaptive
preconditioning for Stochastic Gradient Descent (SGD). These methods improve
convergence by adapting to the local geometry of parameter space. A second
issue is overfitting, which is typically addressed by early stopping. However,
recent work has demonstrated that Bayesian model averaging mitigates this
problem. The posterior can be sampled by using Stochastic Gradient Langevin
Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD
methods inefficient. Here, we propose combining adaptive preconditioners with
SGLD. In support of this idea, we give theoretical properties on asymptotic
convergence and predictive risk. We also provide empirical results for Logistic
Regression, Feedforward Neural Nets, and Convolutional Neural Nets,
demonstrating that our preconditioned SGLD method gives state-of-the-art
performance on these models.Comment: AAAI 201
- …