1 research outputs found
Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence
In this paper, we study stochastic non-convex optimization with non-convex
random functions. Recent studies on non-convex optimization revolve around
establishing second-order convergence, i.e., converging to a nearly
second-order optimal stationary points. However, existing results on stochastic
non-convex optimization are limited, especially with a high probability
second-order convergence. We propose a novel updating step (named NCG-S) by
leveraging a stochastic gradient and a noisy negative curvature of a stochastic
Hessian, where the stochastic gradient and Hessian are based on a proper
mini-batch of random functions. Building on this step, we develop two
algorithms and establish their high probability second-order convergence. To
the best of our knowledge, the proposed stochastic algorithms are the first
with a second-order convergence in {\it high probability} and a time complexity
that is {\it almost linear} in the problem's dimensionality.Comment: This short paper will appear at NIPS 2017 Optimization of Machine
Learning Workshop. Partial results are presented in arXiv:1709.08571. The
second version corrects a statement regarding previous wor