3 research outputs found
Global Convergence and Stability of Stochastic Gradient Descent
In machine learning, stochastic gradient descent (SGD) is widely deployed to
train models using highly non-convex objectives with equally complex noise
models. Unfortunately, SGD theory often makes restrictive assumptions that fail
to capture the non-convexity of real problems, and almost entirely ignore the
complex noise models that exist in practice. In this work, we make substantial
progress on this shortcoming. First, we establish that SGD's iterates will
either globally converge to a stationary point or diverge under nearly
arbitrary nonconvexity and noise models. Under a slightly more restrictive
assumption on the joint behavior of the non-convexity and noise model that
generalizes current assumptions in the literature, we show that the objective
function cannot diverge, even if the iterates diverge. As a consequence of our
results, SGD can be applied to a greater range of stochastic optimization
problems with confidence about its global convergence behavior and stability