79,305 research outputs found
Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent
Although the optimization objectives for learning neural networks are highly
non-convex, gradient-based methods have been wildly successful at learning
neural networks in practice. This juxtaposition has led to a number of recent
studies on provable guarantees for neural networks trained by gradient descent.
Unfortunately, the techniques in these works are often highly specific to the
problem studied in each setting, relying on different assumptions on the
distribution, optimization parameters, and network architectures, making it
difficult to generalize across different settings. In this work, we propose a
unified non-convex optimization framework for the analysis of neural network
training. We introduce the notions of proxy convexity and proxy
Polyak-Lojasiewicz (PL) inequalities, which are satisfied if the original
objective function induces a proxy objective function that is implicitly
minimized when using gradient methods. We show that stochastic gradient descent
(SGD) on objectives satisfying proxy convexity or the proxy PL inequality leads
to efficient guarantees for proxy objective functions. We further show that
many existing guarantees for neural networks trained by gradient descent can be
unified through proxy convexity and proxy PL inequalities.Comment: 15 page
- …