1,412 research outputs found
Frank-Wolfe Algorithms for Saddle Point Problems
We extend the Frank-Wolfe (FW) optimization algorithm to solve constrained
smooth convex-concave saddle point (SP) problems. Remarkably, the method only
requires access to linear minimization oracles. Leveraging recent advances in
FW optimization, we provide the first proof of convergence of a FW-type saddle
point solver over polytopes, thereby partially answering a 30 year-old
conjecture. We also survey other convergence results and highlight gaps in the
theoretical underpinnings of FW-style algorithms. Motivating applications
without known efficient alternatives are explored through structured prediction
with combinatorial penalties as well as games over matching polytopes involving
an exponential number of constraints.Comment: Appears in: Proceedings of the 20th International Conference on
Artificial Intelligence and Statistics (AISTATS 2017). 39 page
Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
In the first part of this dissertation research, we develop a modular
framework that can serve as a recipe for constructing and analyzing iterative
algorithms for convex optimization. Specifically, our work casts optimization
as iteratively playing a two-player zero-sum game. Many existing optimization
algorithms including Frank-Wolfe and Nesterov's acceleration methods can be
recovered from the game by pitting two online learners with appropriate
strategies against each other. Furthermore, the sum of the weighted average
regrets of the players in the game implies the convergence rate. As a result,
our approach provides simple alternative proofs to these algorithms. Moreover,
we demonstrate that our approach of optimization as iteratively playing a game
leads to three new fast Frank-Wolfe-like algorithms for some constraint sets,
which further shows that our framework is indeed generic, modular, and
easy-to-use.
In the second part, we develop a modular analysis of provable acceleration
via Polyak's momentum for certain problems, which include solving the classical
strongly quadratic convex problems, training a wide ReLU network under the
neural tangent kernel regime, and training a deep linear network with an
orthogonal initialization. We develop a meta theorem and show that when
applying Polyak's momentum for these problems, the induced dynamics exhibit a
form where we can directly apply our meta theorem.
In the last part of the dissertation, we show another advantage of the use of
Polyak's momentum -- it facilitates fast saddle point escape in smooth
non-convex optimization. This result, together with those of the second part,
sheds new light on Polyak's momentum in modern non-convex optimization and deep
learning.Comment: PhD dissertation at Georgia Tech. arXiv admin note: text overlap with
arXiv:2010.0161
Convex Optimization: Algorithms and Complexity
This monograph presents the main complexity theorems in convex optimization
and their corresponding algorithms. Starting from the fundamental theory of
black-box optimization, the material progresses towards recent advances in
structural optimization and stochastic optimization. Our presentation of
black-box optimization, strongly influenced by Nesterov's seminal book and
Nemirovski's lecture notes, includes the analysis of cutting plane methods, as
well as (accelerated) gradient descent schemes. We also pay special attention
to non-Euclidean settings (relevant algorithms include Frank-Wolfe, mirror
descent, and dual averaging) and discuss their relevance in machine learning.
We provide a gentle introduction to structural optimization with FISTA (to
optimize a sum of a smooth and a simple non-smooth term), saddle-point mirror
prox (Nemirovski's alternative to Nesterov's smoothing), and a concise
description of interior point methods. In stochastic optimization we discuss
stochastic gradient descent, mini-batches, random coordinate descent, and
sublinear algorithms. We also briefly touch upon convex relaxation of
combinatorial problems and the use of randomness to round solutions, as well as
random walks based methods.Comment: A previous version of the manuscript was titled "Theory of Convex
Optimization for Machine Learning
Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
Optimization is essential in machine learning, statistics, and data science. Among the first-order optimization algorithms, the popular ones include the Frank-Wolfe method, Nesterov's accelerated methods, and Polyak's momentum. While theoretical analysis of the Frank-Wolfe method and Nesterov's methods are available in the literature, the analysis can be quite complicated or less intuitive. Polyak's momentum, on the other hand, is widely used in training neural networks and is currently the default choice of momentum in Pytorch and Tensorflow. It is widely observed that Polyak's momentum helps to train a neural network faster, compared with the case without momentum. However, there are very few examples that exhibit a provable acceleration via Polyak's momentum, compared to vanilla gradient descent. There is an apparent gap between the theory and the practice of Polyak's momentum.
In the first part of this dissertation research, we develop a modular framework that can serve as a recipe for constructing and analyzing iterative algorithms for convex optimization. Specifically, our work casts optimization as iteratively playing a two-player zero-sum game. Many existing optimization algorithms including Frank-Wolfe and Nesterov's acceleration methods can be recovered from the game by pitting two online learners with appropriate strategies against each other. Furthermore, the sum of the weighted average regrets of the players in the game implies the convergence rate. As a result, our approach provides simple alternative proofs to these algorithms. Moreover, we demonstrate that our approach of ``optimization as iteratively playing a game'' leads to three new fast Frank-Wolfe-like algorithms for some constraint sets, which further shows that our framework is indeed generic, modular, and easy-to-use.
In the second part, we develop a modular analysis of provable acceleration via Polyak's momentum for certain problems, which include solving the classical strongly quadratic convex problems, training a wide ReLU network under the neural tangent kernel regime, and training a deep linear network with an orthogonal initialization. We develop a meta theorem and show that when applying Polyak’s momentum for these problems, the induced dynamics exhibit a form where we can directly apply our meta theorem.
In the last part of the dissertation, we show another advantage of the use of Polyak's momentum --- it facilitates fast saddle point escape in smooth non-convex optimization. This result, together with those of the second part, sheds new light on Polyak's momentum in modern non-convex optimization and deep learning.Ph.D
Convex optimization over intersection of simple sets: improved convergence rate guarantees via an exact penalty approach
We consider the problem of minimizing a convex function over the intersection
of finitely many simple sets which are easy to project onto. This is an
important problem arising in various domains such as machine learning. The main
difficulty lies in finding the projection of a point in the intersection of
many sets. Existing approaches yield an infeasible point with an
iteration-complexity of for nonsmooth problems with no
guarantees on the in-feasibility. By reformulating the problem through exact
penalty functions, we derive first-order algorithms which not only guarantees
that the distance to the intersection is small but also improve the complexity
to and for smooth functions. For
composite and smooth problems, this is achieved through a saddle-point
reformulation where the proximal operators required by the primal-dual
algorithms can be computed in closed form. We illustrate the benefits of our
approach on a graph transduction problem and on graph matching
- …