11 research outputs found
Approximate Frank-Wolfe Algorithms over Graph-structured Support Sets
In this paper, we propose approximate Frank-Wolfe (FW) algorithms to solve
convex optimization problems over graph-structured support sets where the
\textit{linear minimization oracle} (LMO) cannot be efficiently obtained in
general. We first demonstrate that two popular approximation assumptions
(\textit{additive} and \textit{multiplicative gap errors)}, are not valid for
our problem, in that no cheap gap-approximate LMO oracle exists in general.
Instead, a new \textit{approximate dual maximization oracle} (DMO) is proposed,
which approximates the inner product rather than the gap. When the objective is
-smooth, we prove that the standard FW method using a -approximate
DMO converges as in general, and as over a
-relaxation of the constraint set. Additionally, when the objective is
-strongly convex and the solution is unique, a variant of FW converges to
with the same per-iteration
complexity. Our empirical results suggest that even these improved bounds are
pessimistic, with significant improvement in recovering real-world images with
graph-structured sparsity.Comment: 30 pages, 8 figure
Linearly Convergent Frank-Wolfe with Backtracking Line-Search
Structured constraints in Machine Learning have recently brought the
Frank-Wolfe (FW) family of algorithms back in the spotlight. While the
classical FW algorithm has poor local convergence properties, the Away-steps
and Pairwise FW variants have emerged as improved variants with faster
convergence. However, these improved variants suffer from two practical
limitations: they require at each iteration to solve a 1-dimensional
minimization problem to set the step-size and also require the Frank-Wolfe
linear subproblems to be solved exactly. In this paper, we propose variants of
Away-steps and Pairwise FW that lift both restrictions simultaneously. The
proposed methods set the step-size based on a sufficient decrease condition,
and do not require prior knowledge of the objective. Furthermore, they inherit
all the favorable convergence properties of the exact line-search version,
including linear convergence for strongly convex functions over polytopes.
Benchmarks on different machine learning problems illustrate large performance
gains of the proposed variants
On Matching Pursuit and Coordinate Descent
Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers a generalized notion of directions. Exploiting the connection between the two algorithms, we present a unified analysis of both, providing affine invariant sublinear O(1/t) rates on smooth objectives and linear convergence on strongly convex objectives. As a byproduct of our affine invariant analysis of matching pursuit, our rates for steepest coordinate descent are the tightest known. Furthermore, we show the first accelerated convergence rate O(1/t^2) for matching pursuit and steepest coordinate descent on convex objectives