183 research outputs found
Recursive Frank-Wolfe algorithms
In the last decade there has been a resurgence of interest in Frank-Wolfe
(FW) style methods for optimizing a smooth convex function over a polytope.
Examples of recently developed techniques include {\em Decomposition-invariant
Conditional Gradient} (DiCG), {\em Blended Condition Gradient} (BCG), and {\em
Frank-Wolfe with in-face directions} (IF-FW) methods. We introduce two
extensions of these techniques. First, we augment DiCG with the {\em working
set} strategy, and show how to optimize over the working set using {\em shadow
simplex steps}. Second, we generalize in-face Frank-Wolfe directions to
polytopes in which faces cannot be efficiently computed, and also describe a
generic recursive procedure that can be used in conjunction with several
FW-style techniques. Experimental results indicate that these extensions are
capable of speeding up original algorithms by orders of magnitude for certain
applications
Walking in the Shadow: A New Perspective on Descent Directions for Constrained Minimization
Descent directions such as movement towards Frank-Wolfe vertices, away steps,
in-face away steps and pairwise directions have been an important design
consideration in conditional gradient descent (CGD) variants. In this work, we
attempt to demystify the impact of movement in these directions towards
attaining constrained minimizers. The best local direction of descent is the
directional derivative of the projection of the gradient, which we refer to as
the of the gradient. We show that the continuous-time
dynamics of moving in the shadow are equivalent to those of PGD however
non-trivial to discretize. By projecting gradients in PGD, one not only ensures
feasibility but is also able to "wrap" around the convex region. We show that
Frank-Wolfe (FW) vertices in fact recover the maximal wrap one can obtain by
projecting gradients, thus providing a new perspective on these steps. We also
claim that the shadow steps give the best direction of descent emanating from
the convex hull of all possible away-steps. Viewing PGD movements in terms of
shadow steps gives linear convergence, dependent on the number of faces. We
combine these insights into a novel - method that uses FW
steps (i.e., wrap around the polytope) and shadow steps (i.e., optimal local
descent direction), while enjoying linear convergence. Our analysis develops
properties of the curve formed by projecting a line on a polytope, which may be
of independent interest, while providing a unifying view of various descent
directions in the CGD literature
- …