136,484 research outputs found
Projection-Free Methods for Solving Convex Bilevel Optimization Problems
When faced with multiple minima of an "inner-level" convex optimization
problem, the convex bilevel optimization problem selects an optimal solution
which also minimizes an auxiliary "outer-level" convex objective of interest.
Bilevel optimization requires a different approach compared to single-level
optimization problems since the set of minimizers for the inner-level objective
is not given explicitly. In this paper, we propose new projection-free methods
for convex bilevel optimization which require only a linear optimization oracle
over the base domain. We provide convergence guarantees for both inner- and
outer-level objectives that hold under our proposed projection-free methods. In
particular, we highlight how our guarantees are affected by the presence or
absence of an optimal dual solution. Lastly, we conduct numerical experiments
that demonstrate the performance of the proposed methods
Accelerating two projection methods via perturbations with application to Intensity-Modulated Radiation Therapy
Constrained convex optimization problems arise naturally in many real-world
applications. One strategy to solve them in an approximate way is to translate
them into a sequence of convex feasibility problems via the recently developed
level set scheme and then solve each feasibility problem using projection
methods. However, if the problem is ill-conditioned, projection methods often
show zigzagging behavior and therefore converge slowly.
To address this issue, we exploit the bounded perturbation resilience of the
projection methods and introduce two new perturbations which avoid zigzagging
behavior. The first perturbation is in the spirit of -step methods and uses
gradient information from previous iterates. The second uses the approach of
surrogate constraint methods combined with relaxed, averaged projections.
We apply two different projection methods in the unperturbed version, as well
as the two perturbed versions, to linear feasibility problems along with
nonlinear optimization problems arising from intensity-modulated radiation
therapy (IMRT) treatment planning. We demonstrate that for all the considered
problems the perturbations can significantly accelerate the convergence of the
projection methods and hence the overall procedure of the level set scheme. For
the IMRT optimization problems the perturbed projection methods found an
approximate solution up to 4 times faster than the unperturbed methods while at
the same time achieving objective function values which were 0.5 to 5.1% lower.Comment: Accepted for publication in Applied Mathematics & Optimizatio
Inexact Model: A Framework for Optimization and Variational Inequalities
In this paper we propose a general algorithmic framework for first-order
methods in optimization in a broad sense, including minimization problems,
saddle-point problems and variational inequalities. This framework allows to
obtain many known methods as a special case, the list including accelerated
gradient method, composite optimization methods, level-set methods, proximal
methods. The idea of the framework is based on constructing an inexact model of
the main problem component, i.e. objective function in optimization or operator
in variational inequalities. Besides reproducing known results, our framework
allows to construct new methods, which we illustrate by constructing a
universal method for variational inequalities with composite structure. This
method works for smooth and non-smooth problems with optimal complexity without
a priori knowledge of the problem smoothness. We also generalize our framework
for strongly convex objectives and strongly monotone variational inequalities.Comment: 41 page
Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem
In this paper, we study a class of stochastic bilevel optimization problems,
also known as stochastic simple bilevel optimization, where we minimize a
smooth stochastic objective function over the optimal solution set of another
stochastic convex optimization problem. We introduce novel stochastic bilevel
optimization methods that locally approximate the solution set of the
lower-level problem via a stochastic cutting plane, and then run a conditional
gradient update with variance reduction techniques to control the error induced
by using stochastic gradients. For the case that the upper-level function is
convex, our method requires
stochastic
oracle queries to obtain a solution that is -optimal for the
upper-level and -optimal for the lower-level. This guarantee
improves the previous best-known complexity of
. Moreover, for the
case that the upper-level function is non-convex, our method requires at most
stochastic
oracle queries to find an -stationary point. In the
finite-sum setting, we show that the number of stochastic oracle calls required
by our method are and
for the convex and non-convex
settings, respectively, where
Frank-Wolfe-type methods for a class of nonconvex inequality-constrained problems
The Frank-Wolfe (FW) method, which implements efficient linear oracles that
minimize linear approximations of the objective function over a fixed compact
convex set, has recently received much attention in the optimization and
machine learning literature. In this paper, we propose a new FW-type method for
minimizing a smooth function over a compact set defined as the level set of a
single difference-of-convex function, based on new generalized
linear-optimization oracles (LO). We show that these LOs can be computed
efficiently with closed-form solutions in some important optimization models
that arise in compressed sensing and machine learning. In addition, under a
mild strict feasibility condition, we establish the subsequential convergence
of our nonconvex FW-type method. Since the feasible region of our generalized
LO typically changes from iteration to iteration, our convergence analysis is
completely different from those existing works in the literature on FW-type
methods that deal with fixed feasible regions among subproblems. Finally,
motivated by the away steps for accelerating FW-type methods for convex
problems, we further design an away-step oracle to supplement our nonconvex
FW-type method, and establish subsequential convergence of this variant.
Numerical results on the matrix completion problem with standard datasets are
presented to demonstrate the efficiency of the proposed FW-type method and its
away-step variant.Comment: We updated grant information and fixed some minor typos in Section
- …