93 research outputs found
Variance Reduced Halpern Iteration for Finite-Sum Monotone Inclusions
Machine learning approaches relying on such criteria as adversarial
robustness or multi-agent settings have raised the need for solving
game-theoretic equilibrium problems. Of particular relevance to these
applications are methods targeting finite-sum structure, which generically
arises in empirical variants of learning problems in these contexts. Further,
methods with computable approximation errors are highly desirable, as they
provide verifiable exit criteria. Motivated by these applications, we study
finite-sum monotone inclusion problems, which model broad classes of
equilibrium problems. Our main contributions are variants of the classical
Halpern iteration that employ variance reduction to obtain improved complexity
guarantees in which component operators in the finite sum are ``on
average'' either cocoercive or Lipschitz continuous and monotone, with
parameter . The resulting oracle complexity of our methods, which provide
guarantees for the last iterate and for a (computable) operator norm residual,
is , which improves
upon existing methods by a factor up to . This constitutes the first
variance reduction-type result for general finite-sum monotone inclusions and
for more specific problems such as convex-concave optimization when operator
norm residual is the optimality measure. We further argue that, up to
poly-logarithmic factors, this complexity is unimprovable in the monotone
Lipschitz setting; i.e., the provided result is near-optimal
Semi-Anchored Multi-Step Gradient Descent Ascent Method for Structured Nonconvex-Nonconcave Composite Minimax Problems
Minimax problems, such as generative adversarial network, adversarial
training, and fair training, are widely solved by a multi-step gradient descent
ascent (MGDA) method in practice. However, its convergence guarantee is
limited. In this paper, inspired by the primal-dual hybrid gradient method, we
propose a new semi-anchoring (SA) technique for the MGDA method. This makes the
MGDA method find a stationary point of a structured nonconvex-nonconcave
composite minimax problem; its saddle-subdifferential operator satisfies the
weak Minty variational inequality condition. The resulting method, named
SA-MGDA, is built upon a Bregman proximal point method. We further develop its
backtracking line-search version, and its non-Euclidean version for smooth
adaptable functions. Numerical experiments, including a fair classification
training, are provided
A general inexact iterative method for monotone operators, equilibrium problems and fıxed point problems of semigroups in Hilbert spaces
Let H be a real Hilbert space. Consider on H a nonexpansive family T = {T(t) :0 ≤ t < ∞} with a common fixed point, a contraction f with the coefficient
Principled Analyses and Design of First-Order Methods with Inexact Proximal Operators
Proximal operations are among the most common primitives appearing in both
practical and theoretical (or high-level) optimization methods. This basic
operation typically consists in solving an intermediary (hopefully simpler)
optimization problem. In this work, we survey notions of inaccuracies that can
be used when solving those intermediary optimization problems. Then, we show
that worst-case guarantees for algorithms relying on such inexact proximal
operations can be systematically obtained through a generic procedure based on
semidefinite programming. This methodology is primarily based on the approach
introduced by Drori and Teboulle (Mathematical Programming, 2014) and on convex
interpolation results, and allows producing non-improvable worst-case analyzes.
In other words, for a given algorithm, the methodology generates both
worst-case certificates (i.e., proofs) and problem instances on which those
bounds are achieved.
Relying on this methodology, we provide three new methods with conceptually
simple proofs: (i) an optimized relatively inexact proximal point method, (ii)
an extension of the hybrid proximal extragradient method of Monteiro and
Svaiter (SIAM Journal on Optimization, 2013), and (iii) an inexact accelerated
forward-backward splitting supporting backtracking line-search, and both (ii)
and (iii) supporting possibly strongly convex objectives. Finally, we use the
methodology for studying a recent inexact variant of the Douglas-Rachford
splitting due to Eckstein and Yao (Mathematical Programming, 2018).
We showcase and compare the different variants of the accelerated inexact
forward-backward method on a factorization and a total variation problem.Comment: Minor modifications including acknowledgments and references. Code
available at https://github.com/mathbarre/InexactProximalOperator
An abstract proximal point algorithm
The proximal point algorithm is a widely used tool for solving a variety of
convex optimization problems such as finding zeros of maximally monotone
operators, fixed points of nonexpansive mappings, as well as minimizing convex
functions. The algorithm works by applying successively so-called "resolvent"
mappings associated to the original object that one aims to optimize. In this
paper we abstract from the corresponding resolvents employed in these problems
the natural notion of jointly firmly nonexpansive families of mappings. This
leads to a streamlined method of proving weak convergence of this class of
algorithms in the context of complete CAT(0) spaces (and hence also in Hilbert
spaces). In addition, we consider the notion of uniform firm nonexpansivity in
order to similarly provide a unified presentation of a case where the algorithm
converges strongly. Methods which stem from proof mining, an applied subfield
of logic, yield in this situation computable and low-complexity rates of
convergence
Accelerated Infeasibility Detection of Constrained Optimization and Fixed-Point Iterations
As first-order optimization methods become the method of choice for solving
large-scale optimization problems, optimization solvers based on first-order
algorithms are being built. Such general-purpose solvers must robustly detect
infeasible or misspecified problem instances, but the computational complexity
of first-order methods for doing so has yet to be formally studied. In this
work, we characterize the optimal accelerated rate of infeasibility detection.
We show that the standard fixed-point iteration achieves a
and rates, respectively, on the normalized iterates and the
fixed-point residual converging to the infimal displacement vector, while the
accelerated fixed-point iteration achieves and
rates. We then provide a matching complexity lower
bound to establish that is indeed the optimal accelerated rate
- …