11,786 research outputs found
An Efficient Primal-Dual Prox Method for Non-Smooth Optimization
We study the non-smooth optimization problems in machine learning, where both
the loss function and the regularizer are non-smooth functions. Previous
studies on efficient empirical loss minimization assume either a smooth loss
function or a strongly convex regularizer, making them unsuitable for
non-smooth optimization. We develop a simple yet efficient method for a family
of non-smooth optimization problems where the dual form of the loss function is
bilinear in primal and dual variables. We cast a non-smooth optimization
problem into a minimax optimization problem, and develop a primal dual prox
method that solves the minimax optimization problem at a rate of
{assuming that the proximal step can be efficiently solved}, significantly
faster than a standard subgradient descent method that has an
convergence rate. Our empirical study verifies the efficiency of the proposed
method for various non-smooth optimization problems that arise ubiquitously in
machine learning by comparing it to the state-of-the-art first order methods
Primal Dual Alternating Proximal Gradient Algorithms for Nonsmooth Nonconvex Minimax Problems with Coupled Linear Constraints
Nonconvex minimax problems have attracted wide attention in machine learning,
signal processing and many other fields in recent years. In this paper, we
propose a primal dual alternating proximal gradient (PDAPG) algorithm and a
primal dual proximal gradient (PDPG-L) algorithm for solving nonsmooth
nonconvex-strongly concave and nonconvex-linear minimax problems with coupled
linear constraints, respectively. The corresponding iteration complexity of the
two algorithms are proved to be
and to reach an
-stationary point, respectively. To our knowledge, they are the
first two algorithms with iteration complexity guarantee for solving the two
classes of minimax problems
An Interior-Point algorithm for Nonlinear Minimax Problems
We present a primal-dual interior-point method for constrained nonlinear, discrete minimax problems where the objective functions and constraints are not necessarily convex. The algorithm uses two merit functions to ensure progress toward the points satisfying the first-order optimality conditions of the original problem. Convergence properties are described and numerical results provided.Discrete min-max, Constrained nonlinear programming, Primal-dual interior-point methods, Stepsize strategies.
Decentralized gradient descent maximization method for composite nonconvex strongly-concave minimax problems
Minimax problems have recently attracted a lot of research interests. A few
efforts have been made to solve decentralized nonconvex strongly-concave (NCSC)
minimax-structured optimization; however, all of them focus on smooth problems
with at most a constraint on the maximization variable. In this paper, we make
the first attempt on solving composite NCSC minimax problems that can have
convex nonsmooth terms on both minimization and maximization variables. Our
algorithm is designed based on a novel reformulation of the decentralized
minimax problem that introduces a multiplier to absorb the dual consensus
constraint. The removal of dual consensus constraint enables the most
aggressive (i.e., local maximization instead of a gradient ascent step) dual
update that leads to the benefit of taking a larger primal stepsize and better
complexity results. In addition, the decoupling of the nonsmoothness and
consensus on the dual variable eases the analysis of a decentralized algorithm;
thus our reformulation creates a new way for interested researchers to design
new (and possibly more efficient) decentralized methods on solving NCSC minimax
problems. We show a global convergence result of the proposed algorithm and an
iteration complexity result to produce a (near) stationary point of the
reformulation. Moreover, a relation is established between the (near)
stationarities of the reformulation and the original formulation. With this
relation, we show that when the dual regularizer is smooth, our algorithm can
have lower complexity results (with reduced dependence on a condition number)
than existing ones to produce a near-stationary point of the original
formulation. Numerical experiments are conducted on a distributionally robust
logistic regression to demonstrate the performance of the proposed algorithm
Minimax strategies and duality with applications in Financial Mathematics
Many topics in Actuarial and Financial Mathematics lead to Minimax or Maximin problems (risk measures optimization, ambiguous setting, robust solutions, Bayesian credibility theory, interest rate risk, etc.). However, minimax problems are usually difficult to address, since they may involve complex vector spaces or constraints. This paper presents an unified approach so as to deal with minimax convex problems. In particular, we will yield a dual problem providing necessary and sufficient optimality conditions that easily apply in practice. Both, duals and optimality conditions are significantly simplified by drawing on the representation of probability measures on convex sets by points, classic problem for Choquet integrals. Important applications in risk analysis are given.Publicad
Doubly Smoothed GDA: Global Convergent Algorithm for Constrained Nonconvex-Nonconcave Minimax Optimization
Nonconvex-nonconcave minimax optimization has received intense attention over
the last decade due to its broad applications in machine learning.
Unfortunately, most existing algorithms cannot be guaranteed to converge
globally and even suffer from limit cycles. To address this issue, we propose a
novel single-loop algorithm called doubly smoothed gradient descent ascent
method (DSGDA), which naturally balances the primal and dual updates. The
proposed DSGDA can get rid of limit cycles in various challenging
nonconvex-nonconcave examples in the literature, including Forsaken,
Bilinearly-coupled minimax, Sixth-order polynomial, and PolarGame. We further
show that under an one-sided Kurdyka-\L{}ojasiewicz condition with exponent
(resp. convex primal/concave dual function), DSGDA can find a
game-stationary point with an iteration complexity of
(resp.
). These match the best results for single-loop
algorithms that solve nonconvex-concave or convex-nonconcave minimax problems,
or problems satisfying the rather restrictive one-sided Polyak-\L{}ojasiewicz
condition. Our work demonstrates, for the first time, the possibility of having
a simple and unified single-loop algorithm for solving nonconvex-nonconcave,
nonconvex-concave, and convex-nonconcave minimax problems
- ā¦