93 research outputs found
On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergence
We introduce a framework for quasi-Newton forward--backward splitting
algorithms (proximal quasi-Newton methods) with a metric induced by diagonal
rank- symmetric positive definite matrices. This special type of
metric allows for a highly efficient evaluation of the proximal mapping. The
key to this efficiency is a general proximal calculus in the new metric. By
using duality, formulas are derived that relate the proximal mapping in a
rank- modified metric to the original metric. We also describe efficient
implementations of the proximity calculation for a large class of functions;
the implementations exploit the piece-wise linear nature of the dual problem.
Then, we apply these results to acceleration of composite convex minimization
problems, which leads to elegant quasi-Newton methods for which we prove
convergence. The algorithm is tested on several numerical examples and compared
to a comprehensive list of alternatives in the literature. Our quasi-Newton
splitting algorithm with the prescribed metric compares favorably against
state-of-the-art. The algorithm has extensive applications including signal
processing, sparse recovery, machine learning and classification to name a few.Comment: arXiv admin note: text overlap with arXiv:1206.115
Nonmonotone local minimax methods for finding multiple saddle points
In this paper, by designing a normalized nonmonotone search strategy with the
Barzilai--Borwein-type step-size, a novel local minimax method (LMM), which is
a globally convergent iterative method, is proposed and analyzed to find
multiple (unstable) saddle points of nonconvex functionals in Hilbert spaces.
Compared to traditional LMMs with monotone search strategies, this approach,
which does not require strict decrease of the objective functional value at
each iterative step, is observed to converge faster with less computations.
Firstly, based on a normalized iterative scheme coupled with a local peak
selection that pulls the iterative point back onto the solution submanifold, by
generalizing the Zhang--Hager (ZH) search strategy in the optimization theory
to the LMM framework, a kind of normalized ZH-type nonmonotone step-size search
strategy is introduced, and then a novel nonmonotone LMM is constructed. Its
feasibility and global convergence results are rigorously carried out under the
relaxation of the monotonicity for the functional at the iterative sequences.
Secondly, in order to speed up the convergence of the nonmonotone LMM, a
globally convergent Barzilai--Borwein-type LMM (GBBLMM) is presented by
explicitly constructing the Barzilai--Borwein-type step-size as a trial
step-size of the normalized ZH-type nonmonotone step-size search strategy in
each iteration. Finally, the GBBLMM algorithm is implemented to find multiple
unstable solutions of two classes of semilinear elliptic boundary value
problems with variational structures: one is the semilinear elliptic equations
with the homogeneous Dirichlet boundary condition and another is the linear
elliptic equations with semilinear Neumann boundary conditions. Extensive
numerical results indicate that our approach is very effective and speeds up
the LMMs significantly.Comment: 32 pages, 7 figures; Accepted by Journal of Computational Mathematics
on January 3, 202
A Riemannian exponential augmented Lagrangian method for computing the projection robust Wasserstein distance
Projecting the distance measures onto a low-dimensional space is an efficient
way of mitigating the curse of dimensionality in the classical Wasserstein
distance using optimal transport. The obtained maximized distance is referred
to as projection robust Wasserstein (PRW) distance. In this paper, we
equivalently reformulate the computation of the PRW distance as an optimization
problem over the Cartesian product of the Stiefel manifold and the Euclidean
space with additional nonlinear inequality constraints. We propose a Riemannian
exponential augmented Lagrangian method (ReALM) with a global convergence
guarantee to solve this problem. Compared with the existing approaches, ReALM
can potentially avoid too small penalty parameters. Moreover, we propose a
framework of inexact Riemannian gradient descent methods to solve the
subproblems in ReALM efficiently. In particular, by using the special structure
of the subproblem, we give a practical algorithm named as the inexact
Riemannian Barzilai-Borwein method with Sinkhorn iteration (iRBBS). The
remarkable features of iRBBS lie in that it performs a flexible number of
Sinkhorn iterations to compute an inexact gradient with respect to the
projection matrix of the problem and adopts the Barzilai-Borwein stepsize based
on the inexact gradient information to improve the performance. We show that
iRBBS can return an -stationary point of the original PRW distance
problem within iterations. Extensive numerical
results on synthetic and real datasets demonstrate that our proposed ReALM as
well as iRBBS outperform the state-of-the-art solvers for computing the PRW
distance.Comment: 25 pages, 20 figures, 4 table
- …