134 research outputs found
Riemannian Smoothing Gradient Type Algorithms]{Riemannian Smoothing Gradient Type Algorithms for Nonsmooth Optimization Problem on Compact Riemannian Submanifold Embedded in Euclidean Space
In this paper, we introduce the notion of generalized -stationarity
for a class of nonconvex and nonsmooth composite minimization problems on
compact Riemannian submanifold embedded in Euclidean space. To find a
generalized -stationarity point, we develop a family of Riemannian
gradient-type methods based on the Moreau envelope technique with a decreasing
sequence of smoothing parameters, namely Riemannian smoothing gradient and
Riemannian smoothing stochastic gradient methods. We prove that the Riemannian
smoothing gradient method has the iteration complexity of
for driving a generalized -stationary
point. To our knowledge, this is the best-known iteration complexity result for
the nonconvex and nonsmooth composite problem on manifolds. For the Riemannian
smoothing stochastic gradient method, one can achieve the iteration complexity
of for driving a generalized -stationary
point. Numerical experiments are conducted to validate the superiority of our
algorithms
Projection Robust Wasserstein Distance and Riemannian Optimization
Projection robust Wasserstein (PRW) distance, or Wasserstein projection
pursuit (WPP), is a robust variant of the Wasserstein distance. Recent work
suggests that this quantity is more robust than the standard Wasserstein
distance, in particular when comparing probability measures in high-dimensions.
However, it is ruled out for practical application because the optimization
model is essentially non-convex and non-smooth which makes the computation
intractable. Our contribution in this paper is to revisit the original
motivation behind WPP/PRW, but take the hard route of showing that, despite its
non-convexity and lack of nonsmoothness, and even despite some hardness results
proved by~\citet{Niles-2019-Estimation} in a minimax sense, the original
formulation for PRW/WPP \textit{can} be efficiently computed in practice using
Riemannian optimization, yielding in relevant cases better behavior than its
convex relaxation. More specifically, we provide three simple algorithms with
solid theoretical guarantee on their complexity bound (one in the appendix),
and demonstrate their effectiveness and efficiency by conducing extensive
experiments on synthetic and real data. This paper provides a first step into a
computational theory of the PRW distance and provides the links between optimal
transport and Riemannian optimization.Comment: Accepted by NeurIPS 2020; The first two authors contributed equally;
fix the confusing parts in the proof and refine the algorithms and complexity
bound
Smoothing algorithms for nonsmooth and nonconvex minimization over the stiefel manifold
We consider a class of nonsmooth and nonconvex optimization problems over the
Stiefel manifold where the objective function is the summation of a nonconvex
smooth function and a nonsmooth Lipschitz continuous convex function composed
with an linear mapping. We propose three numerical algorithms for solving this
problem, by combining smoothing methods and some existing algorithms for smooth
optimization over the Stiefel manifold. In particular, we approximate the
aforementioned nonsmooth convex function by its Moreau envelope in our
smoothing methods, and prove that the Moreau envelope has many favorable
properties. Thanks to this and the scheme for updating the smoothing parameter,
we show that any accumulation point of the solution sequence generated by the
proposed algorithms is a stationary point of the original optimization problem.
Numerical experiments on building graph Fourier basis are conducted to
demonstrate the efficiency of the proposed algorithms.Comment: 22 page
Smoothing Gradient Tracking for Decentralized Optimization over the Stiefel Manifold with Non-smooth Regularizers
Recently, decentralized optimization over the Stiefel manifold has attacked
tremendous attentions due to its wide range of applications in various fields.
Existing methods rely on the gradients to update variables, which are not
applicable to the objective functions with non-smooth regularizers, such as
sparse PCA. In this paper, to the best of our knowledge, we propose the first
decentralized algorithm for non-smooth optimization over Stiefel manifolds. Our
algorithm approximates the non-smooth part of objective function by its Moreau
envelope, and then existing algorithms for smooth optimization can be deployed.
We establish the convergence guarantee with the iteration complexity of
. Numerical experiments conducted under the
decentralized setting demonstrate the effectiveness and efficiency of our
algorithm
- …