Newton method for finding a singularity of a special class of locally Lipschitz continuous vector fields on Riemannian manifolds

In this paper, we extend some results of nonsmooth analysis from Euclidean context to the Riemannian setting. In particular, we discuss the concept and some properties of locally Lipschitz continuous vector fields on Riemannian settings, such as Clarke generalized covariant derivative, upper semicontinuity and Rademacher theorem. We also present a version of Newton method for finding a singularity of a special class of locally Lipschitz continuous vector fields. Under mild conditions, we establish the well-definedness and local convergence of the sequence generated by the method in a neighborhood of a singularity. In particular, a local convergence result for semismooth vector fields is presented. Furthermore, under Kantorovich-type assumptions the convergence of the sequence generated by the Newton method to a solution is established, and its uniqueness in a suitable neighborhood of the starting point is verified

Damped Newton's Method on Riemannian Manifolds

A damped Newton's method to find a singularity of a vector field in Riemannian setting is presented with global convergence study. It is ensured that the sequence generated by the proposed method reduces to a sequence generated by the Riemannian version of the classical Newton's method after a finite number of iterations, consequently its convergence rate is superlinear/quadratic. Moreover, numerical experiments illustrate that the damped Newton's method has better performance than Newton's method in number of iteration and computational time

Multivariate Regression with Gross Errors on Manifold-valued Data

We consider the topic of multivariate regression on manifold-valued output, that is, for a multivariate observation, its output response lies on a manifold. Moreover, we propose a new regression model to deal with the presence of grossly corrupted manifold-valued responses, a bottleneck issue commonly encountered in practical scenarios. Our model first takes a correction step on the grossly corrupted responses via geodesic curves on the manifold, and then performs multivariate linear regression on the corrected data. This results in a nonconvex and nonsmooth optimization problem on manifolds. To this end, we propose a dedicated approach named PALMR, by utilizing and extending the proximal alternating linearized minimization techniques. Theoretically, we investigate its convergence property, where it is shown to converge to a critical point under mild conditions. Empirically, we test our model on both synthetic and real diffusion tensor imaging data, and show that our model outperforms other multivariate regression models when manifold-valued responses contain gross errors, and is effective in identifying gross errors.Comment: 14 pages, submitted to an IEEE journa

A Manifold Proximal Linear Method for Sparse Spectral Clustering with Application to Single-Cell RNA Sequencing Data Analysis

Spectral clustering is one of the fundamental unsupervised learning methods widely used in data analysis. Sparse spectral clustering (SSC) imposes sparsity to the spectral clustering and it improves the interpretability of the model. This paper considers a widely adopted model for SSC, which can be formulated as an optimization problem over the Stiefel manifold with nonsmooth and nonconvex objective. Such an optimization problem is very challenging to solve. Existing methods usually solve its convex relaxation or need to smooth its nonsmooth part using certain smoothing techniques. In this paper, we propose a manifold proximal linear method (ManPL) that solves the original SSC formulation. We also extend the algorithm to solve the multiple-kernel SSC problems, for which an alternating ManPL algorithm is proposed. Convergence and iteration complexity results of the proposed methods are established. We demonstrate the advantage of our proposed methods over existing methods via the single-cell RNA sequencing data analysis

Nonconvex weak sharp minima on Riemannian manifolds

We are to establish necessary conditions (of the primal and dual types) for the set of weak sharp minima of a nonconvex optimization problem on a Riemannian manifold. Here, we are to provide a generalization of some characterizations of weak sharp minima for convex problems on Riemannian manifold introduced by Li et al. (SIAM J. Optim., 21 (2011), pp. 1523--1560) for nonconvex problems. We use the theory of the Fr\'echet and limiting subdifferentials on Riemannian manifold to give the necessary conditions of the dual type. We also consider a theory of contingent directional derivative and a notion of contingent cone on Riemannian manifold to give the necessary conditions of the primal type. Several definitions have been provided for the contingent cone on Riemannian manifold. We show that these definitions, with some modifications, are equivalent. We establish a lemma about the local behavior of a distance function. Using the lemma, we express the Fr\'echet subdifferential (contingent directional derivative) of a distance function on a Riemannian manifold in terms of normal cones (contingent cones), to establish the necessary conditions. As an application, we show how one can use weak sharp minima property to model a Cheeger type constant of a graph as an optimization problem on a Stiefel manifold.Comment: An Application section was adde

Discrete Total Variation of the Normal Vector Field as Shape Prior with Applications in Geometric Inverse Problems

An analogue of the total variation prior for the normal vector field along the boundary of piecewise flat shapes in 3D is introduced. A major class of examples are triangulated surfaces as they occur for instance in finite element computations. The analysis of the functional is based on a differential geometric setting in which the unit normal vector is viewed as an element of the two-dimensional sphere manifold. It is found to agree with the discrete total mean curvature known in discrete differential geometry. A split Bregman iteration is proposed for the solution of discretized shape optimization problems, in which the total variation of the normal appears as a regularizer. Unlike most other priors, such as surface area, the new functional allows for piecewise flat shapes. As two applications, a mesh denoising and a geometric inverse problem of inclusion detection type involving a partial differential equation are considered. Numerical experiments confirm that polyhedral shapes can be identified quite accurately.Comment: arXiv admin note: substantial text overlap with arXiv:1902.0724

Total Variation of the Normal Vector Field as Shape Prior

An analogue of the total variation prior for the normal vector field along the boundary of smooth shapes in 3D is introduced. The analysis of the total variation of the normal vector field is based on a differential geometric setting in which the unit normal vector is viewed as an element of the two-dimensional sphere manifold. It is shown that spheres are stationary points when the total variation of the normal is minimized under an area constraint. Shape calculus is used to characterize the relevant derivatives. Since the total variation functional is non-differentiable whenever the boundary contains flat regions, an extension of the split Bregman method to manifold valued functions is proposed

Proximal Gradient Method for Nonsmooth Optimization over the Stiefel Manifold

We consider optimization problems over the Stiefel manifold whose objective function is the summation of a smooth function and a nonsmooth function. Existing methods for solving this kind of problems can be classified into three classes. Algorithms in the first class rely on information of the subgradients of the objective function and thus tend to converge slowly in practice. Algorithms in the second class are proximal point algorithms, which involve subproblems that can be as difficult as the original problem. Algorithms in the third class are based on operator-splitting techniques, but they usually lack rigorous convergence guarantees. In this paper, we propose a retraction-based proximal gradient method for solving this class of problems. We prove that the proposed method globally converges to a stationary point. Iteration complexity for obtaining an $\epsilon$-stationary solution is also analyzed. Numerical results on solving sparse PCA and compressed modes problems are reported to demonstrate the advantages of the proposed method

A New Constrained Optimization Model for Solving the Nonsymmetric Stochastic Inverse Eigenvalue Problem

The stochastic inverse eigenvalue problem aims to reconstruct a stochastic matrix from its spectrum. While there exists a large literature on the existence of solutions for special settings, there are only few numerical solution methods available so far. Recently, Zhao et al. (2016) proposed a constrained optimization model on the manifold of so-called isospectral matrices and adapted a modified Polak-Ribi\`ere-Polyak conjugate gradient method to the geometry of this manifold. However, not every stochastic matrix is an isospectral one and the model from Zhao et al. is based on the assumption that for each stochastic matrix there exists a (possibly different) isospectral, stochastic matrix with the same spectrum. We are not aware of such a result in the literature, but will see that the claim is at least true for $3 \times 3$ matrices. In this paper, we suggest to extend the above model by considering matrices which differ from isospectral ones only by multiplication with a block diagonal matrix with $2 \times 2$ blocks from the special linear group $SL(2)$, where the number of blocks is given by the number of pairs of complex-conjugate eigenvalues. Every stochastic matrix can be written in such a form, which was not the case for the form of the isospectral matrices. We prove that our model has a minimizer and show how the Polak-Ribi\`ere-Polyak conjugate gradient method works on the corresponding more general manifold. We demonstrate by numerical examples that the new, more general method performs similarly as the one from Zhao et al

Weakly Convex Optimization over Stiefel Manifold Using Riemannian Subgradient-Type Methods

We consider a class of nonsmooth optimization problems over the Stiefel manifold, in which the objective function is weakly convex in the ambient Euclidean space. Such problems are ubiquitous in engineering applications but still largely unexplored. We present a family of Riemannian subgradient-type methods -- namely Riemannain subgradient, incremental subgradient, and stochastic subgradient methods -- to solve these problems and show that they all have an iteration complexity of ${\cal O}(\varepsilon^{-4})$ for driving a natural stationarity measure below $\varepsilon$. In addition, we establish the local linear convergence of the Riemannian subgradient and incremental subgradient methods when the problem at hand further satisfies a sharpness property and the algorithms are properly initialized and use geometrically diminishing stepsizes. To the best of our knowledge, these are the first convergence guarantees for using Riemannian subgradient-type methods to optimize a class of nonconvex nonsmooth functions over the Stiefel manifold. The fundamental ingredient in the proof of the aforementioned convergence results is a new Riemannian subgradient inequality for restrictions of weakly convex functions on the Stiefel manifold, which could be of independent interest. We also show that our convergence results can be extended to handle a class of compact embedded submanifolds of the Euclidean space. Finally, we discuss the sharpness properties of various formulations of the robust subspace recovery and orthogonal dictionary learning problems and demonstrate the convergence performance of the algorithms on both problems via numerical simulations.Comment: 30 pages. Accepted to SIAM Journal on Optimizatio