161 research outputs found
Nonconvex Matrix Factorization is Geodesically Convex: Global Landscape Analysis for Fixed-rank Matrix Optimization From a Riemannian Perspective
We study a general matrix optimization problem with a fixed-rank positive
semidefinite (PSD) constraint. We perform the Burer-Monteiro factorization and
consider a particular Riemannian quotient geometry in a search space that has a
total space equipped with the Euclidean metric. When the original objective f
satisfies standard restricted strong convexity and smoothness properties, we
characterize the global landscape of the factorized objective under the
Riemannian quotient geometry. We show the entire search space can be divided
into three regions: (R1) the region near the target parameter of interest,
where the factorized objective is geodesically strongly convex and smooth; (R2)
the region containing neighborhoods of all strict saddle points; (R3) the
remaining regions, where the factorized objective has a large gradient. To our
best knowledge, this is the first global landscape analysis of the
Burer-Monteiro factorized objective under the Riemannian quotient geometry. Our
results provide a fully geometric explanation for the superior performance of
vanilla gradient descent under the Burer-Monteiro factorization. When f
satisfies a weaker restricted strict convexity property, we show there exists a
neighborhood near local minimizers such that the factorized objective is
geodesically convex. To prove our results we provide a comprehensive landscape
analysis of a matrix factorization problem with a least squares objective,
which serves as a critical bridge. Our conclusions are also based on a result
of independent interest stating that the geodesic ball centered at Y with a
radius 1/3 of the least singular value of Y is a geodesically convex set under
the Riemannian quotient geometry, which as a corollary, also implies a
quantitative bound of the convexity radius in the Bures-Wasserstein space. The
convexity radius obtained is sharp up to constants.Comment: The abstract is shortened to meet the arXiv submission requiremen
From Symmetry to Geometry: Tractable Nonconvex Problems
As science and engineering have become increasingly data-driven, the role of
optimization has expanded to touch almost every stage of the data analysis
pipeline, from the signal and data acquisition to modeling and prediction. The
optimization problems encountered in practice are often nonconvex. While
challenges vary from problem to problem, one common source of nonconvexity is
nonlinearity in the data or measurement model. Nonlinear models often exhibit
symmetries, creating complicated, nonconvex objective landscapes, with multiple
equivalent solutions. Nevertheless, simple methods (e.g., gradient descent)
often perform surprisingly well in practice.
The goal of this survey is to highlight a class of tractable nonconvex
problems, which can be understood through the lens of symmetries. These
problems exhibit a characteristic geometric structure: local minimizers are
symmetric copies of a single "ground truth" solution, while other critical
points occur at balanced superpositions of symmetric copies of the ground
truth, and exhibit negative curvature in directions that break the symmetry.
This structure enables efficient methods to obtain global minimizers. We
discuss examples of this phenomenon arising from a wide range of problems in
imaging, signal processing, and data analysis. We highlight the key role of
symmetry in shaping the objective landscape and discuss the different roles of
rotational and discrete symmetries. This area is rich with observed phenomena
and open problems; we close by highlighting directions for future research.Comment: review paper submitted to SIAM Review, 34 pages, 10 figure
Nonnegative Matrix Factorization for Signal and Data Analytics: Identifiability, Algorithms, and Applications
Nonnegative matrix factorization (NMF) has become a workhorse for signal and
data analytics, triggered by its model parsimony and interpretability. Perhaps
a bit surprisingly, the understanding to its model identifiability---the major
reason behind the interpretability in many applications such as topic mining
and hyperspectral imaging---had been rather limited until recent years.
Beginning from the 2010s, the identifiability research of NMF has progressed
considerably: Many interesting and important results have been discovered by
the signal processing (SP) and machine learning (ML) communities. NMF
identifiability has a great impact on many aspects in practice, such as
ill-posed formulation avoidance and performance-guaranteed algorithm design. On
the other hand, there is no tutorial paper that introduces NMF from an
identifiability viewpoint. In this paper, we aim at filling this gap by
offering a comprehensive and deep tutorial on model identifiability of NMF as
well as the connections to algorithms and applications. This tutorial will help
researchers and graduate students grasp the essence and insights of NMF,
thereby avoiding typical `pitfalls' that are often times due to unidentifiable
NMF formulations. This paper will also help practitioners pick/design suitable
factorization tools for their own problems.Comment: accepted version, IEEE Signal Processing Magazine; supplementary
materials added. Some minor revisions implemente
On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-rank Matrix Optimization
In this paper, we propose a general procedure for establishing the geometric
landscape connections of a Riemannian optimization problem under the embedded
and quotient geometries. By applying the general procedure to the fixed-rank
positive semidefinite (PSD) and general matrix optimization, we establish an
exact Riemannian gradient connection under two geometries at every point on the
manifold and sandwich inequalities between the spectra of Riemannian Hessians
at Riemannian first-order stationary points (FOSPs). These results immediately
imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order
stationary points (SOSPs), and strict saddles of fixed-rank matrix optimization
under the embedded and the quotient geometries. To the best of our knowledge,
this is the first geometric landscape connection between the embedded and the
quotient geometries for fixed-rank matrix optimization and it provides a
concrete example of how these two geometries are connected in Riemannian
optimization. In addition, the effects of the Riemannian metric and quotient
structure on the landscape connection are discussed. We also observe an
algorithmic connection between two geometries with some specific Riemannian
metrics in fixed-rank matrix optimization: there is an equivalence between
gradient flows under two geometries with shared spectra of Riemannian Hessians.
A number of novel ideas and technical ingredients including a unified treatment
for different Riemannian metrics, novel metrics for the Stiefel manifold, and
new horizontal space representations under quotient geometries are developed to
obtain our results. The results in this paper deepen our understanding of
geometric and algorithmic connections of Riemannian optimization under
different Riemannian geometries and provide a few new theoretical insights to
unanswered questions in the literature
Weakly Admissible Meshes and Discrete Extremal Sets
We present a brief survey on (Weakly) Admissible Meshes and corresponding Discrete Extremal Sets, namely Approximate Fekete Points and Discrete Leja Points. These provide new computational tools for polynomial least squares and interpolation on multidimensional compact sets, with different applications such as numerical cubature, digital filtering, spectral and high-order methods for PDEs
A glimpse into the differential topology and geometry of optimal transport
This note exposes the differential topology and geometry underlying some of
the basic phenomena of optimal transportation. It surveys basic questions
concerning Monge maps and Kantorovich measures: existence and regularity of the
former, uniqueness of the latter, and estimates for the dimension of its
support, as well as the associated linear programming duality. It shows the
answers to these questions concern the differential geometry and topology of
the chosen transportation cost. It also establishes new connections --- some
heuristic and others rigorous --- based on the properties of the
cross-difference of this cost, and its Taylor expansion at the diagonal.Comment: 27 page
International Conference on Continuous Optimization (ICCOPT) 2019 Conference Book
The Sixth International Conference on Continuous Optimization took place on the campus of the Technical University of Berlin, August 3-8, 2019. The ICCOPT is a flagship conference of the Mathematical Optimization Society (MOS), organized every three years. ICCOPT 2019 was hosted by the Weierstrass Institute for Applied Analysis and Stochastics (WIAS) Berlin. It included a Summer School and a Conference with a series of plenary and semi-plenary talks, organized and contributed sessions, and poster sessions.
This book comprises the full conference program. It contains, in particular, the scientific program in survey style as well as with all details, and information on the social program, the venue, special meetings, and more
- …