24 research outputs found
Low-rank Tensor Estimation via Riemannian Gauss-Newton: Statistical Optimality and Second-Order Convergence
In this paper, we consider the estimation of a low Tucker rank tensor from a
number of noisy linear measurements. The general problem covers many specific
examples arising from applications, including tensor regression, tensor
completion, and tensor PCA/SVD. We consider an efficient Riemannian
Gauss-Newton (RGN) method for low Tucker rank tensor estimation. Different from
the generic (super)linear convergence guarantee of RGN in the literature, we
prove the first local quadratic convergence guarantee of RGN for low-rank
tensor estimation in the noisy setting under some regularity conditions and
provide the corresponding estimation error upper bounds. A deterministic
estimation error lower bound, which matches the upper bound, is provided that
demonstrates the statistical optimality of RGN. The merit of RGN is illustrated
through two machine learning applications: tensor regression and tensor SVD.
Finally, we provide the simulation results to corroborate our theoretical
findings
Computational and Statistical Thresholds in Multi-layer Stochastic Block Models
We study the problem of community recovery and detection in multi-layer
stochastic block models, focusing on the critical network density threshold for
consistent community structure inference. Using a prototypical two-block model,
we reveal a computational barrier for such multi-layer stochastic block models
that does not exist for its single-layer counterpart: When there are no
computational constraints, the density threshold depends linearly on the number
of layers. However, when restricted to polynomial-time algorithms, the density
threshold scales with the square root of the number of layers, assuming
correctness of a low-degree polynomial hardness conjecture. Our results provide
a nearly complete picture of the optimal inference in multiple-layer stochastic
block models and partially settle the open question in Lei and Lin (2022)
regarding the optimality of the bias-adjusted spectral method.Comment: 31 page
On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-rank Matrix Optimization
In this paper, we propose a general procedure for establishing the geometric
landscape connections of a Riemannian optimization problem under the embedded
and quotient geometries. By applying the general procedure to the fixed-rank
positive semidefinite (PSD) and general matrix optimization, we establish an
exact Riemannian gradient connection under two geometries at every point on the
manifold and sandwich inequalities between the spectra of Riemannian Hessians
at Riemannian first-order stationary points (FOSPs). These results immediately
imply an equivalence on the sets of Riemannian FOSPs, Riemannian second-order
stationary points (SOSPs), and strict saddles of fixed-rank matrix optimization
under the embedded and the quotient geometries. To the best of our knowledge,
this is the first geometric landscape connection between the embedded and the
quotient geometries for fixed-rank matrix optimization and it provides a
concrete example of how these two geometries are connected in Riemannian
optimization. In addition, the effects of the Riemannian metric and quotient
structure on the landscape connection are discussed. We also observe an
algorithmic connection between two geometries with some specific Riemannian
metrics in fixed-rank matrix optimization: there is an equivalence between
gradient flows under two geometries with shared spectra of Riemannian Hessians.
A number of novel ideas and technical ingredients including a unified treatment
for different Riemannian metrics, novel metrics for the Stiefel manifold, and
new horizontal space representations under quotient geometries are developed to
obtain our results. The results in this paper deepen our understanding of
geometric and algorithmic connections of Riemannian optimization under
different Riemannian geometries and provide a few new theoretical insights to
unanswered questions in the literature
One-dimensional Tensor Network Recovery
We study the recovery of the underlying graphs or permutations for tensors in
tensor ring or tensor train format. Our proposed algorithms compare the
matricization ranks after down-sampling, whose complexity is for
-th order tensors. We prove that our algorithms can almost surely recover
the correct graph or permutation when tensor entries can be observed without
noise. We further establish the robustness of our algorithms against
observational noise. The theoretical results are validated by numerical
experiments
Exact Clustering in Tensor Block Model: Statistical Optimality and Computational Limit
High-order clustering aims to identify heterogeneous substructures in
multiway datasets that arise commonly in neuroimaging, genomics, social network
studies, etc. The non-convex and discontinuous nature of this problem pose
significant challenges in both statistics and computation. In this paper, we
propose a tensor block model and the computationally efficient methods,
\emph{high-order Lloyd algorithm} (HLloyd), and \emph{high-order spectral
clustering} (HSC), for high-order clustering. The convergence guarantees and
statistical optimality are established for the proposed procedure under a mild
sub-Gaussian noise assumption. Under the Gaussian tensor block model, we
completely characterize the statistical-computational trade-off for achieving
high-order exact clustering based on three different signal-to-noise ratio
regimes. The analysis relies on new techniques of high-order spectral
perturbation analysis and a "singular-value-gap-free" error bound in tensor
estimation, which are substantially different from the matrix spectral analyses
in the literature. Finally, we show the merits of the proposed procedures via
extensive experiments on both synthetic and real datasets.Comment: 65 page
Recursive Importance Sketching for Rank Constrained Least Squares: Algorithms and High-order Convergence
In this paper, we propose a new {\it \underline{R}ecursive} {\it
\underline{I}mportance} {\it \underline{S}ketching} algorithm for {\it
\underline{R}ank} constrained least squares {\it \underline{O}ptimization}
(RISRO). As its name suggests, the algorithm is based on a new sketching
framework, recursive importance sketching. Several existing algorithms in the
literature can be reinterpreted under the new sketching framework and RISRO
offers clear advantages over them. RISRO is easy to implement and
computationally efficient, where the core procedure in each iteration is only
solving a dimension reduced least squares problem. Different from numerous
existing algorithms with locally geometric convergence rate, we establish the
local quadratic-linear and quadratic rate of convergence for RISRO under some
mild conditions. In addition, we discover a deep connection of RISRO to
Riemannian manifold optimization on fixed rank matrices. The effectiveness of
RISRO is demonstrated in two applications in machine learning and statistics:
low-rank matrix trace regression and phase retrieval. Simulation studies
demonstrate the superior numerical performance of RISRO