Asymptotics and Statistical Inference in High-Dimensional Low-Rank Matrix Models

High-dimensional matrix and tensor data is ubiquitous in machine learning and statistics and often exhibits low-dimensional structure. With the rise of these types of data is the need to develop statistical inference procedures that adequately address the low-dimensional structure in a principled manner. In this dissertation we study asymptotic theory and statistical inference in structured low-rank matrix models in high-dimensional regimes where the column and row dimensions of the matrix are allowed to grow, and we consider a variety of settings for which structured low-rank matrix models manifest. Chapter 1 establishes the general framework for statistical analysis in high-dimensional low-rank matrix models, including introducing entrywise perturbation bounds, asymptotic theory, distributional theory, and statistical inference, illustrated throughout via the matrix denoising model. In Chapter 2, Chapter 3, and Chapter 4 we study the entrywise estimation of singular vectors and eigenvectors in different structured settings, with Chapter 2 considering heteroskedastic and dependent noise, Chapter 3 sparsity, and Chapter 4 additional tensor structure. In Chapter 5 we apply previous asymptotic theory to study a two-sample test for equality of distribution in network analysis, and in Chapter 6 we study a model for shared community memberships across multiple networks, and we propose and analyze a joint spectral clustering algorithm that leverages newly developed asymptotic theory for this setting. Throughout this dissertation we emphasize tools and techniques that are data-driven, nonparametric, and adaptive to signal strength, and, where applicable, noise distribution. The contents of Chapters 2-6 are based on the papers Agterberg et al. (2022b); Agterberg and Sulam (2022); Agterberg and Zhang (2022); Agterberg et al. (2020a) and Agterberg et al. (2022a) respectively, and Chapter 1 contains several novel results

Robust Low-Rank Approximation of Matrices in lp-Space

Low-rank approximation plays an important role in many areas of science and engineering such as signal/image processing, machine learning, data mining, imaging, bioinformatics, pattern classification and computer vision because many real-world data exhibit low-rank property. This dissertation devises advanced algorithms for robust low-rank approximation of a single matrix as well as multiple matrices in the presence of outliers, where the conventional dimensionality reduction techniques such as the celebrated principal component analysis (PCA) are not applicable. The proposed methodology is based on minimizing the entry-wise $\ell_p$-norm of the residual including the challenging nonconvex and nonsmooth case of $p<1$. Theoretical analyses are also presented. Extensive practical applications are discussed. Experimental results demonstrate that the superiority of the proposed methods over the state-of-the-art techniques. Two iterative algorithms are designed for low-rank approximation of a single matrix. The first is the iteratively reweighted singular value decomposition (IR-SVD), where the SVD of a reweighted matrix is performed at each iteration. The second converts the nonconvex $\ell_p$-matrix factorization into a series of easily solvable $\ell_p$-norm minimization with vectors being variables. Applications to image demixing, foreground detection in video surveillance, array signal processing, and direction-of-arrival estimation for source localization in impulsive noise are investigated. The low-rank approximation with missing values, i.e., robust matrix completion, is also addressed. Two algorithms are developed for it. The first iteratively solves a set of linear $\ell_p$-regression problems while the second applies the alternating direction method of multipliers (ADMM) in the $\ell_p$-space. At each iteration of the ADMM, it requires performing a least squares (LS) matrix factorization and calculating the proximity operator of the $p$th power of the $\ell_p$-norm. The LS factorization is efficiently solved using linear LS regression while the proximity operator is obtained by root finding of a scalar nonlinear equation. The two proposed algorithms are scalable to the problem size. Applications to recommender systems, collaborative filtering, and image inpainting are provided. The $\ell_p$-greedy pursuit ($\ell_p$-GP) algorithms are devised for joint robust low-rank approximation of multiple matrices (RLRAMM) with outliers. The $\ell_p$-GP with $0<p<2$ solves the RLRAMM by decomposing it into a series of rank-one approximations. At each iteration, it finds the best rank-one approximation by minimizing the $\ell_p$-norm of the residual and then, the rank-one basis matrices are subtracted from the residual. A successive minimization approach is designed for the $\ell_p$-rank-one fitting. Only weighted medians are required to compute for solving the most attractive case with $p=1$, yielding that the complexity is near-linear with the number and dimension of the matrices. Thus, the $\ell_1$-GP is near-scalable to large-scale problems. The convergence of the $\ell_p$-GP is theoretically proved. In particular, the sum of the $\ell_p$-norms of the residuals decays exponentially. We reveal that the worst-case bound of the convergence rate is related to the $\ell_p$-correlation of the residual and the current solution. The $\ell_p$-GP has a higher compression ratio than the existing methods. For the special case of $p=2$, the orthogonal greedy pursuit (OGP) is further developed to accelerate the convergence, where the cost of weight re-computation is reduced by a recursive update manner. Tighter and more accurate bounds of the convergence rates are theoretically derived for $p=2$. Applications to data compression, robust image reconstruction and computer vision are provided