Computing Large-Scale Matrix and Tensor Decomposition with Structured Factors: A Unified Nonconvex Optimization Perspective

The proposed article aims at offering a comprehensive tutorial for the computational aspects of structured matrix and tensor factorization. Unlike existing tutorials that mainly focus on {\it algorithmic procedures} for a small set of problems, e.g., nonnegativity or sparsity-constrained factorization, we take a {\it top-down} approach: we start with general optimization theory (e.g., inexact and accelerated block coordinate descent, stochastic optimization, and Gauss-Newton methods) that covers a wide range of factorization problems with diverse constraints and regularization terms of engineering interest. Then, we go `under the hood' to showcase specific algorithm design under these introduced principles. We pay a particular attention to recent algorithmic developments in structured tensor and matrix factorization (e.g., random sketching and adaptive step size based stochastic optimization and structure-exploiting second-order algorithms), which are the state of the art---yet much less touched upon in the literature compared to {\it block coordinate descent} (BCD)-based methods. We expect that the article to have an educational values in the field of structured factorization and hope to stimulate more research in this important and exciting direction.Comment: Final Version; to appear in IEEE Signal Processing Magazine; title revised to comply with the journal's rul

Blind Multilinear Identification

We discuss a technique that allows blind recovery of signals or blind identification of mixtures in instances where such recovery or identification were previously thought to be impossible: (i) closely located or highly correlated sources in antenna array processing, (ii) highly correlated spreading codes in CDMA radio communication, (iii) nearly dependent spectra in fluorescent spectroscopy. This has important implications --- in the case of antenna array processing, it allows for joint localization and extraction of multiple sources from the measurement of a noisy mixture recorded on multiple sensors in an entirely deterministic manner. In the case of CDMA, it allows the possibility of having a number of users larger than the spreading gain. In the case of fluorescent spectroscopy, it allows for detection of nearly identical chemical constituents. The proposed technique involves the solution of a bounded coherence low-rank multilinear approximation problem. We show that bounded coherence allows us to establish existence and uniqueness of the recovered solution. We will provide some statistical motivation for the approximation problem and discuss greedy approximation bounds. To provide the theoretical underpinnings for this technique, we develop a corresponding theory of sparse separable decompositions of functions, including notions of rank and nuclear norm that specialize to the usual ones for matrices and operators but apply to also hypermatrices and tensors.Comment: 20 pages, to appear in IEEE Transactions on Information Theor

Faster Robust Tensor Power Method for Arbitrary Order

Tensor decomposition is a fundamental method used in various areas to deal with high-dimensional data. \emph{Tensor power method} (TPM) is one of the widely-used techniques in the decomposition of tensors. This paper presents a novel tensor power method for decomposing arbitrary order tensors, which overcomes limitations of existing approaches that are often restricted to lower-order (less than $3$) tensors or require strong assumptions about the underlying data structure. We apply sketching method, and we are able to achieve the running time of $\widetilde{O}(n^{p-1})$, on the power $p$ and dimension $n$ tensor. We provide a detailed analysis for any $p$-th order tensor, which is never given in previous works

Techniques tensorielles pour le traitement du signal : algorithmes pour la décomposition polyadique canonique

Low rank tensor decomposition has been playing for the last years an important rolein many applications such as blind source separation, telecommunications, sensor array processing,neuroscience, chemometrics, and data mining. The Canonical Polyadic tensor decomposition is veryattractive when compared to standard matrix-based tools, manly on system identification. In this thesis,we propose: (i) several algorithms to compute specific low rank-approximations: finite/iterativerank-1 approximations, iterative deflation approximations, and orthogonal tensor decompositions. (ii)A new strategy to solve multivariate quadratic systems, where this problem is reduced to a best rank-1 tensor approximation problem. (iii) Theoretical results to study and proof the performance or theconvergence of some algorithms. All performances are supported by numerical experiments.L’approximation tensorielle de rang faible joue ces dernières années un rôle importantdans plusieurs applications, telles que la séparation aveugle de source, les télécommunications, letraitement d’antennes, les neurosciences, la chimiométrie, et l’exploration de données. La décompositiontensorielle Canonique Polyadique est très attractive comparativement à des outils matriciels classiques,notamment pour l’identification de systèmes. Dans cette thèse, nous proposons (i) plusieursalgorithmes pour calculer quelques approximations de rang faible spécifique: approximation de rang-1 itérative et en un nombre fini d’opérations, l’approximation par déflation itérative, et la décompositiontensorielle orthogonale; (ii) une nouvelle stratégie pour résoudre des systèmes quadratiquesmultivariés, où ce problème peut être réduit à la meilleure approximation de rang-1 d’un tenseur; (iii)des résultats théoriques pour étudier les performances ou prouver la convergence de quelques algorithmes.Toutes les performances sont illustrées par des simulations informatiques.A aproximação tensorial de baixo posto desempenha nestes últimos anos um papel importanteem várias aplicações, tais como separação cega de fontes, telecomunicações, processamentode antenas, neurociênca, quimiometria e exploração de dados. A decomposição tensorial canônicaé bastante atrativa se comparada às técnicas matriciais clássicas, principalmente na identificação desistemas. Nesta tese, propõe-se (i) vários algoritmos para calcular alguns tipos de aproximação deposto: aproximação de posto-1 iterativa e em um número finito de operações, a aproximação pordeflação iterativa, e a decomposição tensorial ortogonal; (ii) uma nova estratégia para resolver sistemasquadráticos em várias variáveis, em que tal problema pode ser reduzido à melhor aproximaçãode posto-1 de um tensor; (iii) resultados teóricos visando estudar o desempenho ou demonstrar aconvergência de alguns algoritmos. Todas os desempenhos são ilustrados através de simulações computacionais

Structured Tensor Recovery and Decomposition

Tensors, a.k.a. multi-dimensional arrays, arise naturally when modeling higher-order objects and relations. Among ubiquitous applications including image processing, collaborative filtering, demand forecasting and higher-order statistics, there are two recurring themes in general: tensor recovery and tensor decomposition. The first one aims to recover the underlying tensor from incomplete information; the second one is to study a variety of tensor decompositions to represent the array more concisely and moreover to capture the salient characteristics of the underlying data. Both topics are respectively addressed in this thesis. Chapter 2 and Chapter 3 focus on low-rank tensor recovery (LRTR) from both theoretical and algorithmic perspectives. In Chapter 2, we first provide a negative result to the sum of nuclear norms (SNN) model---an existing convex model widely used for LRTR; then we propose a novel convex model and prove this new model is better than the SNN model in terms of the number of measurements required to recover the underlying low-rank tensor. In Chapter 3, we first build up the connection between robust low-rank tensor recovery and the compressive principle component pursuit (CPCP), a convex model for robust low-rank matrix recovery. Then we focus on developing convergent and scalable optimization methods to solve the CPCP problem. In specific, our convergent method, proposed by combining classical ideas from Frank-Wolfe and proximal methods, achieves scalability with linear per-iteration cost. Chapter 4 generalizes the successive rank-one approximation (SROA) scheme for matrix eigen-decomposition to a special class of tensors called symmetric and orthogonally decomposable (SOD) tensor. We prove that the SROA scheme can robustly recover the symmetric canonical decomposition of the underlying SOD tensor even in the presence of noise. Perturbation bounds, which can be regarded as a higher-order generalization of the Davis-Kahan theorem, are provided in terms of the noise magnitude

Tensor Neural Network and Its Numerical Integration

In this paper, we introduce a type of tensor neural network. For the first time, we propose its numerical integration scheme and prove the computational complexity to be the polynomial scale of the dimension. Based on the tensor product structure, we develop an efficient numerical integration method by using fixed quadrature points for the functions of the tensor neural network. The corresponding machine learning method is also introduced for solving high-dimensional problems. Some numerical examples are also provided to validate the theoretical results and the numerical algorithm.Comment: 27 pages, 30 figure