Theory and Algorithms for Reliable Multimodal Data Analysis, Machine Learning, and Signal Processing

Modern engineering systems collect large volumes of data measurements across diverse sensing modalities. These measurements can naturally be arranged in higher-order arrays of scalars which are commonly referred to as tensors. Tucker decomposition (TD) is a standard method for tensor analysis with applications in diverse fields of science and engineering. Despite its success, TD exhibits severe sensitivity against outliers —i.e., heavily corrupted entries that appear sporadically in modern datasets. We study L1-norm TD (L1-TD), a reformulation of TD that promotes robustness. For 3-way tensors, we show, for the first time, that L1-TD admits an exact solution via combinatorial optimization and present algorithms for its solution. We propose two novel algorithmic frameworks for approximating the exact solution to L1-TD, for general N-way tensors. We propose a novel algorithm for dynamic L1-TD —i.e., efficient and joint analysis of streaming tensors. Principal-Component Analysis (PCA) (a special case of TD) is also outlier responsive. We consider Lp-quasinorm PCA (Lp-PCA) for

Three-way compositional data analysis

For the exploratory analysis of three-way data, e.g., measurements of a number of objects, on a number of variables at different points in time, Tucker analysis is one of the most applied technique to study three-way array when the data are approximately trilinear. It can be considered a three way generalization of PCA (Principal Component Analysis). Like PCA, to interpret the results from these methods, it is possible, in addition to inspecting the loadings matrices and core array, inspect visual representation of the outcome. When the data are particular ratios, as in the case of compositional data, these models should consider the special problems that compositional data gives. Aim of this work is describe how an analysis of compositional data by Tucker analysis is possible and how the results should be interpreted. Moreover, a procedure for displaying the results for objects, variables and occasions will be given

Incremental and Adaptive L1-Norm Principal Component Analysis: Novel Algorithms and Applications

L1-norm Principal-Component Analysis (L1-PCA) is known to attain remarkable resistance against faulty/corrupted points among the processed data. However, computing L1-PCA of “big data” with large number of measurements and/or dimensions may be computationally impractical. This work proposes new algorithmic solutions for incremental and adaptive L1-PCA. The first algorithm computes L1-PCA incrementally, processing one measurement at a time, with very low computational and memory requirements; thus, it is appropriate for big data and big streaming data applications. The second algorithm combines the merits of the first one with additional ability to track changes in the nominal signal subspace by revising the computed L1-PCA as new measurements arrive, demonstrating both robustness against outliers and adaptivity to signal-subspace changes. The proposed algorithms are evaluated in an array of experimental studies on subspace estimation, video surveillance (foreground/background separation), image conditioning, and direction-of-arrival (DoA) estimation

Dynamic Algorithms and Asymptotic Theory for Lp-norm Data Analysis

The focus of this dissertation is the development of outlier-resistant stochastic algorithms for Principal Component Analysis (PCA) and the derivation of novel asymptotic theory for Lp-norm Principal Component Analysis (Lp-PCA). Modern machine learning and signal processing applications employ sensors that collect large volumes of data measurements that are stored in the form of data matrices, that are often massive and need to be efficiently processed in order to enable machine learning algorithms to perform effective underlying pattern discovery. One such commonly used matrix analysis technique is PCA. Over the past century, PCA has been extensively used in areas such as machine learning, deep learning, pattern recognition, and computer vision, just to name a few. PCA\u27s popularity can be attributed to its intuitive formulation on the L2-norm, availability of an elegant solution via the singular-value-decomposition (SVD), and asymptotic convergence guarantees. However, PCA has been shown to be highly sensitive to faulty measurements (outliers) because of its reliance on the outlier-sensitive L2-norm. Arguably, the most straightforward approach to impart robustness against outliers is to replace the outlier-sensitive L2-norm by the outlier-resistant L1-norm, thus formulating what is known as L1-PCA. Exact and approximate solvers are proposed for L1-PCA in the literature. On the other hand, in this big-data era, the data matrix may be very large and/or the data measurements may arrive in streaming fashion. Traditional L1-PCA algorithms are not suitable in this setting. In order to efficiently process streaming data, while being resistant against outliers, we propose a stochastic L1-PCA algorithm that computes the dominant principal component (PC) with formal convergence guarantees. We further generalize our stochastic L1-PCA algorithm to find multiple components by propose a new PCA framework that maximizes the recently proposed Barron loss. Leveraging Barron loss yields a stochastic algorithm with a tunable robustness parameter that allows the user to control the amount of outlier-resistance required in a given application. We demonstrate the efficacy and robustness of our stochastic algorithms on synthetic and real-world datasets. Our experimental studies include online subspace estimation, classification, video surveillance, and image conditioning, among other things. Last, we focus on the development of asymptotic theory for Lp-PCA. In general, Lp-PCA for p\u3c2 has shown to outperform PCA in the presence of outliers owing to its outlier resistance. However, unlike PCA, Lp-PCA is perceived as a ``robust heuristic\u27\u27 by the research community due to the lack of theoretical asymptotic convergence guarantees. In this work, we strive to shed light on the topic by developing asymptotic theory for Lp-PCA. Specifically, we show that, for a broad class of data distributions, the Lp-PCs span the same subspace as the standard PCs asymptotically and moreover, we prove that the Lp-PCs are specific rotated versions of the PCs. Finally, we demonstrate the asymptotic equivalence of PCA and Lp-PCA with a wide variety of experimental studies

rTensor: An R Package for Multidimensional Array (Tensor) Unfolding, Multiplication, and Decomposition

rTensor is an R package designed to provide a common set of operations and decompositions for multidimensional arrays (tensors). We provide an S4 class that wraps around the base 'array' class and overloads familiar operations to users of 'array', and we provide additional functionality for tensor operations that are becoming more relevant in recent literature. We also provide a general unfolding operation, for which the k-mode unfolding and the matrix vectorization are special cases of. Finally, package rTensor implements common tensor decompositions such as canonical polyadic decomposition, Tucker decomposition, multilinear principal component analysis, t-singular value decomposition, as well as related matrix-based algorithms such as generalized low rank approximation of matrices and popular value decomposition

Tensor-Train decomposition for image classification problems

In these last years a great effort has been put in the development of new techniques for automatic object classification, also due to the consequences in many applications such as medical imaging or driverless cars. To this end, several mathematical models have been developed from logistic regression to neural networks. A crucial aspect of these so called classification algorithms is the use of algebraic tools to represent and approximate the input data. In this thesis, we examine two different models for image classification based on a particular tensor decomposition named Tensor-Train (TT) decomposition. The use of tensor approaches preserves the multidimensional structure of the data and the neighboring relations among pixels. Furthermore the Tensor-Train, differently from other tensor decompositions, does not suffer from the curse of dimensionality making it an extremely powerful strategy when dealing with high-dimensional data. It also allows data compression when combined with truncation strategies that reduce memory requirements without spoiling classification performance. The first model we propose is based on a direct decomposition of the database by means of the TT decomposition to find basis vectors used to classify a new object. The second model is a tensor dictionary learning model, based on the TT decomposition where the terms of the decomposition are estimated using a proximal alternating linearized minimization algorithm with a spectral stepsize.Negli ultimi anni si è registrato un notevole sviluppo di nuove tecniche per il riconoscimento automatico di oggetti, anche dovuto alle possibili ricadute di tali avanzamenti nel campo medico o automobilistico. A tal fine sono stati sviluppati svariati modelli matematici dai metodi di regressione fino alle reti neurali. Un aspetto cruciale di questi cosiddetti algoritmi di classificazione è l'uso di aspetti algebrici per la rappresentazione e l'approssimazione dei dati in input. In questa tesi esamineremo due diversi modelli per la classificazione di immagini basati sulla decomposizione Tensor-Train (TT). In generale, l'uso di approcci tensoriali è fondamentale per preservare la struttura intrinsecamente multidimensionale dei dati. Inoltre l'occupazione di memoria per la decomposizione Tensor-Train non cresce esponenzialmente all'aumentare dei dati, a differenza di altre decomposizioni tensoriali. Questo la rende particolarmente adatta nel caso di dati di grandi dimensioni. Inoltre permette, attraverso l'uso di opportune strategie di troncamento, di limitare notevolmente l'occupazione di memoria senza ricadute negative sulle performance di classificazione. Il primo modello proposto in questa tesi è basato su una decomposizione diretta del database tramite la decomposizione TT. In questo modo viene determinata una base che verrà di seguito utilizzata nella classificazione di nuove immagini. Il secondo è invece un modello di dictionary learning tensoriale sempre basato sulla decomposizione TT in cui i termini della decomposizione sono determinati utilizzando un nuovo metodo di ottimizzazione alternato con l'utilizzo di passi spettrali

Hybrid Beamforming Design for Millimeter Wave Massive MIMO Communications

Wireless connectivity is a key driver for the digital transformation that is changing how people communicate, do business, consume entertainment and search for information. As the world advances into the fifth generation (5G) and beyond-5G (B5G) of wireless mobile technology, new services and use cases are emerging every day, bringing the demand to expand the broadband capability of mobile networks and to provide ubiquitous access and specific capabilities for any device or application. To fulfill these demands, 5G and B5G systems will rely on innovative technologies, such as the ultradensification, the mmWave, and the massive MIMO. To bring together these technologies, 5G and B5G systems will employ hybrid analog-digital beamforming, which separates the signal processing into the baseband (digital) and the radio-frequency (analog) domains. Unlike conventional beamforming, where every antenna is connected to an RF chain, and the signal is entirely processed in the digital domain, hybrid beamforming uses fewer RF chains than the total number of antennas, resulting in a less expensive and less energy-consuming design. The analog beamforming is usually implemented using switching networks or phase-shifting networks, which impose severe hardware constraints making the hybrid beamforming design very challenging. This thesis addresses the hybrid analog-digital beamforming design and is organized into three parts. In the first part, two adaptive algorithms for solving the switching-network-based hybrid beamforming design problem, also known as the joint antenna selection and beamforming (JASB) problem, are proposed. The adaptive algorithms are based on the minimum mean square error (MMSE) and minimum-variance distortionless response (MVDR) criteria and employ an alternating optimization strategy, in which the beamforming and the antenna selection are designed iteratively. The proposed algorithms can attain high levels of SINR while strictly complying with the hardware limitations. Moreover, the proposed algorithms have very low computational complexity and can track channel variations, making them suitable for non-stationary environments. Numerical simulations have validated the effectiveness of the algorithms in different operation scenarios. The second part addresses the phase-shifting-network-based hybrid beamforming design for narrowband mmWave massive MIMO systems. A novel joint hybrid precoder and combiner design is proposed. The analog precoder and combiner design is formulated as constrained low-rank channel decomposition, which can simultaneously harvest the array gain provided by the massive MIMO system and suppress intra-user and inter-user interferences. The constrained low-rank channel decomposition is solved as a series of successive rank-1 channel decomposition, using the projected block coordinate descent method. The digital precoder and combiner are obtained from the optimal SVD-based solution for the single-user case and the regularized channel diagonalization method for the multi-user case. Simulation results have demonstrated that the proposed design can consistently attain near-optimal performance and provided important insights into the method's convergence and its performance under practical phase-shifter quantization constraints. Finally, the phase-shifting-network-based hybrid beamforming design for frequency-selective mmWave massive MIMO-OFDM systems is considered in the third part. The hybrid beamforming design for MIMO-OFDM systems is significantly more challenging than for narrowband MIMO systems since, in these systems, the analog precoder and combiner are shared among all subcarriers and must be jointly optimized. Thus, by leveraging the OFDM systems' multidimensional structure, the analog precoder and combiner design is formulated as constrained low-rank Tucker2 tensor decomposition and solved by a successive rank-(1,1) tensor decomposition using the projected alternate least square (ALS) method. The digital precoder and combiner are obtained on a per-subcarrier basis using the techniques presented in the second part. Numerical simulations have confirmed the design effectiveness, demonstrating its ability to consistently attain near-optimal performance and outperform other existing design in nearly all scenarios. They also provided insights into the convergence of the proposed method and its performance under practical phase-shifter quantization constraints and highlighted the differences between this design and that for the narrowband massive MIMO systems