Sparse Machine Learning Methods for Autonomous Decision Making

Sparse regression methods are used for the reconstruction of compressed signals, that are usually sparse in some bases; or in feature selection problem, where only few features are meaningful. This thesis overviews the existing Bayesian methods for dealing with sparsity, improves them and provides new models for these problems. The novel models decrease complexity, allow to model structure and provide uncertainty distributions in such applications as medicine and computer vision. The thesis starts with exploring Bayesian sparsity for the problem of compressive back- ground subtraction. Sparsity naturally arises in this problem as foreground usually occupies only small part of the video frame. The use of Bayesian compressive sensing improves the solutions in independent and multi-task scenarios. It also raises an important problem of exploring the structure of the data, as foreground pixels are usually clustered in groups. The problem of structure modelling in sparse problems is addressed with hierarchical Gaussian processes, that are the Bayesian way of imposing structure without specifying its exact patterns. Full Bayesian inference based on expectation propagation is provided for offline and online algorithms. The experiments demonstrate the applicability of these methods for the compressed background subtraction and brain activity localisation problems. The majority of sparse Bayesian methods are computationally intensive. This thesis proposes a novel sparse regression method based on the Bayesian neural networks. It makes the prediction operation fast and additionally estimates the uncertainty of predictions, while requiring a longer training phase. The results are demonstrated in the active learning scenario, where the estimated uncertainty is used for experiment design. Sparse methods are also used as part of other methods such as Gaussian processes that suffer from high computational complexity. The use of active sparse subsets of data improves the performance on large datasets. The thesis proposes a method of dealing with the complexity problem for online data updates using Bayesian filtering

Compressed Sensing Beyond the IID and Static Domains: Theory, Algorithms and Applications

Sparsity is a ubiquitous feature of many real world signals such as natural images and neural spiking activities. Conventional compressed sensing utilizes sparsity to recover low dimensional signal structures in high ambient dimensions using few measurements, where i.i.d measurements are at disposal. However real world scenarios typically exhibit non i.i.d and dynamic structures and are confined by physical constraints, preventing applicability of the theoretical guarantees of compressed sensing and limiting its applications. In this thesis we develop new theory, algorithms and applications for non i.i.d and dynamic compressed sensing by considering such constraints. In the first part of this thesis we derive new optimal sampling-complexity tradeoffs for two commonly used processes used to model dependent temporal structures: the autoregressive processes and self-exciting generalized linear models. Our theoretical results successfully recovered the temporal dependencies in neural activities, financial data and traffic data. Next, we develop a new framework for studying temporal dynamics by introducing compressible state-space models, which simultaneously utilize spatial and temporal sparsity. We develop a fast algorithm for optimal inference on such models and prove its optimal recovery guarantees. Our algorithm shows significant improvement in detecting sparse events in biological applications such as spindle detection and calcium deconvolution. Finally, we develop a sparse Poisson image reconstruction technique and the first compressive two-photon microscope which uses lines of excitation across the sample at multiple angles. We recovered diffraction-limited images from relatively few incoherently multiplexed measurements, at a rate of 1.5 billion voxels per second

Compressive Sensing-Based Grant-Free Massive Access for 6G Massive Communication

The advent of the sixth-generation (6G) of wireless communications has given rise to the necessity to connect vast quantities of heterogeneous wireless devices, which requires advanced system capabilities far beyond existing network architectures. In particular, such massive communication has been recognized as a prime driver that can empower the 6G vision of future ubiquitous connectivity, supporting Internet of Human-Machine-Things for which massive access is critical. This paper surveys the most recent advances toward massive access in both academic and industry communities, focusing primarily on the promising compressive sensing-based grant-free massive access paradigm. We first specify the limitations of existing random access schemes and reveal that the practical implementation of massive communication relies on a dramatically different random access paradigm from the current ones mainly designed for human-centric communications. Then, a compressive sensing-based grant-free massive access roadmap is presented, where the evolutions from single-antenna to large-scale antenna array-based base stations, from single-station to cooperative massive multiple-input multiple-output systems, and from unsourced to sourced random access scenarios are detailed. Finally, we discuss the key challenges and open issues to shed light on the potential future research directions of grant-free massive access.Comment: Accepted by IEEE IoT Journa

COMPRESSIVE SENSING FOR SPECTRAL DOMAIN OPTICAL COHERENCE TOMOGRAPHY

Spectral domain optical coherence tomography (SD OCT) imaging with high axial resolution and a large imaging depth requires a large number of sampling points in the spectral domain. This requires a high-resolution spectrometer with a large linear array camera which leads to a large amount of k-space measurements and a long data acquisition time that makes the imaging susceptible to unavoidable motion artifact. Furthermore such devices can be expensive and require high-speed electronics. In this dissertation, compressive sensing (CS) SD OCT that reconstructs the images using only a portion of the k-space measurements required by the classical Shannon/Nyquist rate was proposed and studied. Several advanced CS SD OCT algorithms have been developed and evaluated. First, modified non-uniform discrete Fourier transform (MNUDFT) matrix was proposed, which enables CS SD OCT using under-sampled non-linear wavenumber spectral data. Second, the noise reduction using Modified-CS was studied which shows that the averaged Modified-CS SD OCT results in better image quality in terms of SNR, local contrast and contrast to noise ratio (CNR), compared to the classical averaging method. Third, a novel three-dimensional (3D) CS SD OCT sampling pattern and reconstruction procedure was proposed. The novel 3D approach enables efficient volumetric image reconstruction using the k-space measurements under-sampled in all three directions and reduces the amount of required measurements to less than 20% of that required by regular SD OCT. CS SD OCT is commonly solved by an iterative algorithm that requires numerous matrix-vector computation, which is computationally complex and time-consuming if solved on CPU-based systems. However, such computation is ideal for parallel processing with graphics processing unit (GPU) which can significantly reduce its computation time. In this dissertation, real-time CS SD OCT was developed on a conventional desktop computer architecture having three GPUs. The GPU-accelerated CS non-uniform in k-space SD OCT and real-time CS SD OCT with dispersion compensation were also proposed and implemented using the same computer architecture. %Real-time CS SD OCT with dispersion compensation was proposed

Learning Overcomplete Dictionaries Based on Atom-by-Atom Updating

International audienceA dictionary learning algorithm learns a set of atoms from some training signals in such a way that each signal can be approximated as a linear combination of only a few atoms. Most dictionary learning algorithms use a two-stage iterative procedure. The first stage is to spars ely approximate the training signals over the current dictionary. The second stage is the update of the dictionary. In this paper we develop some atom-by-atom dictionary learning algorithms, which update the atoms sequentially. Specifically, we propose an efficient alternative to the well-known K-SVD algorithm, and show by various experiments that the proposed algorithm is much faster than K-SVD while its results are better. Moreover, we propose a novel algorithm that instead of alternating between the two dictionary learning stages, performs only the second stage. While in K-SVD each atom is updated along with the nonzero entries of its associated row vector in the coefficient matrix (which we name it its profile), in the new algorithm, each atom is updated along with the whole entries of its profile. As a result, contrary to K-SVD, the support of each profile can be changed while updating the dictionary. To further accelerate the convergence of this algorithm and to have a control on the cardinality of the representations, we then propose its two-stage counterpart by adding the sparse approximation stage. Experimental results on recovery of a known synthetic dictionary and dictionary learning for a class of auto-regressive signals demonstrate the promising performance of the proposed algorithms

Intelligent Biosignal Processing in Wearable and Implantable Sensors

This reprint provides a collection of papers illustrating the state-of-the-art of smart processing of data coming from wearable, implantable or portable sensors. Each paper presents the design, databases used, methodological background, obtained results, and their interpretation for biomedical applications. Revealing examples are brain–machine interfaces for medical rehabilitation, the evaluation of sympathetic nerve activity, a novel automated diagnostic tool based on ECG data to diagnose COVID-19, machine learning-based hypertension risk assessment by means of photoplethysmography and electrocardiography signals, Parkinsonian gait assessment using machine learning tools, thorough analysis of compressive sensing of ECG signals, development of a nanotechnology application for decoding vagus-nerve activity, detection of liver dysfunction using a wearable electronic nose system, prosthetic hand control using surface electromyography, epileptic seizure detection using a CNN, and premature ventricular contraction detection using deep metric learning. Thus, this reprint presents significant clinical applications as well as valuable new research issues, providing current illustrations of this new field of research by addressing the promises, challenges, and hurdles associated with the synergy of biosignal processing and AI through 16 different pertinent studies. Covering a wide range of research and application areas, this book is an excellent resource for researchers, physicians, academics, and PhD or master students working on (bio)signal and image processing, AI, biomaterials, biomechanics, and biotechnology with applications in medicine

l0 Sparse signal processing and model selection with applications

Sparse signal processing has far-reaching applications including compressed sensing, media compression/denoising/deblurring, microarray analysis and medical imaging. The main reason for its popularity is that many signals have a sparse representation given that the basis is suitably selected. However the difficulty lies in developing an efficient method of recovering such a representation. To this aim, two efficient sparse signal recovery algorithms are developed in the first part of this thesis. The first method is based on direct minimization of the l0 norm via cyclic descent, which is called the L0LS-CD (l0 penalized least squares via cyclic descent) algorithm. The other method minimizes smooth approximations of sparsity measures including those of the l0 norm via the majorization minimization (MM) technique, which is called the QC (quadratic concave) algorithm. The L0LS-CD algorithm is developed further by extending it to its multivariate (V-L0LS-CD (vector L0LS-CD)) and group (gL0LS-CD (group L0LS-CD)) regression variants. Computational speed-ups to the basic cyclic descent algorithm are discussed and a greedy version of L0LS-CD is developed. Stability of these algorithms is analyzed and the impact of the penalty parameter and proper initialization on the algorithm performance are highlighted. A suitable method for performance comparison of sparse approximating algorithms in the presence of noise is established. Simulations compare L0LS-CD and V-L0LS-CD with a range of alternatives on under-determined as well as over-determined systems. The QC algorithm is applicable to a class of penalties that are neither convex nor concave but have what we call the quadratic concave property. Convergence proofs of this algorithm are presented and it is compared with the Newton algorithm, concave convex (CC) procedure, as well as with the class of proximity algorithms. Simulations focus on the smooth approximations of the l0 norm and compare them with other l0 denoising algorithms. Next, two applications of sparse modeling are considered. In the first application the L0LS-CD algorithm is extended to recover a sparse transfer function in the presence of coloured noise. The second uses gL0LS-CD to recover the topology of a sparsely connected network of dynamic systems. Both applications use Laguerre basis functions for model expansion. The role of model selection in sparse signal processing is widely neglected in literature. The tuning/penalty parameter of a sparse approximating problem should be selected using a model selection criterion which minimizes a desired discrepancy measure. Compared to the commonly used model selection methods, the SURE (Stein's unbiased risk estimator) estimator stands out as one which does not suffer from the limitations of other methods. Most model selection criterion are developed based on signal or prediction mean squared error. The last section of this thesis develops an SURE criterion instead for parameter mean square error and applies this result to l1 penalized least squares problem with grouped variables. Simulations based on topology identification of a sparse network are presented to illustrate and compare with alternative model selection criteria

Sparse Signal Processing and Statistical Inference for Internet of Things

Data originating from many devices within the Internet of Things (IoT) framework can be modeled as sparse signals. Efficient compression techniques of such data are essential to reduce the memory storage, bandwidth, and transmission power. In this thesis, I develop some theory and propose practical schemes for IoT applications to exploit the signal sparsity for efficient data acquisition and compression under the frameworks of compressed sensing (CS) and transform coding. In the context of CS, the restricted isometry constant of finite Gaussian measurement matrices is investigated, based on the exact distributions of the extreme eigenvalues of Wishart matrices. The analysis determines how aggressively the signal can be sub-sampled and recovered from a small number of linear measurements. The signal reconstruction is guaranteed, with a predefined probability, via various recovery algorithms. Moreover, the measurement matrix design for simultaneously acquiring multiple signals is considered. This problem is important for IoT networks, where a huge number of nodes are involved. In this scenario, the presented analytical methods provide limits on the compression of joint sparse sources by analyzing the weak restricted isometry constant of Gaussian measurement matrices. Regarding transform coding, two efficient source encoders for noisy sparse sources are proposed, based on channel coding theory. The analytical performance is derived in terms of the operational rate-distortion and energy-distortion. Furthermore, a case study for the compression of real signals from a wireless sensor network using the proposed encoders is considered. These techniques can reduce the power consumption and increase the lifetime of IoT networks. Finally, a frame synchronization mechanism has been designed to achieve reliable radio links for IoT devices, where optimal and suboptimal metrics for noncoherent frame synchronization are derived. The proposed tests outperform the commonly used correlation detector, leading to accurate data extraction and reduced power consumption

Wavelet and Multiscale Methods

Various scientific models demand finer and finer resolutions of relevant features. Paradoxically, increasing computational power serves to even heighten this demand. Namely, the wealth of available data itself becomes a major obstruction. Extracting essential information from complex structures and developing rigorous models to quantify the quality of information leads to tasks that are not tractable by standard numerical techniques. The last decade has seen the emergence of several new computational methodologies to address this situation. Their common features are the nonlinearity of the solution methods as well as the ability of separating solution characteristics living on different length scales. Perhaps the most prominent examples lie in multigrid methods and adaptive grid solvers for partial differential equations. These have substantially advanced the frontiers of computability for certain problem classes in numerical analysis. Other highly visible examples are: regression techniques in nonparametric statistical estimation, the design of universal estimators in the context of mathematical learning theory and machine learning; the investigation of greedy algorithms in complexity theory, compression techniques and encoding in signal and image processing; the solution of global operator equations through the compression of fully populated matrices arising from boundary integral equations with the aid of multipole expansions and hierarchical matrices; attacking problems in high spatial dimensions by sparse grid or hyperbolic wavelet concepts. This workshop proposed to deepen the understanding of the underlying mathematical concepts that drive this new evolution of computation and to promote the exchange of ideas emerging in various disciplines

Sparse channel estimation based on compressed sensing theory for UWB systems

Català: L'estimació de canal en receptors wireless esdevé un factor determinant a l'hora de incrementar les prestacions dels sistemes sense fils per tal de satisfer les exigències cada vegades més elevades dels consumidors en quant a velocitats de transmissió i qualitat. En aquesta tesi es proposa explotar la "sparsity" que mostren els canals wireless per tal de millorar els clàssics sistemes d'estimació de canal mitjançant les noves teòries de Compressed Sensing. Així doncs, es proposa un nou model freqüencial de senyal on el canal i un nou algoritme de reconstrucció de senyals sparse que redueix la probabilitat de detecció de falsos camins de propagació millorant d'aquesta manera l'estimació de temps d'arribada.Castellano: En los últimos años, la revolución inalámbrica se ha convertido en una realidad. Wi-fi está en todas partes, impactando significativamente en nuestro estilo de vida. Sin embargo, las comunicaciones inalámbricas nunca tendrán las condiciones de propagación igual que los cables debido a las duras condiciones de la propagación inalámbricas. El canal de radio móvil se caracteriza por la recepción múltiple, eso es que la señal recibida no sólo contiene una camino de propagación, sino también un gran número de ondas reflejadas. Estas ondas reflejadas interfieren con la onda directa, lo que provoca una degradación significativa del rendimiento del enlace. Un sistema inalámbrico debe estar diseñado de tal manera que el efecto adverso del desvanecimiento multicamino sea reducido al mínimo. Afortunadamente, el multipath puede ser visto como diversidad de información dependiendo de la cantidad de Channel State Information (CSI) disponible para el sistema. Sin embargo, en la práctica CSI rara vez se dispone a priori y debe ser estimado. Por otro lado, un canal inalámbrico a menudo puede ser modelado como un canal sparse, en la que el retraso de propagación puede ser muy grande, pero el número de caminos de propagación es normalmente muy pequeño. El conocimiento previo de la sparsity del canal se puede utilizar eficazmente para mejorar la estimación de canal utilizando la nueva teoría de Compressed Sensing (CS). CS se origina en la idea de que no es necesario invertir una gran cantidad de energía en la observación de las entradas de una señal sparse porque la mayoría de ellas será cero. Por lo tanto, CS proporciona un marco sólido para la reducción del número de medidas necesarias para resumir señales sparse. La estimación de canal sparse se centra en este trabajo en Ultra-Wideband (UWB) porque la gran resolución temporal que proporcionan las señales UWB se traduce en un número muy grande de componentes multipath que se pueden resolver. Por lo tanto, UWB mitiga significativamente la distorsión de trayectoria múltiple y proporciona la diversidad multicamino. Esta diversidad junto con la resolución temporal de las señales UWB crear un problema de estimación de canal muy interesante. En esta tesis se estudia el uso de CS en la estimación de canal altamente sparse por medio de un nuevo enfoque de estimación basado en el modelo de frecuencial de la señal UWB. También se propone un nuevo algoritmo llamado extended Orthogonal Matching Pursuit (eOMP) basado en los mismos principios que el clásico OMP, con el fin de mejorar algunas de sus característica.English: In recent years, the wireless revolution has become a reality. Wireless is everywhere having significant impact on our lifestyle. However, wireless will never have the same propagation conditions as wires due to the harsh conditions of the wireless propagation. The mobile radio channel is characterized by multipath reception, that is the signal offered to the receiver contains not only a direct line-of-sight radio wave, but also a large number of reflected radio waves. These reflected waves interfere with the direct wave, which causes significant degradation of the performance of the link. A wireless system has to be designed in such way that the adverse effect of multipath fading is minimized. Fortunately, multipath can be seen as a blessing depending on the amount of Channel State Information (CSI) available to the system. However, in practise CSI is seldom available a priori and needs to be estimated. On the other hand, a wireless channel can often be modeled as a sparse channel in which the delay spread could be very large, but the number of significant paths is normally very small. The prior knowledge of the channel sparseness can be effectively use to improve the channel estimation using the novel Compressed Sensing (CS) theory. CS originates from the idea that is not necessary to invest a lot of power into observing the entries of a sparse signal because most of them will be zero. Therefore, CS provides a robust framework for reducing the number of measurement required to summarize sparse signals. The sparse channel estimation here is focused on Ultra-WideBand (UWB) systems because the very fine time resolution of the UWB signal results in a very large number of resolvable multipath components. Consequently, UWB significantly mitigates multipath distortion and provides path diversity. The rich multipath coupled with the fine time resolution of the UWB signals create a challenging sparse channel estimation problem. This Master Thesis examines the use of CS in the estimation of highly sparse channel by means of a new sparse channel estimation approach based on the frequency domain model of the UWB signal. It is also proposed a new greedy algorithm named extended Orthogonal Matching Pursuit (eOMP) based on the same principles than classical Orthogonal Matching Pursuit (OMP) in order to improve some OMP characteristics. Simulation results show that the new eOMP provides lower false path detection probability compared with classical OMP, which also leads to a better TOA estimation without significant degradation of the channel estimation. Simulation results will also show that the new frequency domain sparse channel model outperforms other models presented in the literature