Structured Sparsity: Discrete and Convex approaches

Compressive sensing (CS) exploits sparsity to recover sparse or compressible signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity is also used to enhance interpretability in machine learning and statistics applications: While the ambient dimension is vast in modern data analysis problems, the relevant information therein typically resides in a much lower dimensional space. However, many solutions proposed nowadays do not leverage the true underlying structure. Recent results in CS extend the simple sparsity idea to more sophisticated {\em structured} sparsity models, which describe the interdependency between the nonzero components of a signal, allowing to increase the interpretability of the results and lead to better recovery performance. In order to better understand the impact of structured sparsity, in this chapter we analyze the connections between the discrete models and their convex relaxations, highlighting their relative advantages. We start with the general group sparse model and then elaborate on two important special cases: the dispersive and the hierarchical models. For each, we present the models in their discrete nature, discuss how to solve the ensuing discrete problems and then describe convex relaxations. We also consider more general structures as defined by set functions and present their convex proxies. Further, we discuss efficient optimization solutions for structured sparsity problems and illustrate structured sparsity in action via three applications.Comment: 30 pages, 18 figure

Doctor of Philosophy

dissertationCine phase contrast (PC) magnetic resonance imaging (MRI) is a useful imaging technique that allows for the quantitative measurement of in-vivo blood velocities over the cardiac cycle. Velocity information can be used to diagnose and learn more about the mechanisms of cardio-vascular disease. Compared to other velocity measuring techniques, PC MRI provides high-resolution 2D and 3D spatial velocity information. Unfortunately, as with many other MRI techniques, PC MRI su ers from long acquisition times which places constraints on temporal and spatial resolution. This dissertation outlines the use of temporally constrained reconstruction (TCR) of radial PC data in order to signi cantly reduce the acquisition time so that higher temporal and spatial resolutions can be achieved. A golden angle-based acquisition scheme and a novel self-gating method were used in order to allow for exible selection of temporal resolution and to ameliorate the di culties associated with external electrocardiogram (ECG) gating. Finally, image reconstruction times for TCR are signi cantly reduced by implementation on a high-performance computer cluster. The TCR algorithm is executed in parallel across multiple GPUs achieving a 50 second reconstruction time for a very large cardiac perfusion data set

Adaptive Sampling and Statistical Inference for Anomaly Detection

Given the rising threat of malware and the increasing inadequacy of signature-based solutions, online performance monitoring has emerged as a critical component of the security infrastructure of data centers and networked systems. Most of the systems that require monitoring are usually large-scale, highly dynamic and time-evolving. These facts add to the complexity of both monitoring and the underlying techniques for anomaly detection. Furthermore, one cannot ignore the costs associated with monitoring and detection which can interfere with the normal operation of a system and deplete the supply of resources available for the system. Therefore, securing modern systems calls for efficient monitoring strategies and anomaly detection techniques that can deal with massive data with high efficiency and report unusual events effectively. This dissertation contributes new algorithms and implementation strategies toward a significant improvement in the effectiveness and efficiency of two components of security infrastructure: (1) system monitoring and (2) anomaly detection. For system monitoring purposes, we develop two techniques which reduce the cost associated with information collection: i) a non-sampling technique and ii) a sampling technique. The non-sampling technique is based on compression and employs the best basis algorithm to automatically select the basis for compressing the data according to the structure of the data. The sampling technique improves upon compressive sampling, a recent signal processing technique for acquiring data at low cost. This enhances the technique of compressive sampling by employing it in an adaptive-rate model wherein the sampling rate for compressive sampling is adaptively tuned to the data being sampled. Our simulation results on measurements collected from a data center show that these two data collection techniques achieve small information loss with reduced monitoring cost. The best basis algorithm can select the basis in which the data is most concisely represented, allowing a reduced sample size for monitoring. The adaptive-rate model for compressive sampling allows us to save 70% in sample size, compared with the constant-rate model. For anomaly detection, this dissertation develops three techniques to allow efficient detection of anomalies. In the first technique, we exploit the properties maintained in the samples of compressive sampling and apply state-of-the-art anomaly detection techniques directly to compressed measurements. Simulation results show that the detection rate of abrupt changes using the compressed measurements is greater than 95% when the size of the measurements is only 18%. In our second approach, we characterize performance-related measurements as a stream of covariance matrices, one for each designated window of time, and then propose a new metric to quantify changes in the covariance matrices. The observed changes are then employed to infer anomalies in the system. In our third approach, anomalies in a system are detected using a low-complexity distributed algorithm when only steams of raw measurement vectors, one for each time window, are available and distributed among multiple locations. We apply our techniques on real network traffic data and show that these two techniques furnish existing methods with more details about the anomalous changes.Ph.D., Electrical Engineering -- Drexel University, 201

New and Provable Results for Network Inference Problems and Multi-agent Optimization Algorithms

abstract: Our ability to understand networks is important to many applications, from the analysis and modeling of biological networks to analyzing social networks. Unveiling network dynamics allows us to make predictions and decisions. Moreover, network dynamics models have inspired new ideas for computational methods involving multi-agent cooperation, offering effective solutions for optimization tasks. This dissertation presents new theoretical results on network inference and multi-agent optimization, split into two parts - The first part deals with modeling and identification of network dynamics. I study two types of network dynamics arising from social and gene networks. Based on the network dynamics, the proposed network identification method works like a `network RADAR', meaning that interaction strengths between agents are inferred by injecting `signal' into the network and observing the resultant reverberation. In social networks, this is accomplished by stubborn agents whose opinions do not change throughout a discussion. In gene networks, genes are suppressed to create desired perturbations. The steady-states under these perturbations are characterized. In contrast to the common assumption of full rank input, I take a laxer assumption where low-rank input is used, to better model the empirical network data. Importantly, a network is proven to be identifiable from low rank data of rank that grows proportional to the network's sparsity. The proposed method is applied to synthetic and empirical data, and is shown to offer superior performance compared to prior work. The second part is concerned with algorithms on networks. I develop three consensus-based algorithms for multi-agent optimization. The first method is a decentralized Frank-Wolfe (DeFW) algorithm. The main advantage of DeFW lies on its projection-free nature, where we can replace the costly projection step in traditional algorithms by a low-cost linear optimization step. I prove the convergence rates of DeFW for convex and non-convex problems. I also develop two consensus-based alternating optimization algorithms --- one for least square problems and one for non-convex problems. These algorithms exploit the problem structure for faster convergence and their efficacy is demonstrated by numerical simulations. I conclude this dissertation by describing future research directions.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

Compressed Sensing For Functional Magnetic Resonance Imaging Data

This thesis addresses the possibility of applying the compressed sensing (CS) framework to Functional Magnetic Resonance Imaging (fMRI) acquisition. The fMRI is one of the non-invasive neuroimaging technique that allows the brain activity to be captured and analysed in a living body. One disadvantage of fMRI is the trade-off between the spatial and temporal resolution of the data. To keep the experiments within a reasonable length of time, the current acquisition technique sacrifices the spatial resolution in favour of the temporal resolution. It is possible to improve this trade-off using compressed sensing. The main contribution of this thesis is to propose a novel reconstruction method, named Referenced Compressed Sensing, which exploits the redundancy between a signal and a correlated reference by using their distance as an objective function. The compressed video sequences reconstructed using Referenced CS have at least 50% higher in terms of Peak Signal-to-Noise Ratio (PSNR) compared to state-of-the-art conventional reconstruction methods. This thesis also addresses two issues related to Referenced CS. Firstly, the relationship between the reference and the reconstruction performance is studied. To maintain the high-quality references, the Running Gaussian Average (RGA) reference estimator is proposed. The reconstructed results have at least 3dB better PSNR performance with the use of RGA references. Secondly, the Referenced CS with Least Squares is proposed. This study shows that by incorporating the correlated reference, it is possible to perform a linear reconstruction as opposed to the iterative reconstruction commonly used in CS. This approach gives at least 19% improvement in PSNR compared to the state of the art, while reduces the computation time by at most 1200 times. The proposed method is applied to the fMRI data. This study shows that, using the same amount of samples, the data reconstructed using Referenced CS has higher resolution than the conventional acquisition technique and has on average 50% higher PSNR than state-of-the-art reconstructions. Lastly, to enhance the feature of interest in the fMRI data, the baseline independent (BI) analysis is proposed. Using the BI analysis shows up to 25% improvement in the accuracy of the Referenced CS feature

Sparsity driven ground moving target indication in synthetic aperture radar

Synthetic aperture radar (SAR) was first invented in the early 1950s as the remote surveillance instruments to produce high resolution 2D images of the illuminated scene with weather-independent, day-or-night performance. Compared to the Real Aperture Radar (RAR), SAR is synthesising a large virtual aperture by moving a small antenna along the platform path. Typical SAR imaging systems are designed with the basic assumption of a static scene, and moving targets are widely known to induce displacements and defocusing in the formed images. While the capabilities of detection, states estimation and imaging for moving targets with SAR are highly desired in both civilian and military applications, the Ground Moving Target Indication (GMTI) techniques can be integrated into SAR systems to realise these challenging missions. The state-of-the- art SAR-based GMTI is often associated with multi-channel systems to improve the detection capabilities compared to the single-channel ones. Motivated by the fact that the SAR imaging is essentially solving an optimisation problem, we investigate the practicality to reformulate the GMTI process into the optimisation form. Furthermore, the moving target sparsities and underlying similarities between the conventional GMTI processing and sparse reconstruction algorithms drive us to consider the compressed sensing theory in SAR/GMTI applications. This thesis aims to establish an end-to-end SAR/GMTI processing framework regularised by target sparsities based on multi-channel SAR models. We have explained the mathematical model of the SAR system and its key properties in details. The common GMTI mechanism and basics of the compressed sensing theory are also introduced in this thesis. The practical implementation of the proposed framework is provided in this work. The developed model is capable of realising various SAR/GMTI tasks including SAR image formation, moving target detection, target state estimation and moving target imaging. We also consider two essential components, i.e. the data pre-processing and elevation map, in this work. The effectiveness of the proposed framework is demonstrated through both simulations and real data. Given that our focus in this thesis is on the development of a complete sparsity-aided SAR/GMTI framework, the contributions of this thesis can be summarised as follows. First, the effects of SAR channel balancing techniques and elevation information in SAR/GMTI applications are analysed in details. We have adapted these essential components to the developed framework for data pre-processing, system specification estimation and better SAR/GMTI accuracies. Although the purpose is on enhancing the proposed sparsity-based SAR/GMTI framework, the exploitation of the DEM in other SAR/GMTI algorithms may be of independent interest. Secondly, we have designed a novel sparsity-aided framework which integrates the SAR/GMTI missions, i.e. SAR imaging, moving target and background decomposition, and target state estimation, into optimisation problems. A practical implementation of the proposed framework with a two stage process and theoretically/experimentally proven algorithms are proposed in this work. The key novelty on utilising optimisations and target sparsities is explained in details. Finally, a practical algorithm for moving target imaging and state estimation is developed to accurately estimate the full target parameters and form target images with relocation and refocusing capabilities. Compared to the previous processing steps for practical applications, the designed algorithm consistently relies on the exploitation of target sparsities which forms the final processing stage of the whole pipeline. All the developed components contribute coherently to establish a complete sparsity driven SAR/GMTI processing framework

Super Resolution of Wavelet-Encoded Images and Videos

In this dissertation, we address the multiframe super resolution reconstruction problem for wavelet-encoded images and videos. The goal of multiframe super resolution is to obtain one or more high resolution images by fusing a sequence of degraded or aliased low resolution images of the same scene. Since the low resolution images may be unaligned, a registration step is required before super resolution reconstruction. Therefore, we first explore in-band (i.e. in the wavelet-domain) image registration; then, investigate super resolution. Our motivation for analyzing the image registration and super resolution problems in the wavelet domain is the growing trend in wavelet-encoded imaging, and wavelet-encoding for image/video compression. Due to drawbacks of widely used discrete cosine transform in image and video compression, a considerable amount of literature is devoted to wavelet-based methods. However, since wavelets are shift-variant, existing methods cannot utilize wavelet subbands efficiently. In order to overcome this drawback, we establish and explore the direct relationship between the subbands under a translational shift, for image registration and super resolution. We then employ our devised in-band methodology, in a motion compensated video compression framework, to demonstrate the effective usage of wavelet subbands. Super resolution can also be used as a post-processing step in video compression in order to decrease the size of the video files to be compressed, with downsampling added as a pre-processing step. Therefore, we present a video compression scheme that utilizes super resolution to reconstruct the high frequency information lost during downsampling. In addition, super resolution is a crucial post-processing step for satellite imagery, due to the fact that it is hard to update imaging devices after a satellite is launched. Thus, we also demonstrate the usage of our devised methods in enhancing resolution of pansharpened multispectral images