72 research outputs found
Combining the Burrows-Wheeler Transform and RCM-LDGM Codes for the Transmission of Sources with Memory at High Spectral Efficiencies
In this paper, we look at the problem of implementing high-throughput Joint SourceChannel (JSC) coding schemes for the transmission of binary sources with memory over AWGN channels. The sources are modeled either by a Markov chain (MC) or a hidden Markov model (HMM). We propose a coding scheme based on the Burrows-Wheeler Transform (BWT) and the parallel concatenation of Rate-Compatible Modulation and Low-Density Generator Matrix (RCM-LDGM) codes. The proposed scheme uses the BWT to convert the original source with memory into a set of independent non-uniform Discrete Memoryless (DMS) binary sources, which are then separately encoded, with optimal rates, using RCM-LDGM codes
When Machine Learning Meets Information Theory: Some Practical Applications to Data Storage
Machine learning and information theory are closely inter-related areas. In this dissertation,
we explore topics in their intersection with some practical applications to data storage.
Firstly, we explore how machine learning techniques can be used to improve data reliability
in non-volatile memories (NVMs). NVMs, such as flash memories, store large volumes of data.
However, as devices scale down towards small feature sizes, they suffer from various kinds of noise and disturbances, thus significantly reducing their reliability. This dissertation explores machine learning techniques to design decoders that make use of natural redundancy (NR) in data for error correction. By NR, we mean redundancy inherent in data, which is not added artificially for error correction. This work studies two different schemes for NR-based error-correcting decoders. In the first scheme, the NR-based decoding algorithm is aware of the data representation scheme (e.g., compression, mapping of symbols to bits, meta-data, etc.), and uses that information for error correction. In the second scenario, the NR-decoder is oblivious of the representation scheme and uses deep neural networks (DNNs) to recognize the file type as well as perform soft decoding on it based on NR. In both cases, these NR-based decoders can be combined with traditional error correction codes (ECCs) to substantially improve their performance.
Secondly, we use concepts from ECCs for designing robust DNNs in hardware. Non-volatile
memory devices like memristors and phase-change memories are used to store the weights of
hardware implemented DNNs. Errors and faults in these devices (e.g., random noise, stuck-at
faults, cell-level drifting etc.) might degrade the performance of such DNNs in hardware. We use
concepts from analog error-correcting codes to protect the weights of noisy neural networks and to design robust neural networks in hardware.
To summarize, this dissertation explores two important directions in the intersection of information theory and machine learning. We explore how machine learning techniques can be useful in improving the performance of ECCs. Conversely, we show how information-theoretic concepts can be used to design robust neural networks in hardware
Proceedings of the 35th WIC Symposium on Information Theory in the Benelux and the 4th joint WIC/IEEE Symposium on Information Theory and Signal Processing in the Benelux, Eindhoven, the Netherlands May 12-13, 2014
Compressive sensing (CS) as an approach for data acquisition has recently received much attention. In CS, the signal recovery problem from the observed data requires the solution of a sparse vector from an underdetermined system of equations. The underlying sparse signal recovery problem is quite general with many applications and is the focus of this talk. The main emphasis will be on Bayesian approaches for sparse signal recovery. We will examine sparse priors such as the super-Gaussian and student-t priors and appropriate MAP estimation methods. In particular, re-weighted l2 and re-weighted l1 methods developed to solve the optimization problem will be discussed. The talk will also examine a hierarchical Bayesian framework and then study in detail an empirical Bayesian method, the Sparse Bayesian Learning (SBL) method. If time permits, we will also discuss Bayesian methods for sparse recovery problems with structure; Intra-vector correlation in the context of the block sparse model and inter-vector correlation in the context of the multiple measurement vector problem
A Unified Coded Deep Neural Network Training Strategy Based on Generalized PolyDot Codes for Matrix Multiplication
This paper has two contributions. First, we propose a novel coded matrix
multiplication technique called Generalized PolyDot codes that advances on
existing methods for coded matrix multiplication under storage and
communication constraints. This technique uses "garbage alignment," i.e.,
aligning computations in coded computing that are not a part of the desired
output. Generalized PolyDot codes bridge between Polynomial codes and MatDot
codes, trading off between recovery threshold and communication costs. Second,
we demonstrate that Generalized PolyDot can be used for training large Deep
Neural Networks (DNNs) on unreliable nodes prone to soft-errors. This requires
us to address three additional challenges: (i) prohibitively large overhead of
coding the weight matrices in each layer of the DNN at each iteration; (ii)
nonlinear operations during training, which are incompatible with linear
coding; and (iii) not assuming presence of an error-free master node, requiring
us to architect a fully decentralized implementation without any "single point
of failure." We allow all primary DNN training steps, namely, matrix
multiplication, nonlinear activation, Hadamard product, and update steps as
well as the encoding/decoding to be error-prone. We consider the case of
mini-batch size , as well as , leveraging coded matrix-vector
products, and matrix-matrix products respectively. The problem of DNN training
under soft-errors also motivates an interesting, probabilistic error model
under which a real number MDS code is shown to correct errors
with probability as compared to for the
more conventional, adversarial error model. We also demonstrate that our
proposed strategy can provide unbounded gains in error tolerance over a
competing replication strategy and a preliminary MDS-code-based strategy for
both these error models.Comment: Presented in part at the IEEE International Symposium on Information
Theory 2018 (Submission Date: Jan 12 2018); Currently under review at the
IEEE Transactions on Information Theor
Compressed sensing with approximate message passing: measurement matrix and algorithm design
Compressed sensing (CS) is an emerging technique that exploits the properties of a sparse or
compressible signal to efficiently and faithfully capture it with a sampling rate far below the
Nyquist rate. The primary goal of compressed sensing is to achieve the best signal recovery
with the least number of samples. To this end, two research directions have been receiving
increasing attention: customizing the measurement matrix to the signal of interest and optimizing
the reconstruction algorithm. In this thesis, contributions in both directions are made
in the Bayesian setting for compressed sensing. The work presented in this thesis focuses on
the approximate message passing (AMP) schemes, a new class of recovery algorithm that takes
advantage of the statistical properties of the CS problem.
First of all, a complete sample distortion (SD) framework is presented to fundamentally quantify
the reconstruction performance for a certain pair of measurement matrix and recovery
scheme. In the SD setting, the non-optimality region of the homogeneous Gaussian matrix
is identified and the novel zeroing matrix is proposed with an improved performance. With the
SD framework, the optimal sample allocation strategy for the block diagonal measurement matrix
are derived for the wavelet representation of natural images. Extensive simulations validate
the optimality of the proposed measurement matrix design.
Motivated by the zeroing matrix, we extend the seeded matrix design in the CS literature to
the novel modulated matrix structure. The major advantage of the modulated matrix over the
seeded matrix lies in the simplicity of its state evolution dynamics. Together with the AMP
based algorithm, the modulated matrix possesses a 1-D performance prediction system, with
which we can optimize the matrix configuration. We then focus on a special modulated matrix
form, designated as the two block matrix, which can also be seen as a generalization of the
zeroing matrix. The effectiveness of the two block matrix is demonstrated through both sparse
and compressible signals. The underlining reason for the improved performance is presented
through the analysis of the state evolution dynamics.
The final contribution of the thesis explores improving the reconstruction algorithm. By taking
the signal prior into account, the Bayesian optimal AMP (BAMP) algorithm is demonstrated
to dramatically improve the reconstruction quality. The key insight for its success is that it
utilizes the minimum mean square error (MMSE) estimator for the CS denoising. However, the
prerequisite of the prior information makes it often impractical. A novel SURE-AMP algorithm
is proposed to address the dilemma. The critical feature of SURE-AMP is that the Stein’s
unbiased risk estimate (SURE) based parametric least square estimator is used to replace the
MMSE estimator. Given the optimization of the SURE estimator only involves the noisy data,
it eliminates the need for the signal prior, thus can accommodate more general sparse models
Intra-Key-Frame Coding and Side Information Generation Schemes in Distributed Video Coding
In this thesis investigation has been made to propose improved schemes for intra-key-frame coding and side information (SI) generation in a distributed video
coding (DVC) framework. From the DVC developments in last few years it has
been observed that schemes put more thrust on intra-frame coding and better
quality side information (SI) generation. In fact both are interrelated as SI
generation is dependent on decoded key frame quality. Hence superior quality
key frames generated through intra-key frame coding will in turn are utilized to
generate good quality SI frames. As a result, DVC needs less number of parity
bits to reconstruct the WZ frames at the decoder. Keeping this in mind, we have
proposed two schemes for intra-key frame coding namely,
(a) Borrows Wheeler Transform based H.264/AVC (Intra) intra-frame coding
(BWT-H.264/AVC(Intra))
(b) Dictionary based H.264/AVC (Intra) intra-frame coding using orthogonal
matching pursuit (DBOMP-H.264/AVC (Intra))
BWT-H.264/AVC (Intra) scheme is a modified version of H.264/AVC (Intra)
scheme where a regularized bit stream is generated prior to compression. This
scheme results in higher compression efficiency as well as high quality decoded
key frames. DBOMP-H.264/AVC (Intra) scheme is based on an adaptive
dictionary and H.264/AVC (Intra) intra-frame coding. The traditional transform
is replaced with a dictionary trained with K-singular value decomposition (K-SVD)
algorithm. The dictionary elements are coded using orthogonal matching pursuit
(OMP).
Further, two side information generation schemes have been suggested namely,
(a) Multilayer Perceptron based side information generation (MLP - SI)
(b) Multivariable support vector regression based side information generation
(MSVR-SI)
MLP-SI scheme utilizes a multilayer perceptron (MLP) to estimate SI frames
from the decoded key frames block-by-block. The network is trained offline using
training patterns from different frames collected from standard video sequences.
MSVR-SI scheme uses an optimized multi variable support vector regression
(M-SVR) to generate SI frames from decoded key frames block-by-block. Like
MLP, the training for M-SVR is made offline with known training patterns apriori.
Both intra-key-frame coding and SI generation schemes are embedded in
the Stanford based DVC architecture and studied individually to compare
performances with their competitive schemes. Visual as well as quantitative
evaluations have been made to show the efficacy of the schemes. To exploit the
usefulness of intra-frame coding schemes in SI generation, four hybrid schemes
have been formulated by combining the aforesaid suggested schemes as follows:
(a) BWT-MLP scheme that uses BWT-H.264/AVC (Intra) intra-frame
coding scheme and MLP-SI side information generation scheme.
(b) BWT-MSVR scheme, where we utilize BWT-H.264/AVC (Intra)
for intra-frame coding followed by MSVR-SI based side information
generation.
(c) DBOMP-MLP scheme is an outcome of putting DBOMP-H.264/AVC
(Intra) intra-frame coding and MLP-SI side information generation
schemes.
(d) DBOMP-MSVR scheme deals with DBOMP-H.264/AVC (Intra)
intra-frame coding and MSVR-SI side information generation together.
The hybrid schemes are also incorporated into the Stanford based DVC
architecture and simulation has been carried out on standard video sequences.
The performance analysis with respect to overall rate distortion, number requests
per SI frame, temporal evaluation, and decoding time requirement has been made
to derive an overall conclusion
- …