3,966 research outputs found
Information theoretic bounds for Compressed Sensing
In this paper we derive information theoretic performance bounds to sensing
and reconstruction of sparse phenomena from noisy projections. We consider two
settings: output noise models where the noise enters after the projection and
input noise models where the noise enters before the projection. We consider
two types of distortion for reconstruction: support errors and mean-squared
errors. Our goal is to relate the number of measurements, , and \snr, to
signal sparsity, , distortion level, , and signal dimension, . We
consider support errors in a worst-case setting. We employ different variations
of Fano's inequality to derive necessary conditions on the number of
measurements and \snr required for exact reconstruction. To derive sufficient
conditions we develop new insights on max-likelihood analysis based on a novel
superposition property. In particular this property implies that small support
errors are the dominant error events. Consequently, our ML analysis does not
suffer the conservatism of the union bound and leads to a tighter analysis of
max-likelihood. These results provide order-wise tight bounds. For output noise
models we show that asymptotically an \snr of together with
measurements is necessary and sufficient for exact
support recovery. Furthermore, if a small fraction of support errors can be
tolerated, a constant \snr turns out to be sufficient in the linear sparsity
regime. In contrast for input noise models we show that support recovery fails
if the number of measurements scales as implying poor
compression performance for such cases. We also consider Bayesian set-up and
characterize tradeoffs between mean-squared distortion and the number of
measurements using rate-distortion theory.Comment: 30 pages, 2 figures, submitted to IEEE Trans. on I
Information Theoretic Limits for Standard and One-Bit Compressed Sensing with Graph-Structured Sparsity
In this paper, we analyze the information theoretic lower bound on the
necessary number of samples needed for recovering a sparse signal under
different compressed sensing settings. We focus on the weighted graph model, a
model-based framework proposed by Hegde et al. (2015), for standard compressed
sensing as well as for one-bit compressed sensing. We study both the noisy and
noiseless regimes. Our analysis is general in the sense that it applies to any
algorithm used to recover the signal. We carefully construct restricted
ensembles for different settings and then apply Fano's inequality to establish
the lower bound on the necessary number of samples. Furthermore, we show that
our bound is tight for one-bit compressed sensing, while for standard
compressed sensing, our bound is tight up to a logarithmic factor of the number
of non-zero entries in the signal
The Sampling Rate-Distortion Tradeoff for Sparsity Pattern Recovery in Compressed Sensing
Recovery of the sparsity pattern (or support) of an unknown sparse vector
from a limited number of noisy linear measurements is an important problem in
compressed sensing. In the high-dimensional setting, it is known that recovery
with a vanishing fraction of errors is impossible if the measurement rate and
the per-sample signal-to-noise ratio (SNR) are finite constants, independent of
the vector length. In this paper, it is shown that recovery with an arbitrarily
small but constant fraction of errors is, however, possible, and that in some
cases computationally simple estimators are near-optimal. Bounds on the
measurement rate needed to attain a desired fraction of errors are given in
terms of the SNR and various key parameters of the unknown vector for several
different recovery algorithms. The tightness of the bounds, in a scaling sense,
as a function of the SNR and the fraction of errors, is established by
comparison with existing information-theoretic necessary bounds. Near
optimality is shown for a wide variety of practically motivated signal models
A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to do this involves the use of Fano's inequality. However, recent work in an information-theoretic setting has shown that an argument based on binary hypothesis testing gives tighter converse results (error lower bounds) than Fano for channel coding problems. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case
"Compressed" Compressed Sensing
The field of compressed sensing has shown that a sparse but otherwise
arbitrary vector can be recovered exactly from a small number of randomly
constructed linear projections (or samples). The question addressed in this
paper is whether an even smaller number of samples is sufficient when there
exists prior knowledge about the distribution of the unknown vector, or when
only partial recovery is needed. An information-theoretic lower bound with
connections to free probability theory and an upper bound corresponding to a
computationally simple thresholding estimator are derived. It is shown that in
certain cases (e.g. discrete valued vectors or large distortions) the number of
samples can be decreased. Interestingly though, it is also shown that in many
cases no reduction is possible
Approximate Sparsity Pattern Recovery: Information-Theoretic Lower Bounds
Recovery of the sparsity pattern (or support) of an unknown sparse vector
from a small number of noisy linear measurements is an important problem in
compressed sensing. In this paper, the high-dimensional setting is considered.
It is shown that if the measurement rate and per-sample signal-to-noise ratio
(SNR) are finite constants independent of the length of the vector, then the
optimal sparsity pattern estimate will have a constant fraction of errors.
Lower bounds on the measurement rate needed to attain a desired fraction of
errors are given in terms of the SNR and various key parameters of the unknown
vector. The tightness of the bounds in a scaling sense, as a function of the
SNR and the fraction of errors, is established by comparison with existing
achievable bounds. Near optimality is shown for a wide variety of practically
motivated signal models
- …