463 research outputs found
The price of ignorance: The impact of side-information on delay for lossless source-coding
Inspired by the context of compressing encrypted sources, this paper
considers the general tradeoff between rate, end-to-end delay, and probability
of error for lossless source coding with side-information. The notion of
end-to-end delay is made precise by considering a sequential setting in which
source symbols are revealed in real time and need to be reconstructed at the
decoder within a certain fixed latency requirement. Upper bounds are derived on
the reliability functions with delay when side-information is known only to the
decoder as well as when it is also known at the encoder.
When the encoder is not ignorant of the side-information (including the
trivial case when there is no side-information), it is possible to have
substantially better tradeoffs between delay and probability of error at all
rates. This shows that there is a fundamental price of ignorance in terms of
end-to-end delay when the encoder is not aware of the side information. This
effect is not visible if only fixed-block-length codes are considered. In this
way, side-information in source-coding plays a role analogous to that of
feedback in channel coding.
While the theorems in this paper are asymptotic in terms of long delays and
low probabilities of error, an example is used to show that the qualitative
effects described here are significant even at short and moderate delays.Comment: 25 pages, 17 figures. Submitted to the IEEE Transactions on
Information Theor
Optimal Lempel-Ziv based lossy compression for memoryless data: how to make the right mistakes
Compression refers to encoding data using bits, so that the representation
uses as few bits as possible. Compression could be lossless: i.e. encoded data
can be recovered exactly from its representation) or lossy where the data is
compressed more than the lossless case, but can still be recovered to within
prespecified distortion metric. In this paper, we prove the optimality of
Codelet Parsing, a quasi-linear time algorithm for lossy compression of
sequences of bits that are independently and identically distributed (\iid) and
Hamming distortion. Codelet Parsing extends the lossless Lempel Ziv algorithm
to the lossy case---a task that has been a focus of the source coding
literature for better part of two decades now. Given \iid sequences \x, the
expected length of the shortest lossy representation such that \x can be
reconstructed to within distortion \dist is given by the rate distortion
function, \rd. We prove the optimality of the Codelet Parsing algorithm for
lossy compression of memoryless bit sequences. It splits the input sequence
naturally into phrases, representing each phrase by a codelet, a potentially
distorted phrase of the same length. The codelets in the lossy representation
of a length- string {\x} have length roughly (\log n)/\rd, and like the
lossless Lempel Ziv algorithm, Codelet Parsing constructs codebooks logarithmic
in the sequence length.Comment: This file is not the final version, and will be updated for the next
few days. (Edited 10/17
One-pass adaptive universal vector quantization
The authors introduce a one-pass adaptive universal quantization technique for real, bounded alphabet, stationary sources. The algorithm is set on line without any prior knowledge of the statistics of the sources which it might encounter and asymptotically achieves ideal performance on all sources that it sees. The system consists of an encoder and a decoder. At increasing intervals, the encoder refines its codebook using knowledge about incoming data symbols. This codebook is then described to the decoder in the form of updates on the previous codebook. The accuracy to which the codebook is described increases as the number of symbols seen, and thus the accuracy to which the codebook is known, grows
The Shannon Cipher System with a Guessing Wiretapper: General Sources
The Shannon cipher system is studied in the context of general sources using
a notion of computational secrecy introduced by Merhav & Arikan. Bounds are
derived on limiting exponents of guessing moments for general sources. The
bounds are shown to be tight for iid, Markov, and unifilar sources, thus
recovering some known results. A close relationship between error exponents and
correct decoding exponents for fixed rate source compression on the one hand
and exponents for guessing moments on the other hand is established.Comment: 24 pages, Submitted to IEEE Transactions on Information Theor
Sequential Source Coding for Stochastic Systems Subject to Finite Rate Constraints
In this paper, we revisit the sequential source coding framework to analyze
fundamental performance limitations of discrete-time stochastic control systems
subject to feedback data-rate constraints in finite-time horizon. The basis of
our results is a new characterization of the lower bound on the minimum
total-rate achieved by sequential codes subject to a total (across time)
distortion constraint and a computational algorithm that allocates optimally
the rate-distortion for any fixed finite-time horizon. This characterization
facilitates the derivation of analytical, non-asymptotic, and
finite-dimensional lower and upper bounds in two control-related scenarios. (a)
A parallel time-varying Gauss-Markov process with identically distributed
spatial components that is quantized and transmitted through a noiseless
channel to a minimum mean-squared error (MMSE) decoder. (b) A time-varying
quantized LQG closed-loop control system, with identically distributed spatial
components and with a random data-rate allocation. Our non-asymptotic lower
bound on the quantized LQG control problem, reveals the absolute minimum
data-rates for (mean square) stability of our time-varying plant for any fixed
finite time horizon. We supplement our framework with illustrative simulation
experiments.Comment: 40 pages, 6 figure
Interactive Communication for Data Exchange
Two parties observing correlated data seek to exchange their data using
interactive communication. How many bits must they communicate? We propose a
new interactive protocol for data exchange which increases the communication
size in steps until the task is done. Next, we derive a lower bound on the
minimum number of bits that is based on relating the data exchange problem to
the secret key agreement problem. Our single-shot analysis applies to all
discrete random variables and yields upper and lower bound of a similar form.
In fact, the bounds are asymptotically tight and lead to a characterization of
the optimal rate of communication needed for data exchange for a general
sequence such as mixture of IID random variables as well as the optimal
second-order asymptotic term in the length of communication needed for data
exchange for the IID random variables, when the probability of error is fixed.
This gives a precise characterization of the asymptotic reduction in the length
of optimal communication due to interaction; in particular, two-sided
Slepian-Wolf compression is strictly suboptimal.Comment: 13 pages (two column), 4 figures, a longer version of ISIT 201
Robust Bayesian compressed sensing over finite fields: asymptotic performance analysis
This paper addresses the topic of robust Bayesian compressed sensing over
finite fields. For stationary and ergodic sources, it provides asymptotic (with
the size of the vector to estimate) necessary and sufficient conditions on the
number of required measurements to achieve vanishing reconstruction error, in
presence of sensing and communication noise. In all considered cases, the
necessary and sufficient conditions asymptotically coincide. Conditions on the
sparsity of the sensing matrix are established in presence of communication
noise. Several previously published results are generalized and extended.Comment: 42 pages, 4 figure
Distributed Joint Source-Channel Coding on a Multiple Access Channel with Side Information
We consider the problem of transmission of several distributed sources over a
multiple access channel (MAC) with side information at the sources and the
decoder. Source-channel separation does not hold for this channel. Sufficient
conditions are provided for transmission of sources with a given distortion.
The source and/or the channel could have continuous alphabets (thus Gaussian
sources and Gaussian MACs are special cases). Various previous results are
obtained as special cases. We also provide several good joint source-channel
coding schemes for a discrete/continuous source and discrete/continuous
alphabet channel. Channels with feedback and fading are also considered.
Keywords: Multiple access channel, side information, lossy joint
source-channel coding, channels with feedback, fading channels.Comment: 49 pages, Technical Report, DRDO-IISc programme on Advanced Research
in Mathematical Engineering, Dept of ECE, Indian Institute of Science,
Bangalore, Indi
-Resolvability
The conventional channel resolvability refers to the minimum rate needed for
an input process to approximate the channel output distribution in total
variation distance. In this paper we study -resolvability, in which
total variation is replaced by the more general distance. A
general one-shot achievability bound for the precision of such an approximation
is developed. Let be a random transformation, be an integer,
and . We show that in the asymptotic setting where
, a (nonnegative) randomness rate above is sufficient to approximate the output distribution
using the channel ,
where , and is also necessary in the
case of finite and . In particular, a randomness
rate of is always sufficient. We
also study the convergence of the approximation error under the high
probability criteria in the case of random codebooks. Moreover, by developing
simple bounds relating and other distance measures, we are able to
determine the exact linear growth rate of the approximation errors measured in
relative entropy and smooth R\'{e}nyi divergences for a fixed-input randomness
rate. The new resolvability result is then used to derive 1) a one-shot upper
bound on the probability of excess distortion in lossy compression, which is
exponentially tight in the i.i.d.~setting, 2) a one-shot version of the mutual
covering lemma, and 3) a lower bound on the size of the eavesdropper list to
include the actual message and a lower bound on the eavesdropper false-alarm
probability in the wiretap channel problem, which is (asymptotically)
ensemble-tight.Comment: 30 pages, 5 figures, presented in part at 2015 IEEE International
Symposium on Information Theory (ISIT
The Rate-Distortion Risk in Estimation from Compressed Data
Consider the problem of estimating a latent signal from a lossy compressed
version of the data. Assume that the data is compressed to a prescribed bitrate
via a procedure that is agnostic to the model describing the relation between
the latent signal and the data. In reconstruction, the latent signal is
estimated to minimize a prescribed risk function. For the above setting and a
given distortion measure between the data and its compressed version, we define
the rate-distortion (RD) risk of an estimator as its risk under the
distribution achieving Shannon's RD function at the prescribed bitrate. We
derive conditions on the compression code under which the true risk in
estimating from the compressed data is asymptotically equivalent to the RD
risk. The main theoretical tools to obtain these conditions are
transportation-cost inequalities in conjunction with properties of compression
codes achieving Shannon's RD function. We show that these conditions typically
hold in a memoryless discrete setting or when the RD achieving distribution is
multivariate normal. Whenever the aforementioned asymptotic equivalence holds,
this RD risk provides an achievable estimation performance in situations when
the data is compressed, communicated, or stored using a procedure that is
agnostic to the latent signal or the ultimate inference task. Furthermore, in
these cases, our results imply a general procedure for designing an estimator
from a dataset undergoing lossy compression without specifying the actual
compression technique. Namely, by designing it based on the RD achieving
distribution.Comment: Under review for the IEEE Transactions on Information Theor
- β¦