15,323 research outputs found
A vector quantization approach to universal noiseless coding and quantization
A two-stage code is a block code in which each block of data is coded in two stages: the first stage codes the identity of a block code among a collection of codes, and the second stage codes the data using the identified code. The collection of codes may be noiseless codes, fixed-rate quantizers, or variable-rate quantizers. We take a vector quantization approach to two-stage coding, in which the first stage code can be regarded as a vector quantizer that “quantizes” the input data of length n to one of a fixed collection of block codes. We apply the generalized Lloyd algorithm to the first-stage quantizer, using induced measures of rate and distortion, to design locally optimal two-stage codes. On a source of medical images, two-stage variable-rate vector quantizers designed in this way outperform standard (one-stage) fixed-rate vector quantizers by over 9 dB. The tail of the operational distortion-rate function of the first-stage quantizer determines the optimal rate of convergence of the redundancy of a universal sequence of two-stage codes. We show that there exist two-stage universal noiseless codes, fixed-rate quantizers, and variable-rate quantizers whose per-letter rate and distortion redundancies converge to zero as (k/2)n -1 log n, when the universe of sources has finite dimension k. This extends the achievability part of Rissanen's theorem from universal noiseless codes to universal quantizers. Further, we show that the redundancies converge as O(n-1) when the universe of sources is countable, and as O(n-1+ϵ) when the universe of sources is infinite-dimensional, under appropriate conditions
Lossless quantum data compression and variable-length coding
In order to compress quantum messages without loss of information it is
necessary to allow the length of the encoded messages to vary. We develop a
general framework for variable-length quantum messages in close analogy to the
classical case and show that lossless compression is only possible if the
message to be compressed is known to the sender. The lossless compression of an
ensemble of messages is bounded from below by its von-Neumann entropy. We show
that it is possible to reduce the number of qbits passing through a quantum
channel even below the von-Neumann entropy by adding a classical side-channel.
We give an explicit communication protocol that realizes lossless and
instantaneous quantum data compression and apply it to a simple example. This
protocol can be used for both online quantum communication and storage of
quantum data.Comment: 16 pages, 5 figure
Variable-Length Coding of Two-Sided Asymptotically Mean Stationary Measures
We collect several observations that concern variable-length coding of
two-sided infinite sequences in a probabilistic setting. Attention is paid to
images and preimages of asymptotically mean stationary measures defined on
subsets of these sequences. We point out sufficient conditions under which the
variable-length coding and its inverse preserve asymptotic mean stationarity.
Moreover, conditions for preservation of shift-invariant -fields and
the finite-energy property are discussed and the block entropies for stationary
means of coded processes are related in some cases. Subsequently, we apply
certain of these results to construct a stationary nonergodic process with a
desired linguistic interpretation.Comment: 20 pages. A few typos corrected after the journal publicatio
On the Convergence Speed of MDL Predictions for Bernoulli Sequences
We consider the Minimum Description Length principle for online sequence
prediction. If the underlying model class is discrete, then the total expected
square loss is a particularly interesting performance measure: (a) this
quantity is bounded, implying convergence with probability one, and (b) it
additionally specifies a `rate of convergence'. Generally, for MDL only
exponential loss bounds hold, as opposed to the linear bounds for a Bayes
mixture. We show that this is even the case if the model class contains only
Bernoulli distributions. We derive a new upper bound on the prediction error
for countable Bernoulli classes. This implies a small bound (comparable to the
one for Bayes mixtures) for certain important model classes. The results apply
to many Machine Learning tasks including classification and hypothesis testing.
We provide arguments that our theorems generalize to countable classes of
i.i.d. models.Comment: 17 page
Information Spectrum Approach to the Source Channel Separation Theorem
A source-channel separation theorem for a general channel has recently been
shown by Aggrawal et. al. This theorem states that if there exist a coding
scheme that achieves a maximum distortion level d_{max} over a general channel
W, then reliable communication can be accomplished over this channel at rates
less then R(d_{max}), where R(.) is the rate distortion function of the source.
The source, however, is essentially constrained to be discrete and memoryless
(DMS). In this work we prove a stronger claim where the source is general,
satisfying only a "sphere packing optimality" feature, and the channel is
completely general. Furthermore, we show that if the channel satisfies the
strong converse property as define by Han & verdu, then the same statement can
be made with d_{avg}, the average distortion level, replacing d_{max}. Unlike
the proofs there, we use information spectrum methods to prove the statements
and the results can be quite easily extended to other situations
- …