24,134 research outputs found
Generalization Bounds via Information Density and Conditional Information Density
We present a general approach, based on an exponential inequality, to derive
bounds on the generalization error of randomized learning algorithms. Using
this approach, we provide bounds on the average generalization error as well as
bounds on its tail probability, for both the PAC-Bayesian and single-draw
scenarios. Specifically, for the case of subgaussian loss functions, we obtain
novel bounds that depend on the information density between the training data
and the output hypothesis. When suitably weakened, these bounds recover many of
the information-theoretic available bounds in the literature. We also extend
the proposed exponential-inequality approach to the setting recently introduced
by Steinke and Zakynthinou (2020), where the learning algorithm depends on a
randomly selected subset of the available training data. For this setup, we
present bounds for bounded loss functions in terms of the conditional
information density between the output hypothesis and the random variable
determining the subset choice, given all training data. Through our approach,
we recover the average generalization bound presented by Steinke and
Zakynthinou (2020) and extend it to the PAC-Bayesian and single-draw scenarios.
For the single-draw scenario, we also obtain novel bounds in terms of the
conditional -mutual information and the conditional maximal leakage.Comment: Published in Journal on Selected Areas in Information Theory (JSAIT).
Important note: the proof of the data-dependent bounds provided in the paper
contains an error, which is rectified in the following document:
https://gdurisi.github.io/files/2021/jsait-correction.pd
Quantum Information Theory of Entanglement and Measurement
We present a quantum information theory that allows for a consistent
description of entanglement. It parallels classical (Shannon) information
theory but is based entirely on density matrices (rather than probability
distributions) for the description of quantum ensembles. We find that quantum
conditional entropies can be negative for entangled systems, which leads to a
violation of well-known bounds in Shannon information theory. Such a unified
information-theoretic description of classical correlation and quantum
entanglement clarifies the link between them: the latter can be viewed as
``super-correlation'' which can induce classical correlation when considering a
tripartite or larger system. Furthermore, negative entropy and the associated
clarification of entanglement paves the way to a natural information-theoretic
description of the measurement process. This model, while unitary and causal,
implies the well-known probabilistic results of conventional quantum mechanics.
It also results in a simple interpretation of the Kholevo theorem limiting the
accessible information in a quantum measurement.Comment: 26 pages with 6 figures. Expanded version of PhysComp'96 contributio
A Quantum Multiparty Packing Lemma and the Relay Channel
Optimally encoding classical information in a quantum system is one of the
oldest and most fundamental challenges of quantum information theory. Holevo's
bound places a hard upper limit on such encodings, while the
Holevo-Schumacher-Westmoreland (HSW) theorem addresses the question of how many
classical messages can be "packed" into a given quantum system. In this
article, we use Sen's recent quantum joint typicality results to prove a
one-shot multiparty quantum packing lemma generalizing the HSW theorem. The
lemma is designed to be easily applicable in many network communication
scenarios. As an illustration, we use it to straightforwardly obtain quantum
generalizations of well-known classical coding schemes for the relay channel:
multihop, coherent multihop, decode-forward, and partial decode-forward. We
provide both finite blocklength and asymptotic results, the latter matching
existing classical formulas. Given the key role of the classical packing lemma
in network information theory, our packing lemma should help open the field to
direct quantum generalization.Comment: 20 page
Coding for Parallel Channels: Gallager Bounds for Binary Linear Codes with Applications to Repeat-Accumulate Codes and Variations
This paper is focused on the performance analysis of binary linear block
codes (or ensembles) whose transmission takes place over independent and
memoryless parallel channels. New upper bounds on the maximum-likelihood (ML)
decoding error probability are derived. These bounds are applied to various
ensembles of turbo-like codes, focusing especially on repeat-accumulate codes
and their recent variations which possess low encoding and decoding complexity
and exhibit remarkable performance under iterative decoding. The framework of
the second version of the Duman and Salehi (DS2) bounds is generalized to the
case of parallel channels, along with the derivation of their optimized tilting
measures. The connection between the generalized DS2 and the 1961 Gallager
bounds, addressed by Divsalar and by Sason and Shamai for a single channel, is
explored in the case of an arbitrary number of independent parallel channels.
The generalization of the DS2 bound for parallel channels enables to re-derive
specific bounds which were originally derived by Liu et al. as special cases of
the Gallager bound. In the asymptotic case where we let the block length tend
to infinity, the new bounds are used to obtain improved inner bounds on the
attainable channel regions under ML decoding. The tightness of the new bounds
for independent parallel channels is exemplified for structured ensembles of
turbo-like codes. The improved bounds with their optimized tilting measures
show, irrespectively of the block length of the codes, an improvement over the
union bound and other previously reported bounds for independent parallel
channels; this improvement is especially pronounced for moderate to large block
lengths.Comment: Submitted to IEEE Trans. on Information Theory, June 2006 (57 pages,
9 figures
Recoverability in quantum information theory
The fact that the quantum relative entropy is non-increasing with respect to
quantum physical evolutions lies at the core of many optimality theorems in
quantum information theory and has applications in other areas of physics. In
this work, we establish improvements of this entropy inequality in the form of
physically meaningful remainder terms. One of the main results can be
summarized informally as follows: if the decrease in quantum relative entropy
between two quantum states after a quantum physical evolution is relatively
small, then it is possible to perform a recovery operation, such that one can
perfectly recover one state while approximately recovering the other. This can
be interpreted as quantifying how well one can reverse a quantum physical
evolution. Our proof method is elementary, relying on the method of complex
interpolation, basic linear algebra, and the recently introduced Renyi
generalization of a relative entropy difference. The theorem has a number of
applications in quantum information theory, which have to do with providing
physically meaningful improvements to many known entropy inequalities.Comment: v5: 26 pages, generalized lower bounds to apply when supp(rho) is
contained in supp(sigma
- …