7 research outputs found
An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory
An identity between two versions of the Chernoff bound on the probability a
certain large deviations event, is established. This identity has an
interpretation in statistical physics, namely, an isothermal equilibrium of a
composite system that consists of multiple subsystems of particles. Several
information--theoretic application examples, where the analysis of this large
deviations probability naturally arises, are then described from the viewpoint
of this statistical mechanical interpretation. This results in several
relationships between information theory and statistical physics, which we
hope, the reader will find insightful.Comment: 29 pages, 1 figure. Submitted to IEEE Trans. on Information Theor
Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions
We consider the problem of signal estimation (denoising) from a statistical
mechanical perspective, using a relationship between the minimum mean square
error (MMSE), of estimating a signal, and the mutual information between this
signal and its noisy version. The paper consists of essentially two parts. In
the first, we derive several statistical-mechanical relationships between a few
important quantities in this problem area, such as the MMSE, the differential
entropy, the Fisher information, the free energy, and a generalized notion of
temperature. We also draw analogies and differences between certain relations
pertaining to the estimation problem and the parallel relations in
thermodynamics and statistical physics. In the second part of the paper, we
provide several application examples, where we demonstrate how certain analysis
tools that are customary in statistical physics, prove useful in the analysis
of the MMSE. In most of these examples, the corresponding
statistical-mechanical systems turn out to consist of strong interactions that
cause phase transitions, which in turn are reflected as irregularities and
discontinuities (similar to threshold effects) in the behavior of the MMSE.Comment: Submitted to the IEEE Transactions on Information Theor