5,278 research outputs found
Novel Lower Bounds on the Entropy Rate of Binary Hidden Markov Processes
Recently, Samorodnitsky proved a strengthened version of Mrs. Gerber's Lemma,
where the output entropy of a binary symmetric channel is bounded in terms of
the average entropy of the input projected on a random subset of coordinates.
Here, this result is applied for deriving novel lower bounds on the entropy
rate of binary hidden Markov processes. For symmetric underlying Markov
processes, our bound improves upon the best known bound in the very noisy
regime. The nonsymmetric case is also considered, and explicit bounds are
derived for Markov processes that satisfy the -RLL constraint
Correlation-powered Information Engines and the Thermodynamics of Self-Correction
Information engines can use structured environments as a resource to generate
work by randomizing ordered inputs and leveraging the increased Shannon entropy
to transfer energy from a thermal reservoir to a work reservoir. We give a
broadly applicable expression for the work production of an information engine,
generally modeled as a memoryful channel that communicates inputs to outputs as
it interacts with an evolving environment. The expression establishes that an
information engine must have more than one memory state in order to leverage
input environment correlations. To emphasize this functioning, we designed an
information engine powered solely by temporal correlations and not by
statistical biases, as employed by previous engines. Key to this is the
engine's ability to synchronize---the engine automatically returns to a desired
dynamical phase when thrown into an unwanted, dissipative phase by corruptions
in the input---that is, by unanticipated environmental fluctuations. This
self-correcting mechanism is robust up to a critical level of corruption,
beyond which the system fails to act as an engine. We give explicit analytical
expressions for both work and critical corruption level and summarize engine
performance via a thermodynamic-function phase diagram over engine control
parameters. The results reveal a new thermodynamic mechanism based on
nonergodicity that underlies error correction as it operates to support
resilient engineered and biological systems.Comment: 22 pages, 13 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/tos.ht
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
A central problem in machine learning involves modeling complex data-sets
using highly flexible families of probability distributions in which learning,
sampling, inference, and evaluation are still analytically or computationally
tractable. Here, we develop an approach that simultaneously achieves both
flexibility and tractability. The essential idea, inspired by non-equilibrium
statistical physics, is to systematically and slowly destroy structure in a
data distribution through an iterative forward diffusion process. We then learn
a reverse diffusion process that restores structure in data, yielding a highly
flexible and tractable generative model of the data. This approach allows us to
rapidly learn, sample from, and evaluate probabilities in deep generative
models with thousands of layers or time steps, as well as to compute
conditional and posterior probabilities under the learned model. We
additionally release an open source reference implementation of the algorithm
Quantifying Self-Organization with Optimal Wavelets
The optimal wavelet basis is used to develop quantitative, experimentally
applicable criteria for self-organization. The choice of the optimal wavelet is
based on the model of self-organization in the wavelet tree. The framework of
the model is founded on the wavelet-domain hidden Markov model and the optimal
wavelet basis criterion for self-organization which assumes inherent increase
in statistical complexity, the information content necessary for maximally
accurate prediction of the system's dynamics. At the same time the method,
presented here for the one-dimensional data of any type, performs superior
denoising and may be easily generalized to higher dimensions.Comment: 12 pages, 3 figure
Source Coding When the Side Information May Be Delayed
For memoryless sources, delayed side information at the decoder does not
improve the rate-distortion function. However, this is not the case for more
general sources with memory, as demonstrated by a number of works focusing on
the special case of (delayed) feedforward. In this paper, a setting is studied
in which the encoder is potentially uncertain about the delay with which
measurements of the side information are acquired at the decoder. Assuming a
hidden Markov model for the sources, at first, a single-letter characterization
is given for the set-up where the side information delay is arbitrary and known
at the encoder, and the reconstruction at the destination is required to be
(near) lossless. Then, with delay equal to zero or one source symbol, a
single-letter characterization is given of the rate-distortion region for the
case where side information may be delayed or not, unbeknownst to the encoder.
The characterization is further extended to allow for additional information to
be sent when the side information is not delayed. Finally, examples for binary
and Gaussian sources are provided.Comment: revised July 201
Occam's Quantum Strop: Synchronizing and Compressing Classical Cryptic Processes via a Quantum Channel
A stochastic process's statistical complexity stands out as a fundamental
property: the minimum information required to synchronize one process generator
to another. How much information is required, though, when synchronizing over a
quantum channel? Recent work demonstrated that representing causal similarity
as quantum state-indistinguishability provides a quantum advantage. We
generalize this to synchronization and offer a sequence of constructions that
exploit extended causal structures, finding substantial increase of the quantum
advantage. We demonstrate that maximum compression is determined by the
process's cryptic order---a classical, topological property closely allied to
Markov order, itself a measure of historical dependence. We introduce an
efficient algorithm that computes the quantum advantage and close noting that
the advantage comes at a cost---one trades off prediction for generation
complexity.Comment: 10 pages, 6 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/oqs.ht
- …