3,935 research outputs found
Circumventing the Curse of Dimensionality in Prediction: Causal Rate-Distortion for Infinite-Order Markov Processes
Predictive rate-distortion analysis suffers from the curse of dimensionality:
clustering arbitrarily long pasts to retain information about arbitrarily long
futures requires resources that typically grow exponentially with length. The
challenge is compounded for infinite-order Markov processes, since conditioning
on finite sequences cannot capture all of their past dependencies. Spectral
arguments show that algorithms which cluster finite-length sequences fail
dramatically when the underlying process has long-range temporal correlations
and can fail even for processes generated by finite-memory hidden Markov
models. We circumvent the curse of dimensionality in rate-distortion analysis
of infinite-order processes by casting predictive rate-distortion objective
functions in terms of the forward- and reverse-time causal states of
computational mechanics. Examples demonstrate that the resulting causal
rate-distortion theory substantially improves current predictive
rate-distortion analyses.Comment: 25 pages, 14 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/cn.ht
Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources
We deal with zero-delay source coding of a vector-valued Gauss-Markov source
subject to a mean-squared error (MSE) fidelity criterion characterized by the
operational zero-delay vector-valued Gaussian rate distortion function (RDF).
We address this problem by considering the nonanticipative RDF (NRDF) which is
a lower bound to the causal optimal performance theoretically attainable (OPTA)
function and operational zero-delay RDF. We recall the realization that
corresponds to the optimal "test-channel" of the Gaussian NRDF, when
considering a vector Gauss-Markov source subject to a MSE distortion in the
finite time horizon. Then, we introduce sufficient conditions to show existence
of solution for this problem in the infinite time horizon. For the asymptotic
regime, we use the asymptotic characterization of the Gaussian NRDF to provide
a new equivalent realization scheme with feedback which is characterized by a
resource allocation (reverse-waterfilling) problem across the dimension of the
vector source. We leverage the new realization to derive a predictive coding
scheme via lattice quantization with subtractive dither and joint memoryless
entropy coding. This coding scheme offers an upper bound to the operational
zero-delay vector-valued Gaussian RDF. When we use scalar quantization, then
for "r" active dimensions of the vector Gauss-Markov source the gap between the
obtained lower and theoretical upper bounds is less than or equal to 0.254r + 1
bits/vector. We further show that it is possible when we use vector
quantization, and assume infinite dimensional Gauss-Markov sources to make the
previous gap to be negligible, i.e., Gaussian NRDF approximates the operational
zero-delay Gaussian RDF. We also extend our results to vector-valued Gaussian
sources of any finite memory under mild conditions. Our theoretical framework
is demonstrated with illustrative numerical experiments.Comment: 32 pages, 9 figures, published in IEEE Journal of Selected Topics in
Signal Processin
The Origins of Computational Mechanics: A Brief Intellectual History and Several Clarifications
The principle goal of computational mechanics is to define pattern and
structure so that the organization of complex systems can be detected and
quantified. Computational mechanics developed from efforts in the 1970s and
early 1980s to identify strange attractors as the mechanism driving weak fluid
turbulence via the method of reconstructing attractor geometry from measurement
time series and in the mid-1980s to estimate equations of motion directly from
complex time series. In providing a mathematical and operational definition of
structure it addressed weaknesses of these early approaches to discovering
patterns in natural systems.
Since then, computational mechanics has led to a range of results from
theoretical physics and nonlinear mathematics to diverse applications---from
closed-form analysis of Markov and non-Markov stochastic processes that are
ergodic or nonergodic and their measures of information and intrinsic
computation to complex materials and deterministic chaos and intelligence in
Maxwellian demons to quantum compression of classical processes and the
evolution of computation and language.
This brief review clarifies several misunderstandings and addresses concerns
recently raised regarding early works in the field (1980s). We show that
misguided evaluations of the contributions of computational mechanics are
groundless and stem from a lack of familiarity with its basic goals and from a
failure to consider its historical context. For all practical purposes, its
modern methods and results largely supersede the early works. This not only
renders recent criticism moot and shows the solid ground on which computational
mechanics stands but, most importantly, shows the significant progress achieved
over three decades and points to the many intriguing and outstanding challenges
in understanding the computational nature of complex dynamic systems.Comment: 11 pages, 123 citations;
http://csc.ucdavis.edu/~cmg/compmech/pubs/cmr.ht
Nearly maximally predictive features and their dimensions
Scientific explanation often requires inferring maximally predictive features from a given data set. Unfortunately, the collection of minimal maximally predictive features for most stochastic processes is uncountably infinite. In such cases, one compromises and instead seeks nearly maximally predictive features. Here, we derive upper bounds on the rates at which the number and the coding cost of nearly maximally predictive features scale with desired predictive power. The rates are determined by the fractal dimensions of a process' mixed-state distribution. These results, in turn, show how widely used finite-order Markov models can fail as predictors and that mixed-state predictive features can offer a substantial improvement.United States. Army Research Office (W911NF-13-1-0390)United States. Army Research Office (W911NF-12-1- 0288
On the Information Rates of the Plenoptic Function
The {\it plenoptic function} (Adelson and Bergen, 91) describes the visual
information available to an observer at any point in space and time. Samples of
the plenoptic function (POF) are seen in video and in general visual content,
and represent large amounts of information. In this paper we propose a
stochastic model to study the compression limits of the plenoptic function. In
the proposed framework, we isolate the two fundamental sources of information
in the POF: the one representing the camera motion and the other representing
the information complexity of the "reality" being acquired and transmitted. The
sources of information are combined, generating a stochastic process that we
study in detail. We first propose a model for ensembles of realities that do
not change over time. The proposed model is simple in that it enables us to
derive precise coding bounds in the information-theoretic sense that are sharp
in a number of cases of practical interest. For this simple case of static
realities and camera motion, our results indicate that coding practice is in
accordance with optimal coding from an information-theoretic standpoint. The
model is further extended to account for visual realities that change over
time. We derive bounds on the lossless and lossy information rates for this
dynamic reality model, stating conditions under which the bounds are tight.
Examples with synthetic sources suggest that in the presence of scene dynamics,
simple hybrid coding using motion/displacement estimation with DPCM performs
considerably suboptimally relative to the true rate-distortion bound.Comment: submitted to IEEE Transactions in Information Theor
Informational and Causal Architecture of Discrete-Time Renewal Processes
Renewal processes are broadly used to model stochastic behavior consisting of
isolated events separated by periods of quiescence, whose durations are
specified by a given probability law. Here, we identify the minimal sufficient
statistic for their prediction (the set of causal states), calculate the
historical memory capacity required to store those states (statistical
complexity), delineate what information is predictable (excess entropy), and
decompose the entropy of a single measurement into that shared with the past,
future, or both. The causal state equivalence relation defines a new subclass
of renewal processes with a finite number of causal states despite having an
unbounded interevent count distribution. We use these formulae to analyze the
output of the parametrized Simple Nonunifilar Source, generated by a simple
two-state hidden Markov model, but with an infinite-state epsilon-machine
presentation. All in all, the results lay the groundwork for analyzing
processes with infinite statistical complexity and infinite excess entropy.Comment: 18 pages, 9 figures, 1 table;
http://csc.ucdavis.edu/~cmg/compmech/pubs/dtrp.ht
Recursively indexed differential pulse code modulation
The performance of a differential pulse code modulation (DPCM) system with a recursively indexed quantizer (RIQ) under various conditions, with first order Gauss-Markov and Laplace-Markov sources as inputs, is studied. When the predictor is matched to the input, the proposed system performs at or close to the optimum entropy constrained DPCM system. If one is willing to accept a 5 percent increase in the rate, the system is very forgiving of predictor mismatch
- …