14 research outputs found
Nearly maximally predictive features and their dimensions
Scientific explanation often requires inferring maximally predictive features from a given data set. Unfortunately, the collection of minimal maximally predictive features for most stochastic processes is uncountably infinite. In such cases, one compromises and instead seeks nearly maximally predictive features. Here, we derive upper bounds on the rates at which the number and the coding cost of nearly maximally predictive features scale with desired predictive power. The rates are determined by the fractal dimensions of a process' mixed-state distribution. These results, in turn, show how widely used finite-order Markov models can fail as predictors and that mixed-state predictive features can offer a substantial improvement.United States. Army Research Office (W911NF-13-1-0390)United States. Army Research Office (W911NF-12-1- 0288
The Origins of Computational Mechanics: A Brief Intellectual History and Several Clarifications
The principle goal of computational mechanics is to define pattern and
structure so that the organization of complex systems can be detected and
quantified. Computational mechanics developed from efforts in the 1970s and
early 1980s to identify strange attractors as the mechanism driving weak fluid
turbulence via the method of reconstructing attractor geometry from measurement
time series and in the mid-1980s to estimate equations of motion directly from
complex time series. In providing a mathematical and operational definition of
structure it addressed weaknesses of these early approaches to discovering
patterns in natural systems.
Since then, computational mechanics has led to a range of results from
theoretical physics and nonlinear mathematics to diverse applications---from
closed-form analysis of Markov and non-Markov stochastic processes that are
ergodic or nonergodic and their measures of information and intrinsic
computation to complex materials and deterministic chaos and intelligence in
Maxwellian demons to quantum compression of classical processes and the
evolution of computation and language.
This brief review clarifies several misunderstandings and addresses concerns
recently raised regarding early works in the field (1980s). We show that
misguided evaluations of the contributions of computational mechanics are
groundless and stem from a lack of familiarity with its basic goals and from a
failure to consider its historical context. For all practical purposes, its
modern methods and results largely supersede the early works. This not only
renders recent criticism moot and shows the solid ground on which computational
mechanics stands but, most importantly, shows the significant progress achieved
over three decades and points to the many intriguing and outstanding challenges
in understanding the computational nature of complex dynamic systems.Comment: 11 pages, 123 citations;
http://csc.ucdavis.edu/~cmg/compmech/pubs/cmr.ht
Spectral Simplicity of Apparent Complexity, Part I: The Nondiagonalizable Metadynamics of Prediction
Virtually all questions that one can ask about the behavioral and structural
complexity of a stochastic process reduce to a linear algebraic framing of a
time evolution governed by an appropriate hidden-Markov process generator. Each
type of question---correlation, predictability, predictive cost, observer
synchronization, and the like---induces a distinct generator class. Answers are
then functions of the class-appropriate transition dynamic. Unfortunately,
these dynamics are generically nonnormal, nondiagonalizable, singular, and so
on. Tractably analyzing these dynamics relies on adapting the recently
introduced meromorphic functional calculus, which specifies the spectral
decomposition of functions of nondiagonalizable linear operators, even when the
function poles and zeros coincide with the operator's spectrum. Along the way,
we establish special properties of the projection operators that demonstrate
how they capture the organization of subprocesses within a complex system.
Circumventing the spurious infinities of alternative calculi, this leads in the
sequel, Part II, to the first closed-form expressions for complexity measures,
couched either in terms of the Drazin inverse (negative-one power of a singular
operator) or the eigenvalues and projection operators of the appropriate
transition dynamic.Comment: 24 pages, 3 figures, 4 tables; current version always at
http://csc.ucdavis.edu/~cmg/compmech/pubs/sdscpt1.ht
Local Causal States and Discrete Coherent Structures
Coherent structures form spontaneously in nonlinear spatiotemporal systems
and are found at all spatial scales in natural phenomena from laboratory
hydrodynamic flows and chemical reactions to ocean, atmosphere, and planetary
climate dynamics. Phenomenologically, they appear as key components that
organize the macroscopic behaviors in such systems. Despite a century of
effort, they have eluded rigorous analysis and empirical prediction, with
progress being made only recently. As a step in this, we present a formal
theory of coherent structures in fully-discrete dynamical field theories. It
builds on the notion of structure introduced by computational mechanics,
generalizing it to a local spatiotemporal setting. The analysis' main tool
employs the \localstates, which are used to uncover a system's hidden
spatiotemporal symmetries and which identify coherent structures as
spatially-localized deviations from those symmetries. The approach is
behavior-driven in the sense that it does not rely on directly analyzing
spatiotemporal equations of motion, rather it considers only the spatiotemporal
fields a system generates. As such, it offers an unsupervised approach to
discover and describe coherent structures. We illustrate the approach by
analyzing coherent structures generated by elementary cellular automata,
comparing the results with an earlier, dynamic-invariant-set approach that
decomposes fields into domains, particles, and particle interactions.Comment: 27 pages, 10 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/dcs.ht
Spectral Simplicity of Apparent Complexity, Part II: Exact Complexities and Complexity Spectra
The meromorphic functional calculus developed in Part I overcomes the
nondiagonalizability of linear operators that arises often in the temporal
evolution of complex systems and is generic to the metadynamics of predicting
their behavior. Using the resulting spectral decomposition, we derive
closed-form expressions for correlation functions, finite-length Shannon
entropy-rate approximates, asymptotic entropy rate, excess entropy, transient
information, transient and asymptotic state uncertainty, and synchronization
information of stochastic processes generated by finite-state hidden Markov
models. This introduces analytical tractability to investigating information
processing in discrete-event stochastic processes, symbolic dynamics, and
chaotic dynamical systems. Comparisons reveal mathematical similarities between
complexity measures originally thought to capture distinct informational and
computational properties. We also introduce a new kind of spectral analysis via
coronal spectrograms and the frequency-dependent spectra of past-future mutual
information. We analyze a number of examples to illustrate the methods,
emphasizing processes with multivariate dependencies beyond pairwise
correlation. An appendix presents spectral decomposition calculations for one
example in full detail.Comment: 27 pages, 12 figures, 2 tables; most recent version at
http://csc.ucdavis.edu/~cmg/compmech/pubs/sdscpt2.ht
Shannon Entropy Rate of Hidden Markov Processes
Hidden Markov chains are widely applied statistical models of stochastic
processes, from fundamental physics and chemistry to finance, health, and
artificial intelligence. The hidden Markov processes they generate are
notoriously complicated, however, even if the chain is finite state: no finite
expression for their Shannon entropy rate exists, as the set of their
predictive features is generically infinite. As such, to date one cannot make
general statements about how random they are nor how structured. Here, we
address the first part of this challenge by showing how to efficiently and
accurately calculate their entropy rates. We also show how this method gives
the minimal set of infinite predictive features. A sequel addresses the
challenge's second part on structure.Comment: 11 pages, 4 figures; supplementary material 10 pages, 7 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/serhmp.ht
Divergent Predictive States: The Statistical Complexity Dimension of Stationary, Ergodic Hidden Markov Processes
Even simply-defined, finite-state generators produce stochastic processes
that require tracking an uncountable infinity of probabilistic features for
optimal prediction. For processes generated by hidden Markov chains the
consequences are dramatic. Their predictive models are generically
infinite-state. And, until recently, one could determine neither their
intrinsic randomness nor structural complexity. The prequel, though, introduced
methods to accurately calculate the Shannon entropy rate (randomness) and to
constructively determine their minimal (though, infinite) set of predictive
features. Leveraging this, we address the complementary challenge of
determining how structured hidden Markov processes are by calculating their
statistical complexity dimension -- the information dimension of the minimal
set of predictive features. This tracks the divergence rate of the minimal
memory resources required to optimally predict a broad class of truly complex
processes.Comment: 16 pages, 6 figures; Supplementary Material, 6 pages, 2 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/icfshmp.ht
Prediction and Dissipation in Nonequilibrium Molecular Sensors: Conditionally Markovian Channels Driven by Memoryful Environments.
Biological sensors must often predict their input while operating under metabolic constraints. However, determining whether or not a particular sensor is evolved or designed to be accurate and efficient is challenging. This arises partly from the functional constraints being at cross purposes and partly since quantifying the prediction performance of even in silico sensors can require prohibitively long simulations, especially when highly complex environments drive sensors out of equilibrium. To circumvent these difficulties, we develop new expressions for the prediction accuracy and thermodynamic costs of the broad class of conditionally Markovian sensors subject to complex, correlated (unifilar hidden semi-Markov) environmental inputs in nonequilibrium steady state. Predictive metrics include the instantaneous memory and the total predictable information (the mutual information between present sensor state and input future), while dissipation metrics include power extracted from the environment and the nonpredictive information rate. Success in deriving these formulae relies on identifying the environment's causal states, the input's minimal sufficient statistics for prediction. Using these formulae, we study large random channels and the simplest nontrivial biological sensor model-that of a Hill molecule, characterized by the number of ligands that bind simultaneously-the sensor's cooperativity. We find that the seemingly impoverished Hill molecule can capture an order of magnitude more predictable information than large random channels