381 research outputs found
Infinite Excess Entropy Processes with Countable-State Generators
We present two examples of finite-alphabet, infinite excess entropy processes
generated by invariant hidden Markov models (HMMs) with countable state sets.
The first, simpler example is not ergodic, but the second is. It appears these
are the first constructions of processes of this type. Previous examples of
infinite excess entropy processes over finite alphabets admit only invariant
HMM presentations with uncountable state sets.Comment: 13 pages, 3 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/ieepcsg.ht
Informational and Causal Architecture of Discrete-Time Renewal Processes
Renewal processes are broadly used to model stochastic behavior consisting of
isolated events separated by periods of quiescence, whose durations are
specified by a given probability law. Here, we identify the minimal sufficient
statistic for their prediction (the set of causal states), calculate the
historical memory capacity required to store those states (statistical
complexity), delineate what information is predictable (excess entropy), and
decompose the entropy of a single measurement into that shared with the past,
future, or both. The causal state equivalence relation defines a new subclass
of renewal processes with a finite number of causal states despite having an
unbounded interevent count distribution. We use these formulae to analyze the
output of the parametrized Simple Nonunifilar Source, generated by a simple
two-state hidden Markov model, but with an infinite-state epsilon-machine
presentation. All in all, the results lay the groundwork for analyzing
processes with infinite statistical complexity and infinite excess entropy.Comment: 18 pages, 9 figures, 1 table;
http://csc.ucdavis.edu/~cmg/compmech/pubs/dtrp.ht
Editorial Comment on the Special Issue of "Information in Dynamical Systems and Complex Systems"
This special issue collects contributions from the participants of the
"Information in Dynamical Systems and Complex Systems" workshop, which cover a
wide range of important problems and new approaches that lie in the
intersection of information theory and dynamical systems. The contributions
include theoretical characterization and understanding of the different types
of information flow and causality in general stochastic processes, inference
and identification of coupling structure and parameters of system dynamics,
rigorous coarse-grain modeling of network dynamical systems, and exact
statistical testing of fundamental information-theoretic quantities such as the
mutual information. The collective efforts reported herein reflect a modern
perspective of the intimate connection between dynamical systems and
information flow, leading to the promise of better understanding and modeling
of natural complex systems and better/optimal design of engineering systems
On Hidden Markov Processes with Infinite Excess Entropy
We investigate stationary hidden Markov processes for which mutual
information between the past and the future is infinite. It is assumed that the
number of observable states is finite and the number of hidden states is
countably infinite. Under this assumption, we show that the block mutual
information of a hidden Markov process is upper bounded by a power law
determined by the tail index of the hidden state distribution. Moreover, we
exhibit three examples of processes. The first example, considered previously,
is nonergodic and the mutual information between the blocks is bounded by the
logarithm of the block length. The second example is also nonergodic but the
mutual information between the blocks obeys a power law. The third example
obeys the power law and is ergodic.Comment: 12 page
The Origins of Computational Mechanics: A Brief Intellectual History and Several Clarifications
The principle goal of computational mechanics is to define pattern and
structure so that the organization of complex systems can be detected and
quantified. Computational mechanics developed from efforts in the 1970s and
early 1980s to identify strange attractors as the mechanism driving weak fluid
turbulence via the method of reconstructing attractor geometry from measurement
time series and in the mid-1980s to estimate equations of motion directly from
complex time series. In providing a mathematical and operational definition of
structure it addressed weaknesses of these early approaches to discovering
patterns in natural systems.
Since then, computational mechanics has led to a range of results from
theoretical physics and nonlinear mathematics to diverse applications---from
closed-form analysis of Markov and non-Markov stochastic processes that are
ergodic or nonergodic and their measures of information and intrinsic
computation to complex materials and deterministic chaos and intelligence in
Maxwellian demons to quantum compression of classical processes and the
evolution of computation and language.
This brief review clarifies several misunderstandings and addresses concerns
recently raised regarding early works in the field (1980s). We show that
misguided evaluations of the contributions of computational mechanics are
groundless and stem from a lack of familiarity with its basic goals and from a
failure to consider its historical context. For all practical purposes, its
modern methods and results largely supersede the early works. This not only
renders recent criticism moot and shows the solid ground on which computational
mechanics stands but, most importantly, shows the significant progress achieved
over three decades and points to the many intriguing and outstanding challenges
in understanding the computational nature of complex dynamic systems.Comment: 11 pages, 123 citations;
http://csc.ucdavis.edu/~cmg/compmech/pubs/cmr.ht
Signatures of Infinity: Nonergodicity and Resource Scaling in Prediction, Complexity, and Learning
We introduce a simple analysis of the structural complexity of
infinite-memory processes built from random samples of stationary, ergodic
finite-memory component processes. Such processes are familiar from the well
known multi-arm Bandit problem. We contrast our analysis with
computation-theoretic and statistical inference approaches to understanding
their complexity. The result is an alternative view of the relationship between
predictability, complexity, and learning that highlights the distinct ways in
which informational and correlational divergences arise in complex ergodic and
nonergodic processes. We draw out consequences for the resource divergences
that delineate the structural hierarchy of ergodic processes and for processes
that are themselves hierarchical.Comment: 8 pages, 1 figure; http://csc.ucdavis.edu/~cmg/compmech/pubs/soi.pd
Statistical Signatures of Structural Organization: The case of long memory in renewal processes
Identifying and quantifying memory are often critical steps in developing a
mechanistic understanding of stochastic processes. These are particularly
challenging and necessary when exploring processes that exhibit long-range
correlations. The most common signatures employed rely on second-order temporal
statistics and lead, for example, to identifying long memory in processes with
power-law autocorrelation function and Hurst exponent greater than .
However, most stochastic processes hide their memory in higher-order temporal
correlations. Information measures---specifically, divergences in the mutual
information between a process' past and future (excess entropy) and minimal
predictive memory stored in a process' causal states (statistical
complexity)---provide a different way to identify long memory in processes with
higher-order temporal correlations. However, there are no ergodic stationary
processes with infinite excess entropy for which information measures have been
compared to autocorrelation functions and Hurst exponents. Here, we show that
fractal renewal processes---those with interevent distribution tails ---exhibit long memory via a phase transition at .
Excess entropy diverges only there and statistical complexity diverges there
and for all . When these processes do have power-law
autocorrelation function and Hurst exponent greater than , they do not
have divergent excess entropy. This analysis breaks the intuitive association
between these different quantifications of memory. We hope that the methods
used here, based on causal states, provide some guide as to how to construct
and analyze other long memory processes.Comment: 13 pages, 2 figures, 3 appendixes;
http://csc.ucdavis.edu/~cmg/compmech/pubs/lrmrp.ht
Beyond the Spectral Theorem: Spectrally Decomposing Arbitrary Functions of Nondiagonalizable Operators
Nonlinearities in finite dimensions can be linearized by projecting them into
infinite dimensions. Unfortunately, often the linear operator techniques that
one would then use simply fail since the operators cannot be diagonalized. This
curse is well known. It also occurs for finite-dimensional linear operators. We
circumvent it by developing a meromorphic functional calculus that can
decompose arbitrary functions of nondiagonalizable linear operators in terms of
their eigenvalues and projection operators. It extends the spectral theorem of
normal operators to a much wider class, including circumstances in which poles
and zeros of the function coincide with the operator spectrum. By allowing the
direct manipulation of individual eigenspaces of nonnormal and
nondiagonalizable operators, the new theory avoids spurious divergences. As
such, it yields novel insights and closed-form expressions across several areas
of physics in which nondiagonalizable dynamics are relevant, including
memoryful stochastic processes, open non unitary quantum systems, and
far-from-equilibrium thermodynamics.
The technical contributions include the first full treatment of arbitrary
powers of an operator. In particular, we show that the Drazin inverse,
previously only defined axiomatically, can be derived as the negative-one power
of singular operators within the meromorphic functional calculus and we give a
general method to construct it. We provide new formulae for constructing
projection operators and delineate the relations between projection operators,
eigenvectors, and generalized eigenvectors.
By way of illustrating its application, we explore several, rather distinct
examples.Comment: 29 pages, 4 figures, expanded historical citations;
http://csc.ucdavis.edu/~cmg/compmech/pubs/bst.ht
Information Symmetries in Irreversible Processes
We study dynamical reversibility in stationary stochastic processes from an
information theoretic perspective. Extending earlier work on the reversibility
of Markov chains, we focus on finitary processes with arbitrarily long
conditional correlations. In particular, we examine stationary processes
represented or generated by edge-emitting, finite-state hidden Markov models.
Surprisingly, we find pervasive temporal asymmetries in the statistics of such
stationary processes with the consequence that the computational resources
necessary to generate a process in the forward and reverse temporal directions
are generally not the same. In fact, an exhaustive survey indicates that most
stationary processes are irreversible. We study the ensuing relations between
model topology in different representations, the process's statistical
properties, and its reversibility in detail. A process's temporal asymmetry is
efficiently captured using two canonical unifilar representations of the
generating model, the forward-time and reverse-time epsilon-machines. We
analyze example irreversible processes whose epsilon-machine presentations
change size under time reversal, including one which has a finite number of
recurrent causal states in one direction, but an infinite number in the
opposite. From the forward-time and reverse-time epsilon-machines, we are able
to construct a symmetrized, but nonunifilar, generator of a process---the
bidirectional machine. Using the bidirectional machine, we show how to directly
calculate a process's fundamental information properties, many of which are
otherwise only poorly approximated via process samples. The tools we introduce
and the insights we offer provide a better understanding of the many facets of
reversibility and irreversibility in stochastic processes.Comment: 32 pages, 17 figures, 2 tables;
http://csc.ucdavis.edu/~cmg/compmech/pubs/pratisp2.ht
Mixing, Ergodic, and Nonergodic Processes with Rapidly Growing Information between Blocks
We construct mixing processes over an infinite alphabet and ergodic processes
over a finite alphabet for which Shannon mutual information between adjacent
blocks of length grows as , where . The processes
are a modification of nonergodic Santa Fe processes, which were introduced in
the context of natural language modeling. The rates of mutual information for
the latter processes are alike and also established in this paper. As an
auxiliary result, it is shown that infinite direct products of mixing processes
are also mixing.Comment: 21 page
- …