7,768 research outputs found
Spectral Simplicity of Apparent Complexity, Part I: The Nondiagonalizable Metadynamics of Prediction
Virtually all questions that one can ask about the behavioral and structural
complexity of a stochastic process reduce to a linear algebraic framing of a
time evolution governed by an appropriate hidden-Markov process generator. Each
type of question---correlation, predictability, predictive cost, observer
synchronization, and the like---induces a distinct generator class. Answers are
then functions of the class-appropriate transition dynamic. Unfortunately,
these dynamics are generically nonnormal, nondiagonalizable, singular, and so
on. Tractably analyzing these dynamics relies on adapting the recently
introduced meromorphic functional calculus, which specifies the spectral
decomposition of functions of nondiagonalizable linear operators, even when the
function poles and zeros coincide with the operator's spectrum. Along the way,
we establish special properties of the projection operators that demonstrate
how they capture the organization of subprocesses within a complex system.
Circumventing the spurious infinities of alternative calculi, this leads in the
sequel, Part II, to the first closed-form expressions for complexity measures,
couched either in terms of the Drazin inverse (negative-one power of a singular
operator) or the eigenvalues and projection operators of the appropriate
transition dynamic.Comment: 24 pages, 3 figures, 4 tables; current version always at
http://csc.ucdavis.edu/~cmg/compmech/pubs/sdscpt1.ht
Capacity and Complexity of HMM Duration Modeling Techniques
The ability of a standard hidden Markov model (HMM) or expanded state HMM (ESHMM) to accurately model duration distributions of phonemes is compared with specific duration-focused approaches such as semi-Markov models or variable transition probabilities. It is demonstrated that either a three-state ESHMM or a standard HMM with an increased number of states is capable of closely matching both Gamma distributions and duration distributions of phonemes from the TIMIT corpus, as measured by Bhattacharyya distance to the true distributions. Standard HMMs are easily implemented with off-the-shelf tools, whereas duration models require substantial algorithmic development and have higher computational costs when implemented, suggesting that a simple adjustment to HMM topologies is perhaps a more efficient solution to the problem of duration than more complex approaches
Causal inference using the algorithmic Markov condition
Inferring the causal structure that links n observables is usually based upon
detecting statistical dependences and choosing simple graphs that make the
joint measure Markovian. Here we argue why causal inference is also possible
when only single observations are present.
We develop a theory how to generate causal graphs explaining similarities
between single objects. To this end, we replace the notion of conditional
stochastic independence in the causal Markov condition with the vanishing of
conditional algorithmic mutual information and describe the corresponding
causal inference rules.
We explain why a consistent reformulation of causal inference in terms of
algorithmic complexity implies a new inference principle that takes into
account also the complexity of conditional probability densities, making it
possible to select among Markov equivalent causal graphs. This insight provides
a theoretical foundation of a heuristic principle proposed in earlier work.
We also discuss how to replace Kolmogorov complexity with decidable
complexity criteria. This can be seen as an algorithmic analog of replacing the
empirically undecidable question of statistical independence with practical
independence tests that are based on implicit or explicit assumptions on the
underlying distribution.Comment: 16 figure
Spectral Simplicity of Apparent Complexity, Part II: Exact Complexities and Complexity Spectra
The meromorphic functional calculus developed in Part I overcomes the
nondiagonalizability of linear operators that arises often in the temporal
evolution of complex systems and is generic to the metadynamics of predicting
their behavior. Using the resulting spectral decomposition, we derive
closed-form expressions for correlation functions, finite-length Shannon
entropy-rate approximates, asymptotic entropy rate, excess entropy, transient
information, transient and asymptotic state uncertainty, and synchronization
information of stochastic processes generated by finite-state hidden Markov
models. This introduces analytical tractability to investigating information
processing in discrete-event stochastic processes, symbolic dynamics, and
chaotic dynamical systems. Comparisons reveal mathematical similarities between
complexity measures originally thought to capture distinct informational and
computational properties. We also introduce a new kind of spectral analysis via
coronal spectrograms and the frequency-dependent spectra of past-future mutual
information. We analyze a number of examples to illustrate the methods,
emphasizing processes with multivariate dependencies beyond pairwise
correlation. An appendix presents spectral decomposition calculations for one
example in full detail.Comment: 27 pages, 12 figures, 2 tables; most recent version at
http://csc.ucdavis.edu/~cmg/compmech/pubs/sdscpt2.ht
Prediction, Retrodiction, and The Amount of Information Stored in the Present
We introduce an ambidextrous view of stochastic dynamical systems, comparing
their forward-time and reverse-time representations and then integrating them
into a single time-symmetric representation. The perspective is useful
theoretically, computationally, and conceptually. Mathematically, we prove that
the excess entropy--a familiar measure of organization in complex systems--is
the mutual information not only between the past and future, but also between
the predictive and retrodictive causal states. Practically, we exploit the
connection between prediction and retrodiction to directly calculate the excess
entropy. Conceptually, these lead one to discover new system invariants for
stochastic dynamical systems: crypticity (information accessibility) and causal
irreversibility. Ultimately, we introduce a time-symmetric representation that
unifies all these quantities, compressing the two directional representations
into one. The resulting compression offers a new conception of the amount of
information stored in the present.Comment: 17 pages, 7 figures, 1 table;
http://users.cse.ucdavis.edu/~cmg/compmech/pubs/pratisp.ht
Strong and Weak Optimizations in Classical and Quantum Models of Stochastic Processes
Among the predictive hidden Markov models that describe a given stochastic
process, the {\epsilon}-machine is strongly minimal in that it minimizes every
R\'enyi-based memory measure. Quantum models can be smaller still. In contrast
with the {\epsilon}-machine's unique role in the classical setting, however,
among the class of processes described by pure-state hidden quantum Markov
models, there are those for which there does not exist any strongly minimal
model. Quantum memory optimization then depends on which memory measure best
matches a given problem circumstance.Comment: 14 pages, 14 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/uemum.ht
Informational and Causal Architecture of Continuous-time Renewal and Hidden Semi-Markov Processes
We introduce the minimal maximally predictive models ({\epsilon}-machines) of
processes generated by certain hidden semi-Markov models. Their causal states
are either hybrid discrete-continuous or continuous random variables and
causal-state transitions are described by partial differential equations.
Closed-form expressions are given for statistical complexities, excess
entropies, and differential information anatomy rates. We present a complete
analysis of the {\epsilon}-machines of continuous-time renewal processes and,
then, extend this to processes generated by unifilar hidden semi-Markov models
and semi-Markov models. Our information-theoretic analysis leads to new
expressions for the entropy rate and the rates of related information measures
for these very general continuous-time process classes.Comment: 16 pages, 7 figures;
http://csc.ucdavis.edu/~cmg/compmech/pubs/ctrp.ht
- …