9,980 research outputs found
The Computational Structure of Spike Trains
Neurons perform computations, and convey the results of those computations
through the statistical structure of their output spike trains. Here we present
a practical method, grounded in the information-theoretic analysis of
prediction, for inferring a minimal representation of that structure and for
characterizing its complexity. Starting from spike trains, our approach finds
their causal state models (CSMs), the minimal hidden Markov models or
stochastic automata capable of generating statistically identical time series.
We then use these CSMs to objectively quantify both the generalizable structure
and the idiosyncratic randomness of the spike train. Specifically, we show that
the expected algorithmic information content (the information needed to
describe the spike train exactly) can be split into three parts describing (1)
the time-invariant structure (complexity) of the minimal spike-generating
process, which describes the spike train statistically; (2) the randomness
(internal entropy rate) of the minimal spike-generating process; and (3) a
residual pure noise term not described by the minimal spike-generating process.
We use CSMs to approximate each of these quantities. The CSMs are inferred
nonparametrically from the data, making only mild regularity assumptions, via
the causal state splitting reconstruction algorithm. The methods presented here
complement more traditional spike train analyses by describing not only spiking
probability and spike train entropy, but also the complexity of a spike train's
structure. We demonstrate our approach using both simulated spike trains and
experimental data recorded in rat barrel cortex during vibrissa stimulation.Comment: Somewhat different format from journal version but same conten
Thermodynamic Depth of Causal States: When Paddling around in Occam's Pool Shallowness Is a Virtue
Thermodynamic depth is an appealing but flawed structural complexity measure.
It depends on a set of macroscopic states for a system, but neither its
original introduction by Lloyd and Pagels nor any follow-up work has considered
how to select these states. Depth, therefore, is at root arbitrary.
Computational mechanics, an alternative approach to structural complexity,
provides a definition for a system's minimal, necessary causal states and a
procedure for finding them. We show that the rate of increase in thermodynamic
depth, or {\it dive}, is the system's reverse-time Shannon entropy rate, and so
depth only measures degrees of macroscopic randomness, not structure. To fix
this we redefine the depth in terms of the causal state
representation----machines---and show that this representation gives
the minimum dive consistent with accurate prediction. Thus, -machines
are optimally shallow.Comment: 11 pages, 9 figures, RevTe
Bayesian Updating, Model Class Selection and Robust Stochastic Predictions of Structural Response
A fundamental issue when predicting structural response by using mathematical models is how to treat both modeling and excitation uncertainty. A general framework for this is presented which uses probability as a multi-valued
conditional logic for quantitative plausible reasoning in the presence of uncertainty due to incomplete information. The
fundamental probability models that represent the structure’s uncertain behavior are specified by the choice of a stochastic
system model class: a set of input-output probability models for the structure and a prior probability distribution over this set
that quantifies the relative plausibility of each model. A model class can be constructed from a parameterized deterministic
structural model by stochastic embedding utilizing Jaynes’ Principle of Maximum Information Entropy. Robust predictive
analyses use the entire model class with the probabilistic predictions of each model being weighted by its prior probability, or if
structural response data is available, by its posterior probability from Bayes’ Theorem for the model class. Additional robustness
to modeling uncertainty comes from combining the robust predictions of each model class in a set of competing candidates
weighted by the prior or posterior probability of the model class, the latter being computed from Bayes’ Theorem. This higherlevel application of Bayes’ Theorem automatically applies a quantitative Ockham razor that penalizes the data-fit of more
complex model classes that extract more information from the data. Robust predictive analyses involve integrals over highdimensional spaces that usually must be evaluated numerically. Published applications have used Laplace's method of
asymptotic approximation or Markov Chain Monte Carlo algorithms
Spectral Simplicity of Apparent Complexity, Part I: The Nondiagonalizable Metadynamics of Prediction
Virtually all questions that one can ask about the behavioral and structural
complexity of a stochastic process reduce to a linear algebraic framing of a
time evolution governed by an appropriate hidden-Markov process generator. Each
type of question---correlation, predictability, predictive cost, observer
synchronization, and the like---induces a distinct generator class. Answers are
then functions of the class-appropriate transition dynamic. Unfortunately,
these dynamics are generically nonnormal, nondiagonalizable, singular, and so
on. Tractably analyzing these dynamics relies on adapting the recently
introduced meromorphic functional calculus, which specifies the spectral
decomposition of functions of nondiagonalizable linear operators, even when the
function poles and zeros coincide with the operator's spectrum. Along the way,
we establish special properties of the projection operators that demonstrate
how they capture the organization of subprocesses within a complex system.
Circumventing the spurious infinities of alternative calculi, this leads in the
sequel, Part II, to the first closed-form expressions for complexity measures,
couched either in terms of the Drazin inverse (negative-one power of a singular
operator) or the eigenvalues and projection operators of the appropriate
transition dynamic.Comment: 24 pages, 3 figures, 4 tables; current version always at
http://csc.ucdavis.edu/~cmg/compmech/pubs/sdscpt1.ht
Synchronization and Control in Intrinsic and Designed Computation: An Information-Theoretic Analysis of Competing Models of Stochastic Computation
We adapt tools from information theory to analyze how an observer comes to
synchronize with the hidden states of a finitary, stationary stochastic
process. We show that synchronization is determined by both the process's
internal organization and by an observer's model of it. We analyze these
components using the convergence of state-block and block-state entropies,
comparing them to the previously known convergence properties of the Shannon
block entropy. Along the way, we introduce a hierarchy of information
quantifiers as derivatives and integrals of these entropies, which parallels a
similar hierarchy introduced for block entropy. We also draw out the duality
between synchronization properties and a process's controllability. The tools
lead to a new classification of a process's alternative representations in
terms of minimality, synchronizability, and unifilarity.Comment: 25 pages, 13 figures, 1 tabl
Informational and Causal Architecture of Discrete-Time Renewal Processes
Renewal processes are broadly used to model stochastic behavior consisting of
isolated events separated by periods of quiescence, whose durations are
specified by a given probability law. Here, we identify the minimal sufficient
statistic for their prediction (the set of causal states), calculate the
historical memory capacity required to store those states (statistical
complexity), delineate what information is predictable (excess entropy), and
decompose the entropy of a single measurement into that shared with the past,
future, or both. The causal state equivalence relation defines a new subclass
of renewal processes with a finite number of causal states despite having an
unbounded interevent count distribution. We use these formulae to analyze the
output of the parametrized Simple Nonunifilar Source, generated by a simple
two-state hidden Markov model, but with an infinite-state epsilon-machine
presentation. All in all, the results lay the groundwork for analyzing
processes with infinite statistical complexity and infinite excess entropy.Comment: 18 pages, 9 figures, 1 table;
http://csc.ucdavis.edu/~cmg/compmech/pubs/dtrp.ht
Bayesian Structural Inference for Hidden Processes
We introduce a Bayesian approach to discovering patterns in structurally
complex processes. The proposed method of Bayesian Structural Inference (BSI)
relies on a set of candidate unifilar HMM (uHMM) topologies for inference of
process structure from a data series. We employ a recently developed exact
enumeration of topological epsilon-machines. (A sequel then removes the
topological restriction.) This subset of the uHMM topologies has the added
benefit that inferred models are guaranteed to be epsilon-machines,
irrespective of estimated transition probabilities. Properties of
epsilon-machines and uHMMs allow for the derivation of analytic expressions for
estimating transition probabilities, inferring start states, and comparing the
posterior probability of candidate model topologies, despite process internal
structure being only indirectly present in data. We demonstrate BSI's
effectiveness in estimating a process's randomness, as reflected by the Shannon
entropy rate, and its structure, as quantified by the statistical complexity.
We also compare using the posterior distribution over candidate models and the
single, maximum a posteriori model for point estimation and show that the
former more accurately reflects uncertainty in estimated values. We apply BSI
to in-class examples of finite- and infinite-order Markov processes, as well to
an out-of-class, infinite-state hidden process.Comment: 20 pages, 11 figures, 1 table; supplementary materials, 15 pages, 11
figures, 6 tables; http://csc.ucdavis.edu/~cmg/compmech/pubs/bsihp.ht
- …