1,534 research outputs found
Blind Construction of Optimal Nonlinear Recursive Predictors for Discrete Sequences
We present a new method for nonlinear prediction of discrete random sequences
under minimal structural assumptions. We give a mathematical construction for
optimal predictors of such processes, in the form of hidden Markov models. We
then describe an algorithm, CSSR (Causal-State Splitting Reconstruction), which
approximates the ideal predictor from data. We discuss the reliability of CSSR,
its data requirements, and its performance in simulations. Finally, we compare
our approach to existing methods using variable-length Markov models and
cross-validated hidden Markov models, and show theoretically and experimentally
that our method delivers results superior to the former and at least comparable
to the latter.Comment: 8 pages, 4 figure
An associative memory for the on-line recognition and prediction of temporal sequences
This paper presents the design of an associative memory with feedback that is
capable of on-line temporal sequence learning. A framework for on-line sequence
learning has been proposed, and different sequence learning models have been
analysed according to this framework. The network model is an associative
memory with a separate store for the sequence context of a symbol. A sparse
distributed memory is used to gain scalability. The context store combines the
functionality of a neural layer with a shift register. The sensitivity of the
machine to the sequence context is controllable, resulting in different
characteristic behaviours. The model can store and predict on-line sequences of
various types and length. Numerical simulations on the model have been carried
out to determine its properties.Comment: Published in IJCNN 2005, Montreal, Canad
Prediction, Retrodiction, and The Amount of Information Stored in the Present
We introduce an ambidextrous view of stochastic dynamical systems, comparing
their forward-time and reverse-time representations and then integrating them
into a single time-symmetric representation. The perspective is useful
theoretically, computationally, and conceptually. Mathematically, we prove that
the excess entropy--a familiar measure of organization in complex systems--is
the mutual information not only between the past and future, but also between
the predictive and retrodictive causal states. Practically, we exploit the
connection between prediction and retrodiction to directly calculate the excess
entropy. Conceptually, these lead one to discover new system invariants for
stochastic dynamical systems: crypticity (information accessibility) and causal
irreversibility. Ultimately, we introduce a time-symmetric representation that
unifies all these quantities, compressing the two directional representations
into one. The resulting compression offers a new conception of the amount of
information stored in the present.Comment: 17 pages, 7 figures, 1 table;
http://users.cse.ucdavis.edu/~cmg/compmech/pubs/pratisp.ht
Synchronization and Control in Intrinsic and Designed Computation: An Information-Theoretic Analysis of Competing Models of Stochastic Computation
We adapt tools from information theory to analyze how an observer comes to
synchronize with the hidden states of a finitary, stationary stochastic
process. We show that synchronization is determined by both the process's
internal organization and by an observer's model of it. We analyze these
components using the convergence of state-block and block-state entropies,
comparing them to the previously known convergence properties of the Shannon
block entropy. Along the way, we introduce a hierarchy of information
quantifiers as derivatives and integrals of these entropies, which parallels a
similar hierarchy introduced for block entropy. We also draw out the duality
between synchronization properties and a process's controllability. The tools
lead to a new classification of a process's alternative representations in
terms of minimality, synchronizability, and unifilarity.Comment: 25 pages, 13 figures, 1 tabl
- …