1,084 research outputs found
A Recipe for the Estimation of Information Flow in a Dynamical System
Information-theoretic quantities, such as entropy and mutual information (MI), can be used to quantify the amount of information needed to describe a dataset or the information shared between two datasets. In the case of a dynamical system, the behavior of the relevant variables can be tightly coupled, such that information about one variable at a given instance in time may provide information about other variables at later instances in time. This is often viewed as a flow of information, and tracking such a flow can reveal relationships among the system variables. Since the MI is a symmetric quantity; an asymmetric quantity, called Transfer Entropy (TE), has been proposed to estimate the directionality of the coupling. However, accurate estimation of entropy-based measures is notoriously difficult. Every method has its own free tuning parameter(s) and there is no consensus on an optimal way of estimating the TE from a dataset. We propose a new methodology to estimate TE and apply a set of methods together as an accuracy cross-check to provide a reliable mathematical tool for any given data set. We demonstrate both the variability in TE estimation across techniques as well as the benefits of the proposed methodology to reliably estimate the directionality of coupling among variables
A Recipe for the Estimation of Information Flow in a Dynamical System
Information-theoretic quantities, such as entropy and mutual information (MI), can be used to quantify the amount of information needed to describe a dataset or the information shared between two datasets. In the case of a dynamical system, the behavior of the relevant variables can be tightly coupled, such that information about one variable at a given instance in time may provide information about other variables at later instances in time. This is often viewed as a flow of information, and tracking such a flow can reveal relationships among the system variables. Since the MI is a symmetric quantity; an asymmetric quantity, called Transfer Entropy (TE), has been proposed to estimate the directionality of the coupling. However, accurate estimation of entropy-based measures is notoriously difficult. Every method has its own free tuning parameter(s) and there is no consensus on an optimal way of estimating the TE from a dataset. We propose a new methodology to estimate TE and apply a set of methods together as an accuracy cross-check to provide a reliable mathematical tool for any given data set. We demonstrate both the variability in TE estimation across techniques as well as the benefits of the proposed methodology to reliably estimate the directionality of coupling among variables
Entropy: The Markov Ordering Approach
The focus of this article is on entropy and Markov processes. We study the
properties of functionals which are invariant with respect to monotonic
transformations and analyze two invariant "additivity" properties: (i)
existence of a monotonic transformation which makes the functional additive
with respect to the joining of independent systems and (ii) existence of a
monotonic transformation which makes the functional additive with respect to
the partitioning of the space of states. All Lyapunov functionals for Markov
chains which have properties (i) and (ii) are derived. We describe the most
general ordering of the distribution space, with respect to which all
continuous-time Markov processes are monotonic (the {\em Markov order}). The
solution differs significantly from the ordering given by the inequality of
entropy growth. For inference, this approach results in a convex compact set of
conditionally "most random" distributions.Comment: 50 pages, 4 figures, Postprint version. More detailed discussion of
the various entropy additivity properties and separation of variables for
independent subsystems in MaxEnt problem is added in Section 4.2.
Bibliography is extende
Definitions of entanglement entropy of spin systems in the valence-bond basis
The valence-bond structure of spin-1/2 Heisenberg antiferromagnets is closely
related to quantum entanglement. We investigate measures of entanglement
entropy based on transition graphs, which characterize state overlaps in the
overcomplete valence-bond basis. The transition graphs can be generated using
projector Monte Carlo simulations of ground states of specific hamiltonians or
using importance-sampling of valence-bond configurations of amplitude-product
states. We consider definitions of entanglement entropy based on the bonds or
loops shared by two subsystems (bipartite entanglement). Results for the
bond-based definition agrees with a previously studied definition using
valence-bond wave functions (instead of the transition graphs, which involve
two states). For the one dimensional Heisenberg chain, with uniform or random
coupling constants, the prefactor of the logarithmic divergence with the size
of the smaller subsystem agrees with exact results. For the ground state of the
two-dimensional Heisenberg model (and also Neel-ordered amplitude-product
states), there is a similar multiplicative violation of the area law. In
contrast, the loop-based entropy obeys the area law in two dimensions, while
still violating it in one dimension - both behaviors in accord with
expectations for proper measures of entanglement entropy.Comment: 9 pages, 8 figures. v2: significantly expande
Efficient transfer entropy analysis of non-stationary neural time series
Information theory allows us to investigate information processing in neural
systems in terms of information transfer, storage and modification. Especially
the measure of information transfer, transfer entropy, has seen a dramatic
surge of interest in neuroscience. Estimating transfer entropy from two
processes requires the observation of multiple realizations of these processes
to estimate associated probability density functions. To obtain these
observations, available estimators assume stationarity of processes to allow
pooling of observations over time. This assumption however, is a major obstacle
to the application of these estimators in neuroscience as observed processes
are often non-stationary. As a solution, Gomez-Herrero and colleagues
theoretically showed that the stationarity assumption may be avoided by
estimating transfer entropy from an ensemble of realizations. Such an ensemble
is often readily available in neuroscience experiments in the form of
experimental trials. Thus, in this work we combine the ensemble method with a
recently proposed transfer entropy estimator to make transfer entropy
estimation applicable to non-stationary time series. We present an efficient
implementation of the approach that deals with the increased computational
demand of the ensemble method's practical application. In particular, we use a
massively parallel implementation for a graphics processing unit to handle the
computationally most heavy aspects of the ensemble method. We test the
performance and robustness of our implementation on data from simulated
stochastic processes and demonstrate the method's applicability to
magnetoencephalographic data. While we mainly evaluate the proposed method for
neuroscientific data, we expect it to be applicable in a variety of fields that
are concerned with the analysis of information transfer in complex biological,
social, and artificial systems.Comment: 27 pages, 7 figures, submitted to PLOS ON
A hierarchical Bayesian network approach for linkage disequilibrium modeling and data-dimensionality reduction prior to genome-wide association studies
<p>Abstract</p> <p>Background</p> <p>Discovering the genetic basis of common genetic diseases in the human genome represents a public health issue. However, the dimensionality of the genetic data (up to 1 million genetic markers) and its complexity make the statistical analysis a challenging task.</p> <p>Results</p> <p>We present an accurate modeling of dependences between genetic markers, based on a forest of hierarchical latent class models which is a particular class of probabilistic graphical models. This model offers an adapted framework to deal with the fuzzy nature of linkage disequilibrium blocks. In addition, the data dimensionality can be reduced through the latent variables of the model which synthesize the information borne by genetic markers. In order to tackle the learning of both forest structure and probability distributions, a generic algorithm has been proposed. A first implementation of our algorithm has been shown to be tractable on benchmarks describing 10<sup>5 </sup>variables for 2000 individuals.</p> <p>Conclusions</p> <p>The forest of hierarchical latent class models offers several advantages for genome-wide association studies: accurate modeling of linkage disequilibrium, flexible data dimensionality reduction and biological meaning borne by latent variables.</p
- âŠ