1,534 research outputs found
Transfer Entropy as a Log-likelihood Ratio
Transfer entropy, an information-theoretic measure of time-directed
information transfer between joint processes, has steadily gained popularity in
the analysis of complex stochastic dynamics in diverse fields, including the
neurosciences, ecology, climatology and econometrics. We show that for a broad
class of predictive models, the log-likelihood ratio test statistic for the
null hypothesis of zero transfer entropy is a consistent estimator for the
transfer entropy itself. For finite Markov chains, furthermore, no explicit
model is required. In the general case, an asymptotic chi-squared distribution
is established for the transfer entropy estimator. The result generalises the
equivalence in the Gaussian case of transfer entropy and Granger causality, a
statistical notion of causal influence based on prediction via vector
autoregression, and establishes a fundamental connection between directed
information transfer and causality in the Wiener-Granger sense
Efficient transfer entropy analysis of non-stationary neural time series
Information theory allows us to investigate information processing in neural
systems in terms of information transfer, storage and modification. Especially
the measure of information transfer, transfer entropy, has seen a dramatic
surge of interest in neuroscience. Estimating transfer entropy from two
processes requires the observation of multiple realizations of these processes
to estimate associated probability density functions. To obtain these
observations, available estimators assume stationarity of processes to allow
pooling of observations over time. This assumption however, is a major obstacle
to the application of these estimators in neuroscience as observed processes
are often non-stationary. As a solution, Gomez-Herrero and colleagues
theoretically showed that the stationarity assumption may be avoided by
estimating transfer entropy from an ensemble of realizations. Such an ensemble
is often readily available in neuroscience experiments in the form of
experimental trials. Thus, in this work we combine the ensemble method with a
recently proposed transfer entropy estimator to make transfer entropy
estimation applicable to non-stationary time series. We present an efficient
implementation of the approach that deals with the increased computational
demand of the ensemble method's practical application. In particular, we use a
massively parallel implementation for a graphics processing unit to handle the
computationally most heavy aspects of the ensemble method. We test the
performance and robustness of our implementation on data from simulated
stochastic processes and demonstrate the method's applicability to
magnetoencephalographic data. While we mainly evaluate the proposed method for
neuroscientific data, we expect it to be applicable in a variety of fields that
are concerned with the analysis of information transfer in complex biological,
social, and artificial systems.Comment: 27 pages, 7 figures, submitted to PLOS ON
JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
Complex systems are increasingly being viewed as distributed information
processing systems, particularly in the domains of computational neuroscience,
bioinformatics and Artificial Life. This trend has resulted in a strong uptake
in the use of (Shannon) information-theoretic measures to analyse the dynamics
of complex systems in these fields. We introduce the Java Information Dynamics
Toolkit (JIDT): a Google code project which provides a standalone, (GNU GPL v3
licensed) open-source code implementation for empirical estimation of
information-theoretic measures from time-series data. While the toolkit
provides classic information-theoretic measures (e.g. entropy, mutual
information, conditional mutual information), it ultimately focusses on
implementing higher-level measures for information dynamics. That is, JIDT
focusses on quantifying information storage, transfer and modification, and the
dynamics of these operations in space and time. For this purpose, it includes
implementations of the transfer entropy and active information storage, their
multivariate extensions and local or pointwise variants. JIDT provides
implementations for both discrete and continuous-valued data for each measure,
including various types of estimator for continuous data (e.g. Gaussian,
box-kernel and Kraskov-Stoegbauer-Grassberger) which can be swapped at run-time
due to Java's object-oriented polymorphism. Furthermore, while written in Java,
the toolkit can be used directly in MATLAB, GNU Octave, Python and other
environments. We present the principles behind the code design, and provide
several examples to guide users.Comment: 37 pages, 4 figure
Synchronisation effects on the behavioural performance and information dynamics of a simulated minimally cognitive robotic agent
Oscillatory activity is ubiquitous in nervous systems, with solid evidence that synchronisation mechanisms underpin cognitive processes. Nevertheless, its informational content and relationship with behaviour are still to be fully understood. In addition, cognitive systems cannot be properly appreciated without taking into account brain–body– environment interactions. In this paper, we developed a model based on the Kuramoto Model of coupled phase oscillators to explore the role of neural synchronisation in the performance of a simulated robotic agent in two different minimally cognitive tasks. We show that there is a statistically significant difference in performance and evolvability depending on the synchronisation regime of the network. In both tasks, a combination of information flow and dynamical analyses show that networks with a definite, but not too strong, propensity for synchronisation are more able to reconfigure, to organise themselves functionally and to adapt to different behavioural conditions. The results highlight the asymmetry of information flow and its behavioural correspondence. Importantly, it also shows that neural synchronisation dynamics, when suitably flexible and reconfigurable, can generate minimally cognitive embodied behaviour
Representation Learning: A Review and New Perspectives
The success of machine learning algorithms generally depends on data
representation, and we hypothesize that this is because different
representations can entangle and hide more or less the different explanatory
factors of variation behind the data. Although specific domain knowledge can be
used to help design representations, learning with generic priors can also be
used, and the quest for AI is motivating the design of more powerful
representation-learning algorithms implementing such priors. This paper reviews
recent work in the area of unsupervised feature learning and deep learning,
covering advances in probabilistic models, auto-encoders, manifold learning,
and deep networks. This motivates longer-term unanswered questions about the
appropriate objectives for learning good representations, for computing
representations (i.e., inference), and the geometrical connections between
representation learning, density estimation and manifold learning
Estimating Gene Interactions Using Information Theoretic Functionals
With an abundance of data resulting from high-throughput technologies, like DNA microarrays,
a race has been on the last few years, to determine the structures and functions of genes and
their products, the proteins. Inference of gene interactions, lies in the core of these efforts.
In all this activity, three important research issues have emerged. First, in much of the current
literature on gene regulatory networks, dependencies among variables in our case genes - are
assumed to be linear in nature, when in fact, in real-life scenarios this is seldom the case.
This disagreement leads to systematic deviation and biased evaluation. Secondly, although
the problem of undersampling, features in every piece of work as one of the major causes for
poor results, in practice it is overlooked and rarely addressed explicitly. Finally, inference
of network structures, although based on rigid mathematical foundations and computational
optimizations, often displays poor fitness values and biologically unrealistic link structures, due
- to a large extend - to the discovery of pairwise only interactions.
In our search for robust, nonlinear measures of dependency, we advocate that mutual information
and related information theoretic functionals (conditional mutual information, total
correlation) are possibly the most suitable candidates to capture both linear and nonlinear
interactions between variables, and resolve higher order dependencies.
To address these issues, we researched and implemented under a common framework, a selection
nonparametric estimators of mutual information for continuous variables. The focus of their
assessment was, their robustness to the limited sample sizes and their expansibility to higher
dimensions - important for the detection of more complex interaction structures. Two different
assessment scenaria were performed, one with simulated data and one with bootstrapping the
estimators in state-of-the-art network inference algorithms and monitor their predictive power
and sensitivity. The tests revealed that, in small sample size regimes, there is a significant difference
in the performance of different estimators, and naive methods such as uniform binning,
gave consistently poor results compared with more sophisticated methods.
Finally, a custom, modular mechanism is proposed, for the inference of gene interactions,
targeting the identi cation of some of the most common substructures in genetic networks,
that we believe will help improve accuracy and predictability scores
Multivariate Granger Causality and Generalized Variance
Granger causality analysis is a popular method for inference on directed
interactions in complex systems of many variables. A shortcoming of the
standard framework for Granger causality is that it only allows for examination
of interactions between single (univariate) variables within a system, perhaps
conditioned on other variables. However, interactions do not necessarily take
place between single variables, but may occur among groups, or "ensembles", of
variables. In this study we establish a principled framework for Granger
causality in the context of causal interactions among two or more multivariate
sets of variables. Building on Geweke's seminal 1982 work, we offer new
justifications for one particular form of multivariate Granger causality based
on the generalized variances of residual errors. Taken together, our results
support a comprehensive and theoretically consistent extension of Granger
causality to the multivariate case. Treated individually, they highlight
several specific advantages of the generalized variance measure, which we
illustrate using applications in neuroscience as an example. We further show
how the measure can be used to define "partial" Granger causality in the
multivariate context and we also motivate reformulations of "causal density"
and "Granger autonomy". Our results are directly applicable to experimental
data and promise to reveal new types of functional relations in complex
systems, neural and otherwise.Comment: added 1 reference, minor change to discussion, typos corrected; 28
pages, 3 figures, 1 table, LaTe
Analysis of Embodied and Situated Systems from an Antireductionist Perspective
The analysis of embodied and situated agents form a dynamical system perspective is often
limited to a geometrical and qualitative description. However, a quantitative analysis is necessary
to achieve a deep understanding of cognitive facts.
The field of embodied cognition is multifaceted, and the first part of this thesis is devoted to exploring
the diverse meanings proposed in the existing literature. This is a preliminary fundamental
step as the creation of synthetic models requires well-founded theoretical and foundational
boundaries for operationalising the concept of embodied and situated cognition in a concrete
neuro-robotic model. By accepting the dynamical system view the agent is conceived as highly
integrated and strictly coupled with the surrounding environment. Therefore the antireductionist
framework is followed during the analysis of such systems, using chaos theory to unveil global
properties and information theory to describe the complex network of interactions among the
heterogeneous sub-components.
In the experimental section, several evolutionary robotics experiments are discussed. This class
of adaptive systems is consistent with the proposed definition of embodied and situated cognition.
In fact, such neuro-robotics platforms autonomously develop a solution to a problem exploiting
the continuous sensorimotor interaction with the environment.
The first experiment is a stress test for chaos theory, a mathematical framework that studies erratic
behaviour in low-dimensional and deterministic dynamical systems. The recorded dataset
consists of the robots’ position in the environment during the execution of the task. Subsequently,
the time series is projected onto a multidimensional phase space in order to study the underlying
dynamic using chaotic numerical descriptors. Finally, such measures are correlated and confronted
with the robots’ behavioural strategy and the performance in novel and unpredictable
environments.
The second experiment explores the possible applications of information-theoretic measures for
the analysis of embodied and situated systems. Data is recorded from perceptual and motor
neurons while robots are executing a wall-following task and pairwise estimations of the mutual
information and the transfer entropy are calculated in order to create an exhaustive map of the
nonlinear interactions among variables. Results show that the set of information-theoretic employed
in this study unveils characteristics of the agent-environemnt interaction and the functional
neural structure.
This work aims at testing the explanatory power and impotence of nonlinear time series analysis
applied to observables recorded from neuro-robotics embodied and situated systems
- …