13 research outputs found
Complex Networks
Introduction to the Special Issue on Complex Networks, Artificial Life
journal.Comment: 7 pages, in pres
A characterization of the Edge of Criticality in Binary Echo State Networks
Echo State Networks (ESNs) are simplified recurrent neural network models
composed of a reservoir and a linear, trainable readout layer. The reservoir is
tunable by some hyper-parameters that control the network behaviour. ESNs are
known to be effective in solving tasks when configured on a region in
(hyper-)parameter space called \emph{Edge of Criticality} (EoC), where the
system is maximally sensitive to perturbations hence affecting its behaviour.
In this paper, we propose binary ESNs, which are architecturally equivalent to
standard ESNs but consider binary activation functions and binary recurrent
weights. For these networks, we derive a closed-form expression for the EoC in
the autonomous case and perform simulations in order to assess their behavior
in the case of noisy neurons and in the presence of a signal. We propose a
theoretical explanation for the fact that the variance of the input plays a
major role in characterizing the EoC
Dynamic Adaptive Computation: Tuning network states to task requirements
Neural circuits are able to perform computations under very diverse
conditions and requirements. The required computations impose clear constraints
on their fine-tuning: a rapid and maximally informative response to stimuli in
general requires decorrelated baseline neural activity. Such network dynamics
is known as asynchronous-irregular. In contrast, spatio-temporal integration of
information requires maintenance and transfer of stimulus information over
extended time periods. This can be realized at criticality, a phase transition
where correlations, sensitivity and integration time diverge. Being able to
flexibly switch, or even combine the above properties in a task-dependent
manner would present a clear functional advantage. We propose that cortex
operates in a "reverberating regime" because it is particularly favorable for
ready adaptation of computational properties to context and task. This
reverberating regime enables cortical networks to interpolate between the
asynchronous-irregular and the critical state by small changes in effective
synaptic strength or excitation-inhibition ratio. These changes directly adapt
computational properties, including sensitivity, amplification, integration
time and correlation length within the local network. We review recent
converging evidence that cortex in vivo operates in the reverberating regime,
and that various cortical areas have adapted their integration times to
processing requirements. In addition, we propose that neuromodulation enables a
fine-tuning of the network, so that local circuits can either decorrelate or
integrate, and quench or maintain their input depending on task. We argue that
this task-dependent tuning, which we call "dynamic adaptive computation",
presents a central organization principle of cortical networks and discuss
first experimental evidence.Comment: 6 pages + references, 2 figure
Complexity and Information: Measuring Emergence, Self-organization, and Homeostasis at Multiple Scales
Concepts used in the scientific study of complex systems have become so
widespread that their use and abuse has led to ambiguity and confusion in their
meaning. In this paper we use information theory to provide abstract and
concise measures of complexity, emergence, self-organization, and homeostasis.
The purpose is to clarify the meaning of these concepts with the aid of the
proposed formal measures. In a simplified version of the measures (focusing on
the information produced by a system), emergence becomes the opposite of
self-organization, while complexity represents their balance. Homeostasis can
be seen as a measure of the stability of the system. We use computational
experiments on random Boolean networks and elementary cellular automata to
illustrate our measures at multiple scales.Comment: 42 pages, 11 figures, 2 table
Multiplex visibility graphs to investigate recurrent neural network dynamics
Source at https://doi.org/10.1038/srep44037 .A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods
Bits from Biology for Computational Intelligence
Computational intelligence is broadly defined as biologically-inspired
computing. Usually, inspiration is drawn from neural systems. This article
shows how to analyze neural systems using information theory to obtain
constraints that help identify the algorithms run by such systems and the
information they represent. Algorithms and representations identified
information-theoretically may then guide the design of biologically inspired
computing systems (BICS). The material covered includes the necessary
introduction to information theory and the estimation of information theoretic
quantities from neural data. We then show how to analyze the information
encoded in a system about its environment, and also discuss recent
methodological developments on the question of how much information each agent
carries about the environment either uniquely, or redundantly or
synergistically together with others. Last, we introduce the framework of local
information dynamics, where information processing is decomposed into component
processes of information storage, transfer, and modification -- locally in
space and time. We close by discussing example applications of these measures
to neural data and other complex systems