Design of continuous-time recurrent neural networks with piecewise-linear activation function for generation of prescribed sequences of bipolar vectors

A recurrent neural network (RNN) can generate a sequence of patterns as the temporal evolution of the output vector. This paper focuses on a continuous-time RNN model with a piecewise-linear activation function that has neither external inputs nor hidden neurons, and studies the problem of finding the parameters of the model so that it generates a given sequence of bipolar vectors. First, a sufficient condition for the model to generate the desired sequence is derived, which is expressed as a system of linear inequalities in the parameters. Next, three approaches to finding solutions of the system of linear inequalities are proposed: One is formulated as a convex quadratic programming problem and others are linear programming problems. Then, two types of sequences of bipolar vectors that can be generated by the model are presented. Finally, the case where the model generates a periodic sequence of bipolar vectors is considered, and a sufficient condition for the trajectory of the state vector to converge to a limit cycle is provided

Sample Path Analysis of Integrate-and-Fire Neurons

Computational neuroscience is concerned with answering two intertwined questions that are based on the assumption that spatio-temporal patterns of spikes form the universal language of the nervous system. First, what function does a specific neural circuitry perform in the elaboration of a behavior? Second, how do neural circuits process behaviorally-relevant information? Non-linear system analysis has proven instrumental in understanding the coding strategies of early neural processing in various sensory modalities. Yet, at higher levels of integration, it fails to help in deciphering the response of assemblies of neurons to complex naturalistic stimuli. If neural activity can be assumed to be primarily driven by the stimulus at early stages of processing, the intrinsic activity of neural circuits interacts with their high-dimensional input to transform it in a stochastic non-linear fashion at the cortical level. As a consequence, any attempt to fully understand the brain through a system analysis approach becomes illusory. However, it is increasingly advocated that neural noise plays a constructive role in neural processing, facilitating information transmission. This prompts to gain insight into the neural code by studying the stochasticity of neuronal activity, which is viewed as biologically relevant. Such an endeavor requires the design of guiding theoretical principles to assess the potential benefits of neural noise. In this context, meeting the requirements of biological relevance and computational tractability, while providing a stochastic description of neural activity, prescribes the adoption of the integrate-and-fire model. In this thesis, founding ourselves on the path-wise description of neuronal activity, we propose to further the stochastic analysis of the integrate-and fire model through a combination of numerical and theoretical techniques. To begin, we expand upon the path-wise construction of linear diffusions, which offers a natural setting to describe leaky integrate-and-fire neurons, as inhomogeneous Markov chains. Based on the theoretical analysis of the first-passage problem, we then explore the interplay between the internal neuronal noise and the statistics of injected perturbations at the single unit level, and examine its implications on the neural coding. At the population level, we also develop an exact event-driven implementation of a Markov network of perfect integrate-and-fire neurons with both time delayed instantaneous interactions and arbitrary topology. We hope our approach will provide new paradigms to understand how sensory inputs perturb neural intrinsic activity and accomplish the goal of developing a new technique for identifying relevant patterns of population activity. From a perturbative perspective, our study shows how injecting frozen noise in different flavors can help characterize internal neuronal noise, which is presumably functionally relevant to information processing. From a simulation perspective, our event-driven framework is amenable to scrutinize the stochastic behavior of simple recurrent motifs as well as temporal dynamics of large scale networks under spike-timing-dependent plasticity

Modeling diversity by strange attractors with application to temporal pattern recognition

This thesis belongs to the general discipline of establishing black-box models from real-word data, more precisely, from measured time-series. This is an old subject and a large amount of papers and books has been written about it. The main difficulty is to express the diversity of data that has essentially the same origin without creating confusion with data that has a different origin. Normally, the diversity of time-series is modeled by a stochastic process, such as filtered white noise. Often, it is reasonable to assume that the time series is generated by a deterministic dynamical system rather than a stochastic process. In this case, the diversity of the data is expressed by the variability of the parameters of the dynamical system. The parameter variability itself is then, once again, modeled by a stochastic process. In both cases the diversity is generated by some form of exogenous noise. In this thesis a further step has been taken. A single chaotic dynamical system is used to model the data and their diversity. Indeed, a chaotic system produces a whole family of trajectories that are different but nonetheless very similar. It is believed that chaotic dynamics not only are a convenient means to represent diversity but that in many cases the origin of diversity stems actually from chaotic dynamic. Since the approach of this thesis explores completely new grounds the most suitable kind of data is considered, namely approximately periodic signals. In nature such time-series are rather common, in particular the physiological signal of living beings, such as the electrocardiograms (ECG), parts of speech signals, electroencephalograms (EEG), etc. Since there are strong arguments in favor of the chaotic nature of these signals, they appear to be the best candidates for modeling diversity by chaos. It should be stressed however, that the modeling approach pursued in this thesis is thought to be quite general and not limited to signals produced by chaotic dynamics in nature. The intended application of the modeling effort in this thesis is temporal signal classification. The reason for this is twofold. Firstly, classification is one of the basic building block of any cognitive system. Secondly, the recently studied phenomenon of synchronization of chaotic systems suggests a way to test a signal against its chaotic model. The essential content of this work can now be formulated as follows. Thesis: The diversity of approximately periodic signals found in nature can be modeled by means of chaotic dynamics. This kind of modeling technique, together with selective properties of the synchronization of chaotic systems, can be exploited for pattern recognition purposes. This Thesis is advocated by means of the following five points. Models of randomness (Chapter 2) It is argued that the randomness observed in nature is not necessarily the result of exogenous noise, but it could be endogenally generated by deterministic chaotic dynamics. The diversity of real signals is compared with signals produced by the most common chaotic systems. Qualitative resonance (Chapter 3) The behavior of chaotic systems forced by periodic or approximately periodic input signals is studied theoretically and by numerical simulation. It is observed that the chaotic system "locks" approximately to an input signal that is related to its internal chaotic dynamic. In contrast to this, its chaotic behavior is reinforced when the input signal has nothing to do with its internal dynamics. This new phenomenon is called "qualitative resonance". Modeling and recognizing (Chapter 4) In this chapter qualitative resonance is used for pattern recognition. The core of the method is a chaotic dynamical system that is able to reproduce the class of time-series that is to be recognized. This model is excited in a suitable way by an input signal such that qualitative resonance is realized. This means that if the input signal belongs to the modeled class of time-series, the system approximately "locks" into it. If not, the trajectory of the system and the input signal remain unrelated. Automated design of the recognizer (Chapters 5 and 6) For the kind of signals considered in this thesis a systematic design method of the recognizer is presented. The model used is a system of Lur'e type, i.e. a model where the linear dynamic and nonlinear static part are separated. The identification of the model parameters from the given data proceed iteratively, adapting in turn the linear and the nonlinear part. Thus, the difficult nonlinear dynamical system identification task is decomposed into the easier problems of linear dynamical and nonlinear static system identification. The way to apply the approximately periodic input signal in order to realize qualitative resonance is chosen with the help of periodic control theory. Validation (Chapter 7) The pattern recognition method has been validated on the following examples — A synthetic example — Laboratory measurement from Colpitts oscillator — ECG — EEG — Vowels of a speech signals In the first four cases a binary classification and in the last example a classification with five classes was performed. To the best of the knowledge of the author the recognition method is original. Chaotic systems have been already used to produce pseudo-noise and to model signal diversity. Also, parameter identification of chaotic systems has been already carried out. However, the direct establishment of the model from the given data and its subsequent use for classification based on the phenomenon of qualitative resonance is entirely new

Chaotic price dynamics of agricultural commodities

Traditionally, commodity prices have been analyzed and modeled in the context of linear generating processes. The purpose of this dissertation is to address the adequacy of this work through examination of the critical assumption of independence in the residual process of linearly specified models. As an alternative, a test procedure is developed and utilized to demonstrate the appropriateness of applying generalized conditional heteroscedastic time series models (GARCH) to agricultural commodity prices. In addition, a distinction is made between testing for independence and testing for chaos in commodity prices. The price series of interest derive from the major international agricultural commodity markets, sampled monthly over the period 1960--1994. The results of the present analysis suggest that for bananas, beef, coffee, soybeans, wool and wheat seasonally adjusted growth rates, ARCH-GARCH models account for some of the non-linear dependence in these commodity price series. As an alternative to the ARCH-GARCH models, several neural network models were estimated and in some cases outperformed the ARCH family of models in terms of forecast ability. This further demonstrated the nonlinearity present in these time series. Although, further examination is needed, all prices were found to be non-linearly dependent. It was determined by use of different statistical measures for testing for deterministic chaos that wheat prices may be an example of such behavior. Therefore, their may be something to be gained in terms of short-run forecast accuracy by using semi-parametric modeling approaches as applied to wheat prices

Stochastic switching in biology: from genotype to phenotype

There has been a resurgence of interest in non-equilibrium stochastic processes in recent years, driven in part by the observation that the number of molecules (genes, mRNA, proteins) involved in gene expression are often of order 1–1000. This means that deterministic mass-action kinetics tends to break down, and one needs to take into account the discrete, stochastic nature of biochemical reactions. One of the major consequences of molecular noise is the occurrence of stochastic biological switching at both the genotypic and phenotypic levels. For example, individual gene regulatory networks can switch between graded and binary responses, exhibit translational/transcriptional bursting, and support metastability (noise-induced switching between states that are stable in the deterministic limit). If random switching persists at the phenotypic level then this can confer certain advantages to cell populations growing in a changing environment, as exemplified by bacterial persistence in response to antibiotics. Gene expression at the single-cell level can also be regulated by changes in cell density at the population level, a process known as quorum sensing. In contrast to noise-driven phenotypic switching, the switching mechanism in quorum sensing is stimulus-driven and thus noise tends to have a detrimental effect. A common approach to modeling stochastic gene expression is to assume a large but finite system and to approximate the discrete processes by continuous processes using a system-size expansion. However, there is a growing need to have some familiarity with the theory of stochastic processes that goes beyond the standard topics of chemical master equations, the system-size expansion, Langevin equations and the Fokker–Planck equation. Examples include stochastic hybrid systems (piecewise deterministic Markov processes), large deviations and the Wentzel–Kramers–Brillouin (WKB) method, adiabatic reductions, and queuing/renewal theory. The major aim of this review is to provide a self-contained survey of these mathematical methods, mainly within the context of biological switching processes at both the genotypic and phenotypic levels. However, applications to other examples of biological switching are also discussed, including stochastic ion channels, diffusion in randomly switching environments, bacterial chemotaxis, and stochastic neural networks

Networked Dynamical Systems: Privacy, Control, and Cognition

Many natural and man-made systems, ranging from thenervous system to power and transportation grids to societies, exhibitdynamic behaviors that evolve over a sparse and complex network. This networked aspect raises significant challenges and opportunities for the identification, analysis, and control of such dynamic behaviors. While some of these challenges emanate from the networked aspect \emph{per se} (such as the sparsity of connections between system components and the interplay between nodal \emph{communication} and network dynamics), various challenges arise from the specific application areas (such as privacy concerns in cyber-physical systems or the need for \emph{scalable} algorithm designs due to the large size of various biological and engineered networks). On the other hand, networked systems provide significant opportunities and allow for performance and robustness levels that are far beyond reach for centralized systems, with examples ranging from the Internet (of Things) to the smart grid and the brain. This dissertation aims to address several of these challenges and harness these opportunities. The dissertation is divided into three parts. In the first part, we study privacy concerns whose resolution is vital for the utility of networked cyber-physical systems. We study the problems of average consensus and convex optimization as two principal distributed computations occurring over networks and design algorithm with rigorous privacy guarantees that provide a \emph{best achievable} tradeoff between network utility and privacy. In the second part, we analyze networks with resource constraints. More specifically, we study three problems of stabilization under communication (bandwidth and latency) limitations in sensing and actuation, optimal time-varying control scheduling problem under limited number of actuators and control energy, and the structure identification problem of under-sensed networks (i.e., networks with latent nodes). Finally in the last part, we focus on the intersection of networked dynamical systems and neuroscience and draw connections between brain network dynamics and two extensively studied but yet not fully understood neuro-cognitive phenomena: goal-driven selective attention and neural oscillations. Using a novel axiomatic approach, we establish these connections in the form of necessary and/or sufficient conditions on the network structure that match the network output trajectories with experimentally observed brain activity