Entropy in Dynamic Systems

In order to measure and quantify the complex behavior of real-world systems, either novel mathematical approaches or modifications of classical ones are required to precisely predict, monitor, and control complicated chaotic and stochastic processes. Though the term of entropy comes from Greek and emphasizes its analogy to energy, today, it has wandered to different branches of pure and applied sciences and is understood in a rather rough way, with emphasis placed on the transition from regular to chaotic states, stochastic and deterministic disorder, and uniform and non-uniform distribution or decay of diversity. This collection of papers addresses the notion of entropy in a very broad sense. The presented manuscripts follow from different branches of mathematical/physical sciences, natural/social sciences, and engineering-oriented sciences with emphasis placed on the complexity of dynamical systems. Topics like timing chaos and spatiotemporal chaos, bifurcation, synchronization and anti-synchronization, stability, lumped mass and continuous mechanical systems modeling, novel nonlinear phenomena, and resonances are discussed

Interpreting recurrent neural networks behaviour via excitable network attractors

Introduction: Machine learning provides fundamental tools both for scientific research and for the development of technologies with significant impact on society. It provides methods that facilitate the discovery of regularities in data and that give predictions without explicit knowledge of the rules governing a system. However, a price is paid for exploiting such flexibility: machine learning methods are typically black-boxes where it is difficult to fully understand what the machine is doing or how it is operating. This poses constraints on the applicability and explainability of such methods. Methods: Our research aims to open the black-box of recurrent neural networks, an important family of neural networks used for processing sequential data. We propose a novel methodology that provides a mechanistic interpretation of behaviour when solving a computational task. Our methodology uses mathematical constructs called excitable network attractors, which are invariant sets in phase space composed of stable attractors and excitable connections between them. Results and Discussion: As the behaviour of recurrent neural networks depends both on training and on inputs to the system, we introduce an algorithm to extract network attractors directly from the trajectory of a neural network while solving tasks. Simulations conducted on a controlled benchmark task confirm the relevance of these attractors for interpreting the behaviour of recurrent neural networks, at least for tasks that involve learning a finite number of stable states and transitions between them.Comment: revised versio

Coupling functions:universal insights into dynamical interaction mechanisms

The dynamical systems found in Nature are rarely isolated. Instead they interact and influence each other. The coupling functions that connect them contain detailed information about the functional mechanisms underlying the interactions and prescribe the physical rule specifying how an interaction occurs. Here, we aim to present a coherent and comprehensive review encompassing the rapid progress made recently in the analysis, understanding and applications of coupling functions. The basic concepts and characteristics of coupling functions are presented through demonstrative examples of different domains, revealing the mechanisms and emphasizing their multivariate nature. The theory of coupling functions is discussed through gradually increasing complexity from strong and weak interactions to globally-coupled systems and networks. A variety of methods that have been developed for the detection and reconstruction of coupling functions from measured data is described. These methods are based on different statistical techniques for dynamical inference. Stemming from physics, such methods are being applied in diverse areas of science and technology, including chemistry, biology, physiology, neuroscience, social sciences, mechanics and secure communications. This breadth of application illustrates the universality of coupling functions for studying the interaction mechanisms of coupled dynamical systems

Interpreting multi-stable behaviour in input-driven recurrent neural networks

Recurrent neural networks (RNNs) are computational models inspired by the brain. Although RNNs stand out as state-of-the-art machine learning models to solve challenging tasks as speech recognition, handwriting recognition, language translation, and others, they are plagued by the so-called vanishing/exploding gradient issue. This prevents us from training RNNs with the aim of learning long term dependencies in sequential data. Moreover, a problem of interpretability affects these models, known as the ``black-box issue'' of RNNs. We attempt to open the black box by developing a mechanistic interpretation of errors occurring during the computation. We do this from a dynamical system theory perspective, specifically building on the notion of Excitable Network Attractors. Our methodology is effective at least for those tasks where a number of attractors and a switching pattern between them must be learned. RNNs can be seen as massively large nonlinear dynamical systems driven by external inputs. When it comes to analytically investigate RNNs, often in the literature the input-driven property is neglected or dropped in favour of tight constraints on the input driving the dynamics, which do not match the reality of RNN applications. Trying to bridge this gap, we framed RNNs dynamics driven by generic input sequences in the context of nonautonomous dynamical system theory. This brought us to enquire deeply into a fundamental principle established for RNNs known as the echo state property (ESP). In particular, we argue that input-driven RNNs can be reliable computational models even without satisfying the classical ESP formulation. We prove a sort of input-driven fixed point theorem and exploit it to (i) demonstrate the existence and uniqueness of a global attracting solution for strongly (in amplitude) input-driven RNNs, (ii) deduce the existence of multiple responses for certain input signals which can be reliably exploited for computational purposes, and (iii) study the stability of attracting solutions w.r.t. input sequences. Finally, we highlight the active role of the input in determining qualitative changes in the RNN dynamics, e.g. the number of stable responses, in contrast to commonly known qualitative changes due to variations of model parameters

Stochastic resonance in chua's circuit driven by alpha-stable noise

Thesis (Master)--Izmir Institute of Technology, Electronics and Communication Engineering, Izmir, 2012Includes bibliographical references (leaves: 75-80)Text in English; Abstract: Turkish and Englishx, 80 leavesThe main aim of this thesis is to investigate the stochastic resonance (SR) in Chua's circuit driven by alpha-stable noise which has better approximation to a real-world signal than Gaussian distribution. SR is a phenomenon in which the response of a nonlinear system to a sub-threshold (weak) input signal is enhanced with the addition of an optimal amount of noise. There have been an increasing amount of applications based on SR in various fields. Almost all studies related to SR in chaotic systems assume that the noise is Gaussian, which leads researchers to investigate the cases in which the noise is non-Gaussian hence has infinite variance. In this thesis, the spectral power amplification which is used to quantify the SR has been evaluated through fractional lower order Wigner Ville distribution of the response of a system and analyzed for various parameters of alpha-stable noise. The results provide a visible SR effect in Chua’s circuit driven by symmetric and skewed-symmetric alpha-stable noise distributions. Furthermore, a series of simulations reveal that the mean residence time that is the average time spent by the trajectory in an attractor can vary depending on different alpha-stable noise parameters

The physics of climate variability and climate change

The climate system is a forced, dissipative, nonlinear, complex and heterogeneous system that is out of thermodynamic equilibrium. The system exhibits natural variability on many scales of motion, in time as well as space, and it is subject to various external forcings, natural as well as anthropogenic. This paper reviews the observational evidence on climate phenomena and the governing equations of planetary-scale flow, as well as presenting the key concept of a hierarchy of models as used in the climate sciences. Recent advances in the application of dynamical systems theory, on the one hand, and of nonequilibrium statistical physics, on the other, are brought together for the first time and shown to complement each other in helping understand and predict the system's behavior. These complementary points of view permit a self-consistent handling of subgrid-scale phenomena as stochastic processes, as well as a unified handling of natural climate variability and forced climate change, along with a treatment of the crucial issues of climate sensitivity, response, and predictability

Slow manifolds for stochastic koper models with stable Lévy noises

The Koper model is a vector field in which the differential equations describe the electrochemical oscillations appearing in diffusion processes. This work focuses on the understanding of the slow dynamics of a stochastic Koper model perturbed by stable Lévy noise. We establish the slow manifold for a stochastic Koper model with stable Lévy noise and verify exponential tracking properties. We also present two practical examples to demonstrate the analytical results with numerical simulations

Rate and Noise-Induced Tipping Working in Concert

Tipping is the rapid, and often irreversible, change in the state of a system. Rate-induced tipping occurswhen a ramp parameter changes rapidly enough to cause the system to tip between co-existing, attracting states, while noise-induced tipping occurs when there are random transitions between two attractors of the underlying deterministic system. This work builds theory for tipping events in low-dimensional dynamical systems with additive noise and time-dependent parameters, in which noise is not vanishingly small. The central question is understanding what information can be extracted from the theory of large deviations for noise levels outside the validity of the approach, where the guiding principles are geometric dynamical systems methods and Monte Carlo simulations. Both tipping mechanisms are first considered within a model of the oceanic carbon cycle, in which the key objective is understanding how the system tips from a stable fixed point to a stable periodic orbit. While rate-induced tipping away from the fixed point is straightforward, the noise-induced tipping is challenging due to a periodic orbit forming the basin boundary for tipping. Noisy trajectories will tend to cycle around the periodic orbit as the noise vanishes, but as the noise becomes slightly larger, the escaping paths become resistant to cycling. An interesting phenomena exposed is that a subset of the unstable manifold of the fixed point in the Euler-Lagrange system, with Maslov index zero, determines where the noisy trajectories escape. After considering tipping mechanisms individually, we consider a one-dimensional differential equation with both additive noise and a ramp parameter. The addition of noise to the system can cause it to tip well below the critical rate at which rate-induced tipping would occur. We achieve this by finding a global minimizer of the Freidlin-Wentzell functional of large deviation theory that represents the most probable path for tipping. This is realized as a heteroclinic connection for the Euler-Lagrange system associated with the Freidlin-Wentzell action and it exists for all rates less than or equal to the critical rate. This framework is extended to show the existence of a heteroclinic orbit for a fairly general class of functions.Doctor of Philosoph