Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Among the various architectures of Recurrent Neural Networks, Echo State Networks (ESNs) emerged due to their simplified and inexpensive training procedure. These networks are known to be sensitive to the setting of hyper-parameters, which critically affect their behaviour. Results show that their performance is usually maximized in a narrow region of hyper-parameter space called edge of chaos. Finding such a region requires searching in hyper-parameter space in a sensible way: hyper-parameter configurations marginally outside such a region might yield networks exhibiting fully developed chaos, hence producing unreliable computations. The performance gain due to optimizing hyper-parameters can be studied by considering the memory--nonlinearity trade-off, i.e., the fact that increasing the nonlinear behavior of the network degrades its ability to remember past inputs, and vice-versa. In this paper, we propose a model of ESNs that eliminates critical dependence on hyper-parameters, resulting in networks that provably cannot enter a chaotic regime and, at the same time, denotes nonlinear behaviour in phase space characterised by a large memory of past inputs, comparable to the one of linear networks. Our contribution is supported by experiments corroborating our theoretical findings, showing that the proposed model displays dynamics that are rich-enough to approximate many common nonlinear systems used for benchmarking

Edge of chaos of the classical kicked top map: Sensitivity to initial conditions

We focus on the frontier between the chaotic and regular regions for the classical version of the quantum kicked top. We show that the sensitivity to the initial conditions is numerically well characterised by $\xi=e_q^{\lambda_q t}$, where $e_{q}^{x}\equiv [ 1+(1-q) x]^{\frac{1}{1-q}} (e_1^x=e^x)$, and $\lambda_q$ is the $q$-generalization of the Lyapunov coefficient, a result that is consistent with nonextensive statistical mechanics, based on the entropy $S_q=(1- \sum_ip_i^q)/(q-1) (S_1 =-\sum_i p_i \ln p_i$). Our analysis shows that $q$ monotonically increases from zero to unity when the kicked-top perturbation parameter $\alpha$ increases from zero (unperturbed top) to $\alpha_c$, where $\alpha_c \simeq 3.2$. The entropic index $q$ remains equal to unity for $\alpha \ge \alpha_c$, parameter values for which the phase space is fully chaotic.Comment: To appear in "Complexity, Metastability and Nonextensivity" (World Scientific, Singapore, 2005), Eds. C. Beck, A. Rapisarda and C. Tsalli

The Simplicial Characterisation of TS networks: Theory and applications

We use the visibility algorithm to construct the time series networks obtained from the time series of different dynamical regimes of the logistic map. We define the simplicial characterisers of networks which can analyse the simplicial structure at both the global and local levels. These characterisers are used to analyse the TS networks obtained in different dynamical regimes of the logisitic map. It is seen that the simplicial characterisers are able to distinguish between distinct dynamical regimes. We also apply the simplicial characterisers to time series networks constructed from fMRI data, where the preliminary results indicate that the characterisers are able to differentiate between distinct TS networks.Comment: 11 pages, 2 figures, 4 tables. Accepted for publication in Proceedings of the 4th International Conference on Applications in Nonlinear Dynamics (ICAND 2016

Nonextensivity at the edge of chaos of a new universality class of one-dimensional unimodal dissipative maps

We introduce a new universality class of one-dimensional unimodal dissipative maps. The new family, from now on referred to as the ($z_1,z_2$)-{\it logarithmic map}, corresponds to a generalization of the $z$-logistic map. The Feigenbaum-like constants of these maps are determined. It has been recently shown that the probability density of sums of iterates at the edge of chaos of the $z$-logistic map is numerically consistent with a $q$-Gaussian, the distribution which, under appropriate constraints, optimizes the nonadditive entropy $S_q$. We focus here on the presently generalized maps to check whether they constitute a new universality class with regard to $q$-Gaussian attractor distributions. We also study the generalized $q$-entropy production per unit time on the new unimodal dissipative maps, both for strong and weak chaotic cases. The $q$-sensitivity indices are obtained as well. Our results are, like those for the $z$-logistic maps, numerically compatible with the $q$-generalization of a Pesin-like identity for ensemble averages.Comment: 17 pages, 10 figures. To appear in European Physical Journal

Herding as a Learning System with Edge-of-Chaos Dynamics

Herding defines a deterministic dynamical system at the edge of chaos. It generates a sequence of model states and parameters by alternating parameter perturbations with state maximizations, where the sequence of states can be interpreted as "samples" from an associated MRF model. Herding differs from maximum likelihood estimation in that the sequence of parameters does not converge to a fixed point and differs from an MCMC posterior sampling approach in that the sequence of states is generated deterministically. Herding may be interpreted as a"perturb and map" method where the parameter perturbations are generated using a deterministic nonlinear dynamical system rather than randomly from a Gumbel distribution. This chapter studies the distinct statistical characteristics of the herding algorithm and shows that the fast convergence rate of the controlled moments may be attributed to edge of chaos dynamics. The herding algorithm can also be generalized to models with latent variables and to a discriminative learning setting. The perceptron cycling theorem ensures that the fast moment matching property is preserved in the more general framework

Symptoms of complexity in a tourism system

Tourism destinations behave as dynamic evolving complex systems, encompassing numerous factors and activities which are interdependent and whose relationships might be highly nonlinear. Traditional research in this field has looked after a linear approach: variables and relationships are monitored in order to forecast future outcomes with simplified models and to derive implications for management organisations. The limitations of this approach have become apparent in many cases, and several authors claim for a new and different attitude. While complex systems ideas are amongst the most promising interdisciplinary research themes emerged in the last few decades, very little has been done so far in the field of tourism. This paper presents a brief overview of the complexity framework as a means to understand structures, characteristics, relationships, and explores the implications and contributions of the complexity literature on tourism systems. The objective is to allow the reader to gain a deeper appreciation of this point of view.Comment: 32 pages, 3 figures, 1 table; accepted in Tourism Analysi

Topological and Dynamical Complexity of Random Neural Networks

Random neural networks are dynamical descriptions of randomly interconnected neural units. These show a phase transition to chaos as a disorder parameter is increased. The microscopic mechanisms underlying this phase transition are unknown, and similarly to spin-glasses, shall be fundamentally related to the behavior of the system. In this Letter we investigate the explosion of complexity arising near that phase transition. We show that the mean number of equilibria undergoes a sharp transition from one equilibrium to a very large number scaling exponentially with the dimension on the system. Near criticality, we compute the exponential rate of divergence, called topological complexity. Strikingly, we show that it behaves exactly as the maximal Lyapunov exponent, a classical measure of dynamical complexity. This relationship unravels a microscopic mechanism leading to chaos which we further demonstrate on a simpler class of disordered systems, suggesting a deep and underexplored link between topological and dynamical complexity

Chaos and Order in Nature/Creation: A Reading of Genesis l-2:4a in Dialogue with Science and Philosophy

With inspiration from post-modern scientific theories (complexity theory, chaos theory, relativity theory, uncertainty theory, no-singularity/boundary theory), and from philosophical understandings of nature (ecstatic naturalism and Taoism), the author offers an innovative reading of the Genesis creation stories, focusing on the concepts of order and chaos. While criticizing the dichotomous dualism that underpins the human ordering system, she connects these rich meanings and wisdom signified by nature with theological discourse through a discussion of the infinity of God, the abjection of origin, the autonomy of creatures, and nature's complex and fluid manifestations.ye

How does flow in a pipe become turbulent?

The transition to turbulence in pipe flow does not follow the scenario familiar from Rayleigh-Benard or Taylor-Couette flow since the laminar profile is stable against infinitesimal perturbations for all Reynolds numbers. Moreover, even when the flow speed is high enough and the perturbation sufficiently strong such that turbulent flow is established, it can return to the laminar state without any indication of the imminent decay. In this parameter range, the lifetimes of perturbations show a sensitive dependence on initial conditions and an exponential distribution. The turbulence seems to be supported by three-dimensional travelling waves which appear transiently in the flow field. The boundary between laminar and turbulent dynamics is formed by the stable manifold of an invariant chaotic state. We will also discuss the relation between observations in short, periodically continued domains, and the dynamics in fully extended puffs.Comment: for the proceedings of statphys 2