Attraction Radii in Binary Hopfield Nets are Hard to Compute

Peer reviewe

Memories, attractors, space and vowels

Higher cognitive capacities, such as navigating complex environments or learning new languages, rely on the possibility to memorize, in the brain, continuous noisy variables. Memories are generally understood to be realized, e.g. in the cortex and in the hippocampus, as configurations of activity towards which specific populations of neurons are \u201cattracted\u201d, i.e towards which they dynamically converge, if properly cued. Distinct memories are thus considered as separate attractors of the dynamics, embedded within the same neuronal connectivity structure. But what if the underlying variables are continuous, such as a position in space or the resonant frequency of a phoneme? If such variables are continuous and the experience to be retained in memory has even a minimal temporal duration, highly correlated, yet imprecisely determined values of those variables will occur at successive time instants. And if memories are idealized as point-like in time, still distinct memories will be highly correlated. How does the brain self-organize to deal with noisy correlated memories? In this thesis, we try to approach the question along three interconnected itineraries. In Part II we first ask the opposite: we derive how many uncorrelated memories a network of neurons would be able to precisely store, as discrete attractors, if the neurons were optimally connected. Then, we compare the results with those obtained when memories are allowed to be retrieved imprecisely and connections are based on self-organization. We find that a simple strategy is available in the brain to facilitate the storage of memories: it amounts to making them more sparse, i.e. to silencing those neurons which are not very active in the configuration of activity to be memorized. We observe that the more the distribution of activity in the memory is complex, the more this strategy leads to store a higher number of memories, as compared with the maximal load in networks endowed with the theoretically optimal connection weights. In part III we ask, starting from experimental observations of spatially selective cells in quasi-realistic environments, how can the brain store, as a continuous attractor, complex and irregular spatial information. We find indications that while continuous attractors, per se, are too brittle to deal with irregularities, there seem to be other mathematical objects, which we refer to as quasi-attractive continuous manifolds, which may have this function. Such objects, which emerge as soon as a tiny amount of quenched irregularity is introduced in would-be continuous attractors, seem to persist over a wide range of noise levels and then break up, in a phase transition, when the variability reaches a critical threshold, lying just above that seen in the experimental measurements. Moreover, we find that the operational range is squeezed from behind, as it were, by a third phase, in which the spatially selective units cannot dynamically converge towards a localized state. Part IV, which is more exploratory, is motivated by the frequency characteristics of vowels. We hypothesize that also phonemes of different languages could be stored as separate fixed points in the brain through a sort of two-dimensional cognitive map. In our preliminary results, we show that a continuous quasi-attractor model, trained with noisy recorded vowels, can effectively learn them through a self-organized procedure and retrieve them separately, as fixed points on a quasi-attractive manifold. Overall, this thesis attempts to contribute to the search for general principles underlying memory, intended as an emergent collective property of networks in the brain, based on self-organization, imperfections and irregularities

Towards a continuous dynamic model of the Hopfield theory on neuronal interaction and memory storage

The purpose of this work is to study the Hopfield model for neuronal interaction and memory storage, in particular the convergence to the stored patterns. Since the hypothesis of symmetric synapses is not true for the brain, we will study how we can extend it to the case of asymmetric synapses using a probabilistic approach. We then focus on the description of another feature of the memory process and brain: oscillations. Using the Kuramoto model we will be able to describe them completely, gaining the presence of synchronization between neurons. Our aim is therefore to understand how and why neurons can be seen as oscillators and to establish a strong link between this model and the Hopfield approach

Biologically inspired evolutionary temporal neural circuits

Biological neural networks have always motivated creation of new artificial neural networks, and in this case a new autonomous temporal neural network system. Among the more challenging problems of temporal neural networks are the design and incorporation of short and long-term memories as well as the choice of network topology and training mechanism. In general, delayed copies of network signals can form short-term memory (STM), providing a limited temporal history of events similar to FIR filters, whereas the synaptic connection strengths as well as delayed feedback loops (ER circuits) can constitute longer-term memories (LTM). This dissertation introduces a new general evolutionary temporal neural network framework (GETnet) through automatic design of arbitrary neural networks with STM and LTM. GETnet is a step towards realization of general intelligent systems that need minimum or no human intervention and can be applied to a broad range of problems. GETnet utilizes nonlinear moving average/autoregressive nodes and sub-circuits that are trained by enhanced gradient descent and evolutionary search in terms of architecture, synaptic delay, and synaptic weight spaces. The mixture of Lamarckian and Darwinian evolutionary mechanisms facilitates the Baldwin effect and speeds up the hybrid training. The ability to evolve arbitrary adaptive time-delay connections enables GETnet to find novel answers to many classification and system identification tasks expressed in the general form of desired multidimensional input and output signals. Simulations using Mackey-Glass chaotic time series and fingerprint perspiration-induced temporal variations are given to demonstrate the above stated capabilities of GETnet

Complexity of fixed point counting problems in Boolean Networks

A Boolean network (BN) with $n$ components is a discrete dynamical system described by the successive iterations of a function $f:\{0,1\}^n \to \{0,1\}^n$. This model finds applications in biology, where fixed points play a central role. For example, in genetic regulations, they correspond to cell phenotypes. In this context, experiments reveal the existence of positive or negative influences among components: component $i$ has a positive (resp. negative) influence on component $j$ meaning that $j$ tends to mimic (resp. negate) $i$. The digraph of influences is called signed interaction digraph (SID), and one SID may correspond to a large number of BNs (which is, in average, doubly exponential according to $n$). The present work opens a new perspective on the well-established study of fixed points in BNs. When biologists discover the SID of a BN they do not know, they may ask: given that SID, can it correspond to a BN having at least/at most $k$ fixed points? Depending on the input, we prove that these problems are in $\textrm{P}$ or complete for $\textrm{NP}$, $\textrm{NP}^{\textrm{NP}}$, \textrm{NP}^{\textrm{#P}} or $\textrm{NEXPTIME}$. In particular, we prove that it is $\textrm{NP}$-complete (resp. $\textrm{NEXPTIME}$-complete) to decide if a given SID can correspond to a BN having at least two fixed points (resp. no fixed point).Comment: 43 page

Proceedings of AUTOMATA 2010: 16th International workshop on cellular automata and discrete complex systems

International audienceThese local proceedings hold the papers of two catgeories: (a) Short, non-reviewed papers (b) Full paper

Morphogenesis and Growth Driven by Selection of Dynamical Properties

Organisms are understood to be complex adaptive systems that evolved to thrive in hostile environments. Though widely studied, the phenomena of organism development and growth, and their relationship to organism dynamics is not well understood. Indeed, the large number of components, their interconnectivity, and complex system interactions all obscure our ability to see, describe, and understand the functioning of biological organisms. Here we take a synthetic and computational approach to the problem, abstracting the organism as a cellular automaton. Such systems are discrete digital models of real-world environments, making them more accessible and easier to study then their physical world counterparts. In such simplified synthetic models, we find that the structure of the cellular network greatly impacts the dynamics of the organism as a whole. In the physical world, for example, the network property wherein some cells depend on phosphorus produces the cyclical boom-bust dynamics of algae on the surface of a pond. Using techniques of synthetic biology and cellular automata, such local properties can be abstractly specified, and the long-term, system-wide, and dynamical consequences of localized assumptions can be carefully explored. This thesis explores the potential impacts of Darwinian selection of dynamical properties on long term cellular differentiation and organism growth. The focus here is on the relationship between organism homogeneity (or heterogeneity) and the dynamical properties of robustness, adaptivity, and chromatic symmetry. This dissertation applies an experimental approach to test the following three hypotheses: (1) cellular differentiation increases the expected robustness in an organism’s dynamics, (2) cellular differentiation leads to more uniform adaptivity as the organism grows, and (3) for organisms with symmetry, growth by segment elongation is more likely than growth by segment reduplication. To explore these hypotheses, we address several obstacles in the experimental study of dynamical systems, including computational time limits and big data

Exact diagonalization studies of quantum simulators

Understand and tame complex quantum mechanical systems to build quantum technologies is one of the most important scientific endeavour nowadays. In this effort, Atomic, molecular and Optical systems have clearly played a major role in producing proofs of concept of several important applications. Notable examples are Quantum Simulators for difficult problems in other branches of physics i.e. spin systems, disordered systems, etc., and small sized Quantum Computers. In particular, ultracold atomic gases and trapped ion experiments are nowadays at the forefront in the field. This fantastic experimental effort needs to be accompanied by a matching theoretical and numerical one. The main two reasons are: 1) theoretical work is needed to identify suitable regimes where the AMO systems can be used as efficient quantum simulators of important problems in physics and mathematics, 2) thorough numerical work is needed to benchmark the results of the experiments in parameter regions where a solution to the problem can be found with classical devices. In this dissertation, we present several important examples of systems, which can be numerically solved. The technique used, which is common to all the work presented in the dissertation, is exact diagonalization. This technique works solely for systems of a small number of particles and/or a small number of available quantum states. Despite this limitation, one can study a large variety of quantum systems in relevant parameter regimes. A notable advantage is that it allows one to compute not only the ground state of the system but also most of the spectrum and, in some cases, to study dynamics. The dissertation is organized in the following way. First, we provide an introduction, outlining the importance of this technique for quantum simulation and quantum validation and certification. In Chapter 2, we detail the exact diagonalization technique and present an example of use for the phases of the 1D Bose-Hubbard chain. Then in Chapters 3 to 6, we present a number of important uses of exact diagonalization. In Chapter 3, we study the quantum Hall phases, which are found in two-component bosons subjected to artificial gauge fields. In Chapter 4, we turn into dynamical gauge fields, presenting the topological phases which appear in a bosonic system trapped in a small lattice. In Chapter 5, a very different problem is tackled, that of using an ultracold atomic gases to simulate a spin model. Quantum simulation is again the goal of Chapter 6, where we propose a way in which the number-partitioning problem can be solved by means of a quantum simulator made with trapped ions. Finally, in Chapter 7, we collect the main conclusions of the dissertation and provide a brief outlook.Entendre i controlar sistemes complexos regits per la mecànica quàntica per a construir tecnologies quàntiques es un dels reptes mes rellevants de la ciència en l’actualitat. Els sistemes atòmics, moleculars i òptics han jugat clarament un rol capital en aquest esforç, produint proves de concepte per a diverses aplicacions de consideració. Exemples notables en son els simuladors quàntics dissenyats per a resoldre problemes complicats d’altres branques de la física, com ara sistemes d’espins, sistemes desordenats, etc.... i ordinadors quàntics de dimensions reduïdes. En particular, els experiments amb gasos d’àtoms ultrafreds i amb trampes iòniques son la punta de llança del camp en l’actualitat. El fantàstic afany experimental ha d’anar associat amb d’altres teòric i numèric que el corresponguin. Les raons principals son: 1) els estudis teòrics son necessaris per tal d’identificar règims adients en que els sistemes AMO puguin esser emprats com a simuladors quàntics eficients de problemes rellevants de la Física i les Matemàtiques, 2) els treballs numèrics exhaustius son necessaris per a contrastar els resultats dels experiments en regions de paràmetres en que els dispositius clàssics son capaços de trobar solucions. En aquesta tesi, presentem diversos exemples de sistemes rellevants que poden esser resolts numèricament. La tècnica emprada -que es comuna per a tot el treball- es la diagonalització exacta. L’ús d’aquesta tècnica es limitat a sistemes amb nombres baixos partícules i/o pocs estats quàntics accessibles. Malgrat aquesta limitació, es poden estudiar una gran varietat de sistemes quàntics en els règims rellevants dels paràmetres de control. Un avantatge notable es el fet que permet calcular no nomes l’estat de mínima energia del sistema, sinó que també la majoria de l’espectre i, en alguns casos, àdhuc estudiar-ne la dinàmica. La tesi s’organitza tal i com prossegueix. En primer lloc, proveïm una introducció, subratllant la importància d’aquesta tècnica per a la simulació quàntica i la validació quàntica i certificació. En el capítol 2, detallem la tècnica de la diagonalització exacta i presentem un exemple del seu us per a les fases per a una cadena de Bose-Hubbard unidimensional. En els capítols del 3 al 6, presentem alguns usos rellevants de la diagonalització exacta. En el capítol 3, estudiem les fases degudes a l’efecte Hall quàntic en un sistema de dues components de bosons sotmesos a camps de gauge artificials. En el capítol 4, canviem a camps de gauge dinàmics, presentant les fases topològiques que apareixen en un sistema de bosons atrapats en una petita xarxa reticular. En el capítol 5, s’hi tracta un problema ben diferent, el d’emprar gasos d’àtoms ultrafreds per a per a simular un model d’espín. La simulació quàntica es de nou l’objectiu del capítol 6, en que proposem una forma en que el problema de la partició de nombres pot esser resolt per mitja d’un simulador quàntic construït amb trampes iòniques. Finalment, en el capítol 7, recollim les conclusions principals del treball i donem una breu opinió del futur d’aquesta investigació.Entender y controlar sistemas complejos regidos por la mecánica cuántica para construir tecnologías cuánticas es una de los retos científicos más relevantes en la actualidad. Los sistemas atómicos, moleculares y ópticos han jugado claramente un rol capital en este esfuerzo, produciendo pruebas de concepto para diversas aplicaciones de consideración. Notables ejemplos son los simuladores cuánticos diseñados para resolver problemas complicados de otras ramas de la física, como lo son los sistemas de espines, sistemas desordenados, etc.. . . i los ordenadores cuánticos de dimensiones reducidas. En particular, los experimentos con gases de átomos ultrafríos y con trampas iónicas son la punta de lanza del campo en la actualidad. El fantástico empeño experimental tiene que ir asociado a otros teórico y numérico que le correspondan. Las principales razones son: 1) los estudios teóricos son necesarios para identificar regímenes adecuados en que los sistemas AMO puedan ser usados cómo simuladores cuánticos eficientes para problemas relevantes de la Física y las Matemáticas, 2) los trabajos numéricos exhaustivos son necesarios para contrastar los resultados de los experimentos en regiones de parámetros en que los dispositivos clásicos sean capaces de encontrar soluciones. En esta tesis, presentamos diferentes ejemplos de sistemas relevantes que pueden ser resueltos numéricamente. La técnica usada –que es común en todo el trabajo– es la diagonalización exacta. El uso de ésta técnica está restringido a sistemas con números bajos de partículas i/o estados cuánticos accesibles. A pesar de esta limitación, se puede estudiar gran variedad de sistemas cuánticos en los regímenes relevantes de los parámetros de control. Una ventaja notable es que permite calcular no sólo el estado de mínima energía del sistema, sino que también la mayoría del espectro e, en algunos casos, incluso estudiar la dinámica. La tesis se organiza como sigue. En primer lugar, ofrecemos una introducción, subrayando la importancia de esta técnica para la simulación cuántica y la validación cuántica y certificación. En el capítulo 2, detallamos la técnica de la diagonalización exacta y presentamos un ejemplo de su uso para una cadena de Bose-Hubbard unidimensional. En los capítulos del 3 al 6, presentamos algunos usos relevantes de la diagonalización exacta. En el capítulo 3, estudiamos las fases debidas al efecto Hall cuántico en un sistema de dos componentes de bosones sometidos a campos de gauge artificiales. En el capítulo 4, cambiamos hacia campos gauge dinámicos, presentando las fases topológicas que aparecen en un sistema de bosones atrapados en una pequeña malla reticular. En el capítulo 5, se trata un problema bien diferente, el de usar gases de átomos ultrafríos para simular un modelo de espín. La simulación cuántica es de nuevo el objetivo del capítulo 6, en que proponemos una forma en que el problema de la partición de números puede ser resuelta mediante un simulador cuántico construido con trampas iónicas. Finalmente, en el capítulo 7, recogemos las conclusiones principales de los trabajos y damos una breve opinión del futuro de ésta investigaciónPostprint (published version

UNCOVERING PATTERNS IN COMPLEX DATA WITH RESERVOIR COMPUTING AND NETWORK ANALYTICS: A DYNAMICAL SYSTEMS APPROACH

In this thesis, we explore methods of uncovering underlying patterns in complex data, and making predictions, through machine learning and network science. With the availability of more data, machine learning for data analysis has advanced rapidly. However, there is a general lack of approaches that might allow us to 'open the black box'. In the machine learning part of this thesis, we primarily use an architecture called Reservoir Computing for time-series prediction and image classification, while exploring how information is encoded in the reservoir dynamics. First, we investigate the ways in which a Reservoir Computer (RC) learns concepts such as 'similar' and 'different', and relationships such as 'blurring', 'rotation' etc. between image pairs, and generalizes these concepts to different classes unseen during training. We observe that the high dimensional reservoir dynamics display different patterns for different relationships. This clustering allows RCs to perform significantly better in generalization with limited training compared with state-of-the-art pair-based convolutional/deep Siamese Neural Networks. Second, we demonstrate the utility of an RC in the separation of superimposed chaotic signals. We assume no knowledge of the dynamical equations that produce the signals, and require only that the training data consist of finite time samples of the component signals. We find that our method significantly outperforms the optimal linear solution to the separation problem, the Wiener filter. To understand how representations of signals are encoded in an RC during learning, we study its dynamical properties when trained to predict chaotic Lorenz signals. We do so by using a novel, mathematical fixed-point-finding technique called directional fibers. We find that, after training, the high dimensional RC dynamics includes fixed points that map to the known Lorenz fixed points, but the RC also has spurious fixed points, which are relevant to how its predictions break down. While machine learning is a useful data processing tool, its success often relies on a useful representation of the system's information. In contrast, systems with a large numbers of interacting components may be better analyzed by modeling them as networks. While numerous advances in network science have helped us analyze such systems, tools that identify properties on networks modeling multi-variate time-evolving data (such as disease data) are limited. We close this gap by introducing a novel data-driven, network-based Trajectory Profile Clustering (TPC) algorithm for 1) identification of disease subtypes and 2) early prediction of subtype/disease progression patterns. TPC identifies subtypes by clustering patients with similar disease trajectory profiles derived from bipartite patient-variable networks. Applying TPC to a Parkinson’s dataset, we identify 3 distinct subtypes. Additionally, we show that TPC predicts disease subtype 4 years in advance with 74% accuracy