Neural Distributed Autoassociative Memories: A Survey

Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension. The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons). Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints. Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors.Comment: 31 page

Common Radio Resource Management Strategies for Quality of Service Support in Heterogeneous Wireless Networks

Hoy en día existen varias tecnologías que coexisten en una misma zona formando un sistema heterogéneo. Además, este hecho se espera que se vuelva más acentuado con todas las nuevas tecnologías que se están estandarizando actualmente. Hasta ahora, generalmente son los usuarios los que eligen la tecnología a la que se van a conectar, ya sea configurando sus terminales o usando terminales distintos. Sin embargo, esta solución es incapaz de aprovechar al máximo todos los recursos. Para ello es necesario un nuevo conjunto de estrategias. Estas estrategias deben gestionar los recursos radioeléctricos conjuntamente y asegurar la satisfacción de la calidad de servicio de los usuarios. Siguiendo esta idea, esta Tesis propone dos nuevos algoritmos. El primero es un algoritmo de asignación dinámica de recusos conjunto (JDRA) capaz de asignar recursos a usuarios y de distribuir usuarios entre tecnologías al mismo tiempo. El algoritmo está formulado en términos de un problema de optimización multi-objetivo que se resuelve usando redes neuronales de Hopfield (HNNs). Las HNNs son interesantes ya que se supone que pueden alcanzar soluciones sub-óptimas en cortos periodos de tiempo. Sin embargo, implementaciones reales de las HNNs en ordenadores pierden esta rápida respuesta. Por ello, en esta Tesis se analizan las causas y se estudian posibles mejoras. El segundo algoritmo es un algoritmo de control de admisión conjunto (JCAC) que admite y rechaza usuarios teniendo en cuenta todas las tecnologías al mismo tiempo. La principal diferencia con otros algorimos propuestos es que éstos últimos toman las dicisiones de admisión en cada tecnología por separado. Por ello, se necesita de algún mecanismo para seleccionar la tecnología a la que los usuarios se van a conectar. Por el contrario, la técnica propuesta en esta Tesis es capaz de tomar decisiones en todo el sistema heterogéneo. Por lo tanto, los usuarios no se enlazan con ninguna tecnología antes de ser admitidos.Calabuig Soler, D. (2010). Common Radio Resource Management Strategies for Quality of Service Support in Heterogeneous Wireless Networks [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/7348Palanci

Collective stability of networks of winner-take-all circuits

The neocortex has a remarkably uniform neuronal organization, suggesting that common principles of processing are employed throughout its extent. In particular, the patterns of connectivity observed in the superficial layers of the visual cortex are consistent with the recurrent excitation and inhibitory feedback required for cooperative-competitive circuits such as the soft winner-take-all (WTA). WTA circuits offer interesting computational properties such as selective amplification, signal restoration, and decision making. But, these properties depend on the signal gain derived from positive feedback, and so there is a critical trade-off between providing feedback strong enough to support the sophisticated computations, while maintaining overall circuit stability. We consider the question of how to reason about stability in very large distributed networks of such circuits. We approach this problem by approximating the regular cortical architecture as many interconnected cooperative-competitive modules. We demonstrate that by properly understanding the behavior of this small computational module, one can reason over the stability and convergence of very large networks composed of these modules. We obtain parameter ranges in which the WTA circuit operates in a high-gain regime, is stable, and can be aggregated arbitrarily to form large stable networks. We use nonlinear Contraction Theory to establish conditions for stability in the fully nonlinear case, and verify these solutions using numerical simulations. The derived bounds allow modes of operation in which the WTA network is multi-stable and exhibits state-dependent persistent activities. Our approach is sufficiently general to reason systematically about the stability of any network, biological or technological, composed of networks of small modules that express competition through shared inhibition.Comment: 7 Figure

Asymptotic behavior of memristive circuits

The interest in memristors has risen due to their possible application both as memory units and as computational devices in combination with CMOS. This is in part due to their nonlinear dynamics, and a strong dependence on the circuit topology. We provide evidence that also purely memristive circuits can be employed for computational purposes. In the present paper we show that a polynomial Lyapunov function in the memory parameters exists for the case of DC controlled memristors. Such Lyapunov function can be asymptotically approximated with binary variables, and mapped to quadratic combinatorial optimization problems. This also shows a direct parallel between memristive circuits and the Hopfield-Little model. In the case of Erdos-Renyi random circuits, we show numerically that the distribution of the matrix elements of the projectors can be roughly approximated with a Gaussian distribution, and that it scales with the inverse square root of the number of elements. This provides an approximated but direct connection with the physics of disordered system and, in particular, of mean field spin glasses. Using this and the fact that the interaction is controlled by a projector operator on the loop space of the circuit. We estimate the number of stationary points of the approximate Lyapunov function and provide a scaling formula as an upper bound in terms of the circuit topology only.Comment: 20 pages, 8 figures; proofs corrected, figures changed; results substantially unchanged; to appear in Entrop

Statistical Physics and Representations in Real and Artificial Neural Networks

This document presents the material of two lectures on statistical physics and neural representations, delivered by one of us (R.M.) at the Fundamental Problems in Statistical Physics XIV summer school in July 2017. In a first part, we consider the neural representations of space (maps) in the hippocampus. We introduce an extension of the Hopfield model, able to store multiple spatial maps as continuous, finite-dimensional attractors. The phase diagram and dynamical properties of the model are analyzed. We then show how spatial representations can be dynamically decoded using an effective Ising model capturing the correlation structure in the neural data, and compare applications to data obtained from hippocampal multi-electrode recordings and by (sub)sampling our attractor model. In a second part, we focus on the problem of learning data representations in machine learning, in particular with artificial neural networks. We start by introducing data representations through some illustrations. We then analyze two important algorithms, Principal Component Analysis and Restricted Boltzmann Machines, with tools from statistical physics

Physics of epigenetic landscapes and statistical inference by cells

Biology is currently in the midst of a revolution. Great technological advances have led to unprecedented quantitative data at the whole genome level. However, new techniques are needed to deal with this deluge of high-dimensional data. Therefore, statistical physics has the potential to help develop systems biology level models that can incorporate complex data. Additionally, physicists have made great strides in understanding non-equilibrium thermodynamics. However, the consequences of these advances have yet to be fully incorporated into biology. There are three specific problems that I address in my dissertation. First, a common metaphor for describing development is a rugged "epigenetic landscape" where cell fates are represented as attracting valleys resulting from a complex regulatory network. I introduce a framework for explicitly constructing epigenetic landscapes that combines genomic data with techniques from spin-glass physics. The model reproduces known reprogramming protocols and identifies candidate transcription factors for reprogramming to novel cell fates, suggesting epigenetic landscapes are a powerful paradigm for understanding cellular identity. Second, I examine the dynamics of cellular reprogramming. By reanalyzing all available time-series data, I show that gene expression dynamics during reprogramming follow a simple one-dimensional reaction coordinate that is independent of both the time and details of experimental protocol used. I show that such a reaction coordinate emerges naturally from epigenetic landscape models of cell identity where cellular reprogramming is viewed as a "barrier-crossing" between the starting and ending cell fates. Overall, the analysis and model suggest that gene expression dynamics during reprogramming follow a canonical trajectory consistent with the idea of an "optimal path"' in gene expression space for reprogramming. Third, an important task of cells is to perform complex computations in response to external signals. Intricate networks are required to sense and process signals, and since cells are inherently non-equilibrium systems, these networks naturally consume energy. Since there is a deep connection between thermodynamics, computation, and information, a natural question is what constraints does thermodynamics place on statistical estimation and learning. I modeled a single chemical receptor and established the first fundamental relationship between the energy consumption and statistical accuracy of a receptor in a cell