Neural Distributed Autoassociative Memories: A Survey

Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension. The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons). Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints. Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors.Comment: 31 page

Segregation of cortical head direction cell assemblies on alternating theta cycles

High-level cortical systems for spatial navigation, including entorhinal grid cells, critically depend on input from the head direction system. We examined spiking rhythms and modes of synchrony between neurons participating in head direction networks for evidence of internal processing, independent of direct sensory drive, which may be important for grid cell function. We found that head direction networks of rats were segregated into at least two populations of neurons firing on alternate theta cycles (theta cycle skipping) with fixed synchronous or anti-synchronous relationships. Pairs of anti-synchronous theta cycle skipping neurons exhibited larger differences in head direction tuning, with a minimum difference of 40 degrees of head direction. Septal inactivation preserved the head direction signal, but eliminated theta cycle skipping of head direction cells and grid cell spatial periodicity. We propose that internal mechanisms underlying cycle skipping in head direction networks may be critical for downstream spatial computation by grid cells.We kindly thank S. Gillet, J. Hinman, E. Newman and L. Ewell for their invaluable consultations and comments on previous versions of this manuscript, as well as M. Connerney, S. Eriksson, C. Libby and T. Ware for technical assistance and behavioral training. This work was supported by grants from the National Institute of Mental Health (R01 MH60013 and MH61492) and the Office of Naval Research Multidisciplinary University Research Initiative (N00014-10-1-0936). (R01 MH60013 - National Institute of Mental Health; MH61492 - National Institute of Mental Health; N00014-10-1-0936 - Office of Naval Research Multidisciplinary University Research Initiative)Accepted manuscrip

The Missing Link between Morphemic Assemblies and Behavioral Responses:a Bayesian Information-Theoretical model of lexical processing

We present the Bayesian Information-Theoretical (BIT) model of lexical processing: A mathematical model illustrating a novel approach to the modelling of language processes. The model shows how a neurophysiological theory of lexical processing relying on Hebbian association and neural assemblies can directly account for a variety of effects previously observed in behavioural experiments. We develop two information-theoretical measures of the distribution of usages of a morpheme or word, and use them to predict responses in three visual lexical decision datasets investigating inflectional morphology and polysemy. Our model offers a neurophysiological basis for the effects of morpho-semantic neighbourhoods. These results demonstrate how distributed patterns of activation naturally result in the arisal of symbolic structures. We conclude by arguing that the modelling framework exemplified here, is a powerful tool for integrating behavioural and neurophysiological results

Big data and the SP theory of intelligence

This article is about how the "SP theory of intelligence" and its realisation in the "SP machine" may, with advantage, be applied to the management and analysis of big data. The SP system -- introduced in the article and fully described elsewhere -- may help to overcome the problem of variety in big data: it has potential as "a universal framework for the representation and processing of diverse kinds of knowledge" (UFK), helping to reduce the diversity of formalisms and formats for knowledge and the different ways in which they are processed. It has strengths in the unsupervised learning or discovery of structure in data, in pattern recognition, in the parsing and production of natural language, in several kinds of reasoning, and more. It lends itself to the analysis of streaming data, helping to overcome the problem of velocity in big data. Central in the workings of the system is lossless compression of information: making big data smaller and reducing problems of storage and management. There is potential for substantial economies in the transmission of data, for big cuts in the use of energy in computing, for faster processing, and for smaller and lighter computers. The system provides a handle on the problem of veracity in big data, with potential to assist in the management of errors and uncertainties in data. It lends itself to the visualisation of knowledge structures and inferential processes. A high-parallel, open-source version of the SP machine would provide a means for researchers everywhere to explore what can be done with the system and to create new versions of it.Comment: Accepted for publication in IEEE Acces