The Performance of Associative Memory Models with Biologically Inspired Connectivity

This thesis is concerned with one important question in artificial neural networks, that is, how biologically inspired connectivity of a network affects its associative memory performance. In recent years, research on the mammalian cerebral cortex, which has the main responsibility for the associative memory function in the brains, suggests that the connectivity of this cortical network is far from fully connected, which is commonly assumed in traditional associative memory models. It is found to be a sparse network with interesting connectivity characteristics such as the “small world network” characteristics, represented by short Mean Path Length, high Clustering Coefficient, and high Global and Local Efficiency. Most of the networks in this thesis are therefore sparsely connected. There is, however, no conclusive evidence of how these different connectivity characteristics affect the associative memory performance of a network. This thesis addresses this question using networks with different types of connectivity, which are inspired from biological evidences. The findings of this programme are unexpected and important. Results show that the performance of a non-spiking associative memory model is found to be predicted by its linear correlation with the Clustering Coefficient of the network, regardless of the detailed connectivity patterns. This is particularly important because the Clustering Coefficient is a static measure of one aspect of connectivity, whilst the associative memory performance reflects the result of a complex dynamic process. On the other hand, this research reveals that improvements in the performance of a network do not necessarily directly rely on an increase in the network’s wiring cost. Therefore it is possible to construct networks with high associative memory performance but relatively low wiring cost. Particularly, Gaussian distributed connectivity in a network is found to achieve the best performance with the lowest wiring cost, in all examined connectivity models. Our results from this programme also suggest that a modular network with an appropriate configuration of Gaussian distributed connectivity, both internal to each module and across modules, can perform nearly as well as the Gaussian distributed non-modular network. Finally, a comparison between non-spiking and spiking associative memory models suggests that in terms of associative memory performance, the implication of connectivity seems to transcend the details of the actual neural models, that is, whether they are spiking or non-spiking neurons

Analysis of Oscillator Neural Networks for Sparsely Coded Phase Patterns

We study a simple extended model of oscillator neural networks capable of storing sparsely coded phase patterns, in which information is encoded both in the mean firing rate and in the timing of spikes. Applying the methods of statistical neurodynamics to our model, we theoretically investigate the model's associative memory capability by evaluating its maximum storage capacities and deriving its basins of attraction. It is shown that, as in the Hopfield model, the storage capacity diverges as the activity level decreases. We consider various practically and theoretically important cases. For example, it is revealed that a dynamically adjusted threshold mechanism enhances the retrieval ability of the associative memory. It is also found that, under suitable conditions, the network can recall patterns even in the case that patterns with different activity levels are stored at the same time. In addition, we examine the robustness with respect to damage of the synaptic connections. The validity of these theoretical results is confirmed by reasonable agreement with numerical simulations.Comment: 23 pages, 11 figure

Pathological slow-wave activity and impaired working memory binding in post-traumatic amnesia

Associative binding is key to normal memory function and is transiently disrupted during periods of post-traumatic amnesia (PTA) following traumatic brain injury (TBI). Electrophysiological abnormalities including low-frequency activity are common following TBI. Here, we investigate associative memory binding during PTA and test the hypothesis that misbinding is caused by pathological slowing of brain activity disrupting cortical communication. Thirty acute moderate-severe TBI patients (25 males; 5 females) and 26 healthy controls (20 males; 6 females) were tested with a precision working memory paradigm requiring the association of object and location information. Electrophysiological effects of TBI were assessed using resting-state EEG in a subsample of 17 patients and 21 controls. PTA patients showed abnormalities in working memory function and made significantly more misbinding errors than patients who were not in PTA and controls. The distribution of localisation responses was abnormally biased by the locations of non-target items for patients in PTA suggesting a specific impairment of object and location binding. Slow wave activity was increased following TBI. Increases in the delta-alpha ratio indicative of an increase in low-frequency power specifically correlated with binding impairment in working memory. Connectivity changes in TBI did not correlate with binding impairment. Working memory and electrophysiological abnormalities normalised at six-month follow-up. These results show that patients in PTA show high rates of misbinding that are associated with a pathological shift towards lower frequency oscillations

Analysing and enhancing the performance of associative memory architectures

This thesis investigates the way in which information about the structure of a set of training data with 'natural' characteristics may be used to positively influence the design of associative memory neural network models of the Hopfield type. This is done with a view to reducing the level of connectivity in models of this type. There are three strands to this work. Firstly, an empirical evaluation of the implementation of existing theory is given. Secondly, a number of existing theories are combined to produce novel network models and training regimes. Thirdly, new strategies for constructing and training associative memories based on knowledge of the structure of the training data are proposed. The first conclusion of this work is that, under certain circumstances, performance benefits may be gained by establishing the connectivity in a non-random fashion, guided by the knowledge gained from the structure of the training data. These performance improvements exist in relation to networks in which sparse connectivity is established in a purely random manner. This dilution occurs prior to the training of the network. Secondly, it is verified that, as predicted by existing theory, targeted post-training dilution of network connectivity provides greater performance when compared with networks in which connections are removed at random. Finally, an existing tool for the analysis of the attractor performance of neural networks of this type has been modified and improved. Furthermore, a novel, comprehensive performance analysis tool is proposed

Memory in autism spectrum disorder: a meta-analysis of experimental studies

To address inconsistencies in the literature on memory in Autism Spectrum Disorder (ASD), we report the first ever meta-analysis of short-term (STM) and episodic long-term (LTM) memory in ASD, evaluating the effects of type of material, type of retrieval and the role of inter-item relations. Analysis of 64 studies comparing individuals with ASD and typical development (TD) showed greater difficulties in ASD compared to TD individuals in STM (Hedges’ g=-0.53 [95%CI -0.90; -0.16], p=.005, I²=96%) compared to LTM (g=-0.30 [95%CI -0.42; -0.17], p<.00001, I²=24%), a small difficulty in verbal LTM (g=-0.21, p=.01), contrasting with a medium difficulty for visual LTM (g= -0.41, p=.0002) in ASD compared to TD individuals. We also found a general diminution in free recall compared to cued recall and recognition (LTM, free recall: g=-0.38, p<.00001, cued recall: g=-0.08, p=.58, recognition: g=-0.15, p=.16; STM, free recall: g=-0.59, p=.004, recognition: g=-0.33, p=.07). We discuss these results in terms of their relation to semantic memory. The limited diminution in verbal LTM and preserved overall recognition and cued recall (supported retrieval) may result from a greater overlap of these tasks with semantic long-term representations which are overall preserved in ASD. By contrast, difficulties in STM or free recall may result from less overlap with the semantic system or may involve additional cognitive operations and executive demands. These findings highlight the need to support STM functioning in ASD and acknowledge the potential benefit of using verbal materials at encoding and broader forms of memory support at retrieval to enhance performance

Exploring the Event-Related Potentials' Time Course of Associative Recognition in Autism

Behavioral data on episodic recollection in autism spectrum disorders (ASD) point limited relational memory functioning. However, the involvement of successive memory processes in the profile of episodic memory in ASD needs more study. Here, we used event-related potentials (ERP) to investigate the time course of episodic recollection with an associative recognition paradigm with picture pairs. Twenty-two participants with ASD and 32 with typical development (TD), all right-handed, were included. Behavioral results confirmed difficulties in correctly recognizing identical pairs in the ASD relative to TD group. We found an unexpected amplitude decrement on the P2 (220-270 msec) and FN400 (350-470 msec) potentials, suggesting diminished priming and familiarity effects in the ASD relative to TD group. However, ERP data revealed that the recognition of associative information relies on the same electrophysiological process (old/new effect in the 600-700-msec late positive component) in ASD participants as in TD ones, with a parietal extension in the ASD group. These results suggest that the electrophysiological processes of associative recognition are qualitatively similar in individuals with and without ASD but may differ quantitatively. This difference may be driven by the reduced early processing of picture pairs that may in turn lead to their diminished integration into the semantic memory system, being partially compensated by a greater involvement of associative memory during the recollection process. Other studies would be useful to go further in identifying these cognitive processes involved in atypical recognition in ASD and their neural substrates. LAY SUMMARY: We identified diminished performance on the associative recognition of picture pairs in adolescents and young adults with autism when compared to typical development. Electrophysiological data revealed qualitative similarities but quantitative differences between-group, with diminished priming and familiarity processes partially compensated by an enhanced parietal recollection process

Emulating long-term synaptic dynamics with memristive devices

The potential of memristive devices is often seeing in implementing neuromorphic architectures for achieving brain-like computation. However, the designing procedures do not allow for extended manipulation of the material, unlike CMOS technology, the properties of the memristive material should be harnessed in the context of such computation, under the view that biological synapses are memristors. Here we demonstrate that single solid-state TiO2 memristors can exhibit associative plasticity phenomena observed in biological cortical synapses, and are captured by a phenomenological plasticity model called triplet rule. This rule comprises of a spike-timing dependent plasticity regime and a classical hebbian associative regime, and is compatible with a large amount of electrophysiology data. Via a set of experiments with our artificial, memristive, synapses we show that, contrary to conventional uses of solid-state memory, the co-existence of field- and thermally-driven switching mechanisms that could render bipolar and/or unipolar programming modes is a salient feature for capturing long-term potentiation and depression synaptic dynamics. We further demonstrate that the non-linear accumulating nature of memristors promotes long-term potentiating or depressing memory transitions

Low-frequency oscillatory correlates of auditory predictive processing in cortical-subcortical networks: a MEG-study

Emerging evidence supports the role of neural oscillations as a mechanism for predictive information processing across large-scale networks. However, the oscillatory signatures underlying auditory mismatch detection and information flow between brain regions remain unclear. To address this issue, we examined the contribution of oscillatory activity at theta/alpha-bands (4–8/8–13 Hz) and assessed directed connectivity in magnetoencephalographic data while 17 human participants were presented with sound sequences containing predictable repetitions and order manipulations that elicited prediction-error responses. We characterized the spectro-temporal properties of neural generators using a minimum-norm approach and assessed directed connectivity using Granger Causality analysis. Mismatching sequences elicited increased theta power and phase-locking in auditory, hippocampal and prefrontal cortices, suggesting that theta-band oscillations underlie prediction-error generation in cortical-subcortical networks. Furthermore, enhanced feedforward theta/alpha-band connectivity was observed in auditory-prefrontal networks during mismatching sequences, while increased feedback connectivity in the alpha-band was observed between hippocampus and auditory regions during predictable sounds. Our findings highlight the involvement of hippocampal theta/alpha-band oscillations towards auditory prediction-error generation and suggest a spectral dissociation between inter-areal feedforward vs. feedback signalling, thus providing novel insights into the oscillatory mechanisms underlying auditory predictive processing

Improving Associative Memory in a Network of Spiking Neurons

In this thesis we use computational neural network models to examine the dynamics and functionality of the CA3 region of the mammalian hippocampus. The emphasis of the project is to investigate how the dynamic control structures provided by inhibitory circuitry and cellular modification may effect the CA3 region during the recall of previously stored information. The CA3 region is commonly thought to work as a recurrent auto-associative neural network due to the neurophysiological characteristics found, such as, recurrent collaterals, strong and sparse synapses from external inputs and plasticity between coactive cells. Associative memory models have been developed using various configurations of mathematical artificial neural networks which were first developed over 40 years ago. Within these models we can store information via changes in the strength of connections between simplified model neurons (two-state). These memories can be recalled when a cue (noisy or partial) is instantiated upon the net. The type of information they can store is quite limited due to restrictions caused by the simplicity of the hard-limiting nodes which are commonly associated with a binary activation threshold. We build a much more biologically plausible model with complex spiking cell models and with realistic synaptic properties between cells. This model is based upon some of the many details we now know of the neuronal circuitry of the CA3 region. We implemented the model in computer software using Neuron and Matlab and tested it by running simulations of storage and recall in the network. By building this model we gain new insights into how different types of neurons, and the complex circuits they form, actually work. The mammalian brain consists of complex resistive-capacative electrical circuitry which is formed by the interconnection of large numbers of neurons. A principal cell type is the pyramidal cell within the cortex, which is the main information processor in our neural networks. Pyramidal cells are surrounded by diverse populations of interneurons which have proportionally smaller numbers compared to the pyramidal cells and these form connections with pyramidal cells and other inhibitory cells. By building detailed computational models of recurrent neural circuitry we explore how these microcircuits of interneurons control the flow of information through pyramidal cells and regulate the efficacy of the network. We also explore the effect of cellular modification due to neuronal activity and the effect of incorporating spatially dependent connectivity on the network during recall of previously stored information. In particular we implement a spiking neural network proposed by Sommer and Wennekers (2001). We consider methods for improving associative memory recall using methods inspired by the work by Graham and Willshaw (1995) where they apply mathematical transforms to an artificial neural network to improve the recall quality within the network. The networks tested contain either 100 or 1000 pyramidal cells with 10% connectivity applied and a partial cue instantiated, and with a global pseudo-inhibition.We investigate three methods. Firstly, applying localised disynaptic inhibition which will proportionalise the excitatory post synaptic potentials and provide a fast acting reversal potential which should help to reduce the variability in signal propagation between cells and provide further inhibition to help synchronise the network activity. Secondly, implementing a persistent sodium channel to the cell body which will act to non-linearise the activation threshold where after a given membrane potential the amplitude of the excitatory postsynaptic potential (EPSP) is boosted to push cells which receive slightly more excitation (most likely high units) over the firing threshold. Finally, implementing spatial characteristics of the dendritic tree will allow a greater probability of a modified synapse existing after 10% random connectivity has been applied throughout the network. We apply spatial characteristics by scaling the conductance weights of excitatory synapses which simulate the loss in potential in synapses found in the outer dendritic regions due to increased resistance. To further increase the biological plausibility of the network we remove the pseudo-inhibition and apply realistic basket cell models with differing configurations for a global inhibitory circuit. The networks are configured with; 1 single basket cell providing feedback inhibition, 10% basket cells providing feedback inhibition where 10 pyramidal cells connect to each basket cell and finally, 100% basket cells providing feedback inhibition. These networks are compared and contrasted for efficacy on recall quality and the effect on the network behaviour. We have found promising results from applying biologically plausible recall strategies and network configurations which suggests the role of inhibition and cellular dynamics are pivotal in learning and memory