Short-term plasticity as cause-effect hypothesis testing in distal reward learning

Asynchrony, overlaps and delays in sensory-motor signals introduce ambiguity as to which stimuli, actions, and rewards are causally related. Only the repetition of reward episodes helps distinguish true cause-effect relationships from coincidental occurrences. In the model proposed here, a novel plasticity rule employs short and long-term changes to evaluate hypotheses on cause-effect relationships. Transient weights represent hypotheses that are consolidated in long-term memory only when they consistently predict or cause future rewards. The main objective of the model is to preserve existing network topologies when learning with ambiguous information flows. Learning is also improved by biasing the exploration of the stimulus-response space towards actions that in the past occurred before rewards. The model indicates under which conditions beliefs can be consolidated in long-term memory, it suggests a solution to the plasticity-stability dilemma, and proposes an interpretation of the role of short-term plasticity.Comment: Biological Cybernetics, September 201

Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks

Biological plastic neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifetime learning. The interplay of these elements leads to the emergence of adaptive behavior and intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) use simulated evolution in-silico to breed plastic neural networks with a large variety of dynamics, architectures, and plasticity rules: these artificial systems are composed of inputs, outputs, and plastic components that change in response to experiences in an environment. These systems may autonomously discover novel adaptive algorithms, and lead to hypotheses on the emergence of biological adaptation. EPANNs have seen considerable progress over the last two decades. Current scientific and technological advances in artificial neural networks are now setting the conditions for radically new approaches and results. In particular, the limitations of hand-designed networks could be overcome by more flexible and innovative solutions. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and developments are presented

Rare neural correlations implement robotic conditioning with delayed rewards and disturbances

Neural conditioning associates cues and actions with following rewards. The environments in which robots operate, however, are pervaded by a variety of disturbing stimuli and uncertain timing. In particular, variable reward delays make it difficult to reconstruct which previous actions are responsible for following rewards. Such an uncertainty is handled by biological neural networks, but represents a challenge for computational models, suggesting the lack of a satisfactory theory for robotic neural conditioning. The present study demonstrates the use of rare neural correlations in making correct associations between rewards and previous cues or actions. Rare correlations are functional in selecting sparse synapses to be eligible for later weight updates if a reward occurs. The repetition of this process singles out the associating and reward-triggering pathways, and thereby copes with distal rewards. The neural network displays macro-level classical and operant conditioning, which is demonstrated in an interactive real-life human-robot interaction. The proposed mechanism models realistic conditioning in humans and animals and implements similar behaviors in neuro-robotic platforms

Normal spatial learning and improved spatial working memory in mice (mus musculus) lacking dopamine d4 receptors

Dopamine terminals in the hippocampus and prefrontal cortex modulate cognitive processes such as spatial learning and working memory. Because dopamine D4 receptors are expressed in these brain areas we have analyzed mutant mice lacking this receptor subtype (Drd4-/-). Wild-type and Drd4-/- mice were challenged in two spatial learning paradigms: the Morris water maze and an alternation T-maze. Drd4-/- mice showed normal place learning ability to find a hidden platform based on spatial extra-maze cues. In addition, Drd4-/- mice were able to find a new platform location with the same learning plasticity as wild type-mice. Spatial working memory assessed on a T maze showed that Drd4-/- mice were more efficient than wild-type mice in acquiring the maximum plateau of correct alternation scores. These results provide further evidence that the functional consequence of lacking D4 receptors is more evident in behaviors dependent on the integrity of the prefrontal cortex.Fil: Falzone, Tomas Luis. Consejo Nacional de Investigaciones Científicas y Técnicas. Instituto de Investigaciones en Ingeniería Genética y Biología Molecular "Dr. Héctor N. Torres"; ArgentinaFil: Avale, Maria Elena. Consejo Nacional de Investigaciones Científicas y Técnicas. Instituto de Investigaciones en Ingeniería Genética y Biología Molecular "Dr. Héctor N. Torres"; ArgentinaFil: Gelman, Diego Matias. Consejo Nacional de Investigaciones Científicas y Técnicas. Instituto de Investigaciones en Ingeniería Genética y Biología Molecular "Dr. Héctor N. Torres"; ArgentinaFil: Rubinstein, Marcelo. Consejo Nacional de Investigaciones Científicas y Técnicas. Instituto de Investigaciones en Ingeniería Genética y Biología Molecular "Dr. Héctor N. Torres"; Argentin

A rapidly acquired foraging-based working memory task, sensitive to hippocampal lesions, reveals age-dependent and age-independent behavioural changes in a mouse model of amyloid pathology

© 2018 Elsevier Inc. Three experiments examined the ability of mice to forage efficiently for liquid rewards in pots located in an open field arena. Search behaviour was unconstrained other than by the walls of the arena. All mice acquired the task within 4 days of training, with one trial per day. Experiment 1 tested the hypothesis that hippocampal lesions would disrupt foraging behaviour using extramaze cues. Mice with hippocampal lesions showed normal latency to initiate foraging and to complete the task relative to sham-operated mice. However, lesioned mice showed increased perseverative responding (sensitization) to recently rewarded locations, increased total working memory errors and an increased propensity to search near previously rewarded locations. In Experiment 2, the extramaze cues were obscured and each pot was identified by a unique pattern. Under these conditions, mice with hippocampal lesions showed comparable working memory errors to control mice. However, lesioned mice continued to display increased perseverative responding and altered search strategies. Experiment 3 tested the hypothesis that age-related accumulation of amyloid would disrupt foraging behaviour in transgenic PDAPP mice expressing the V717F amyloid precursor protein (APP) mutation. Consistent with previous findings, PDAPP mice showed both age-dependent and age-independent behavioural changes. More specifically, 14–16 month-old PDAPP mice showed a deficit in perseverative responding and working memory errors. In contrast, changes in search behaviour, such as systematic circling, were present throughout development. The latter indicates that APP overexpression contributed to some features of the PDAPP behavioural phenotype, whereas working memory and flexible responding was sensitive to ageing and β-amyloid burden. In conclusion, the present study provided novel insight into the role of the hippocampus and the effects of APP overexpression on memory and search behaviour in an open-field foraging task

Preservation of long-term memory and synaptic plasticity despite short-term impairments in the Tc1 mouse model of Down syndrome

Down syndrome (DS) is a genetic disorder arising from the presence of a third copy of the human chromosome 21 (Hsa21). Recently, O’Doherty and colleagues in an earlier study generated a new genetic mouse model of DS (Tc1) that carries an almost complete Hsa21. Since DS is the most common genetic cause of mental retardation, we have undertaken a detailed analysis of cognitive function and synaptic plasticity in Tc1 mice. Here we show that Tc1 mice have impaired spatial working memory (WM) but spared long-term spatial reference memory (RM) in the Morris watermaze. Similarly, Tc1 mice are selectively impaired in short-term memory (STM) but have intact long-term memory (LTM) in the novel object recognition task. The pattern of impaired STM and normal LTM is paralleled by a corresponding phenotype in long-term potentiation (LTP). Freely-moving Tc1 mice exhibit reduced LTP 1 h after induction but normal maintenance over days in the dentate gyrus of the hippocampal formation. Biochemical analysis revealed a reduction in membrane surface expression of the AMPAR (α-amino-3-hydroxy-5-methyl-4-propionic acid receptor) subunit GluR1 in the hippocampus of Tc1 mice, suggesting a potential mechanism for the impairment in early LTP. Our observations also provide further evidence that STM and LTM for hippocampus-dependent tasks are subserved by parallel processing streams

How can cognitive enrichment revert the effects of stress

Tese de mestrado. Biologia (Biologia Humana e Ambiente). Universidade de Lisboa, Faculdade de Ciências, 2011O sistema nervoso é responsável pelos processos que tornam a vida humana possível. Permite-nos pensar, sonhar e criar memórias. Controla as nossas acções mais básicas e involuntárias como piscar os olhos, respirar e manter os nossos corações a bater. Os neurónios são as unidades básicas do sistema nervoso, são células notáveis pela especialização para comunicação intercelular que apresentam. Normalmente os neurónios apresentam quatro regiões distintas: o corpo celular, dendrites, axónio e terminais pré-sinápticos. Entre a ponta de cada terminal sináptico e o ponto de contacto no neurónio seguinte há um pequeno espaço chamado sinapse. Cada neurónio pode ligar-se a cerca de 1000 a 10000 outros neurónios. Apesar de muitas destas conexões serem especializadas, todos os neurónios utilizam uma de duas possíveis formas de transmissão sináptica: eléctrica ou química. A força de ambas as formas de transmissão sináptica pode ser aumentada ou diminuída através da actividade celular. Esta plasticidade sináptica é vital para a formação de memórias e para os processos de aprendizagem. Long term potentiation de sinapses químicas é um dos modelos mais estudados para a formação de memórias no cérebro de mamíferos. O sistema nervoso pode ser dividido em duas partes principais: o sistema nervoso central (SNC), que é constituído pelo cérebro e espinal medula, e o sistema nervoso periférico (SNP), que consiste nos nervos craniais e espinais e gânglios associados. O sistema nervoso central pode ser dividido em sete partes: espinal medula, medula, ponte, cerebelo, mesencéfalo, diencéfalo e os hemisférios cerebrais. O córtex prefrontal poderá estar relacionado com a organização da informação interna e externa que é necessária para produzir comportamentos complexos. É a área cerebral que mais espaço ocupa no cérebro humano, e é maior na nossa espécie do que em outros primatas. Este facto poderá ser mais um indício de que o córtex prefrontal está envolvido em algumas capacidades consideradas inteligentes. Está também envolvido no processamento de informação cognitiva e emocional e nos processos de memória de trabalho ou working memory (WM). A working memory é uma memória temporária que é bastante útil no cumprimento de tarefas. O hipocampo é outra zona do cérebro, também muito importante na formação de memória (em especial, memória espacial) e na aprendizagem. As funções do hipocampo podem varar entre espécies, mas a maior parte dos estudos são convergentes, no que respeita a caracterização da anatomia e fisiologia desta área. O hipocampo está ligado ao córtex prefrontal, sendo que esta conexão é feia por axónios que têm origem no subículo e terminam nos neurónios piramidais do PFC. O stress é um aspecto comum nas nossas vidas diárias, mas ainda assim há alguma ambiguidade no que toca à sua definição. É um desafio real ou percepcionado, quer endógeno quer exógeno, que perturba o equilíbrio natural do organismo – homeostase. Os humanos e outros animais respondem a estes desafios desencadeando uma série de mecanismos neuronais, endócrinos, neuro-endócrinos e metabólicos. O impacto do stress no cérebro tem recebida imensa atenção nos últimos anos, tanto da parte de neurologistas como de leigos. De facto, o cérebro é o principal órgão por detrás da resposta ao stress: determina o que é ou não uma ameaça, coordena as reacções que cada um tem, e altera-se, estrutural e funcionalmente como resultado de experiências stressantes. Uma das principais respostas ao stress é a activação do eixo HPA (hipotálamo-pituitária-adrenal). Os danos no córtex prefrontal e no hipocampo em consequência de episódios de stress ou de stress crónico estão bem descritos. Em 2007, Cerqueira demonstrou que a plasticidade sináptica entre o hipocampo e o córtex prefrontal diminui depois de um período de stress crónico e que os efeitos do stress neste circuito são devidos a atrofia neuronal, e não a perda de neurónios. Assim, este trabalho propôs-se a tentar compreender como é que estes efeitos do stress poderão ser revertidos. Para isso, utilizámos ratos Wistar Han como modelo animal, e criámos quatro grupos distintos: animais stressados, animais stressados que foram sujeitos a uma tarefa cognitiva, animais controlo e animais controlo que foram submetidos à mesma tarefa. Posteriormente, avaliámos dois parâmetros em cada grupo: plasticidade sináptica e análise morfológica. Para esse efeito utilizámos um protocolo de electrofisiologia. Um estímulo eléctrico no hipocampo gera uma resposta no córtex prefrontal, e essa resposta pode ser aumentada se for induzido LTP. No entanto esse aumento (ou potenciação do sinal) não é tão elevado em animais stressados, sendo por isso uma boa medida para avaliar se o hole board consegue ou não reverter a perda de plasticidade sináptica neste circuito. A análise morfológica consistiu em reconstruir neurónios em três dimensões, utilizando o software Neurolucida e assim conseguir avaliar a quantidade de dendrites, o seu comprimento, e a densidade de espinhas de cada neurónio estudado. Os nossos resultados demonstram que o treino no hole board foi bem sucedido, já que todos os animais aprenderam a tarefa. No entanto, os animais stressados têm mais dificuldade em aprender a tarefa. No protocolo de electrofisiologia, os resultados foram os esperados, os animais stressados demonstraram menor plasticidade sináptica, enquanto que os animais que fizeram a tarefa cognitiva demonstraram maior plasticidade sináptica, embora o aumento da resposta do prefrontal córtex não fosse tão elevado como nos controlos. Em relação à análise morfológica, os resultados também corresponderam ao esperado. Em praticamente todos os parâmetros foi visível a recuperação dos animais que treinaram no hole board, como por exemplo no número de ramos dendríticos, no seu comprimento ou ate na densidade dendrítica de espinhas. Em conclusão, este estudo demonstrou que uma tarefa cognitiva, neste caso o hole board, pode reverter alguns efeitos do stress, incluindo a perda de plasticidade sináptica e a atrofia neuronal na conexão entre i hipocampo e o córtex prefrontal.It is known that stress has negative effects on our health. It is important to understand how stress affects us and how we can revert those effects. We proposed the hypothesis that some consequences of stress might be reverted through cognitive enrichment. So, using Wistar Han rats as an animal model, we formed four groups of animals: control animals, control animals that performed a cognitive enrichment task (hole-board), stressed animals, and stressed animals that performed the same task. We then evaluated two parameters on every animal: synaptic plasticity between the prefrontal cortex (PFC) and the hippocampus and morphological correlate. After analyzing the results, we could conclude that the cognitive enrichment task can, indeed, revert some effects of stress, including the loss of synaptic plasticity and neuronal atrophy

The influence of dopamine on prediction, action and learning

In this thesis I explore functions of the neuromodulator dopamine in the context of autonomous learning and behaviour. I first investigate dopaminergic influence within a simulated agent-based model, demonstrating how modulation of synaptic plasticity can enable reward-mediated learning that is both adaptive and self-limiting. I describe how this mechanism is driven by the dynamics of agentenvironment interaction and consequently suggest roles for both complex spontaneous neuronal activity and specific neuroanatomy in the expression of early, exploratory behaviour. I then show how the observed response of dopamine neurons in the mammalian basal ganglia may also be modelled by similar processes involving dopaminergic neuromodulation and cortical spike-pattern representation within an architecture of counteracting excitatory and inhibitory neural pathways, reflecting gross mammalian neuroanatomy. Significantly, I demonstrate how combined modulation of synaptic plasticity and neuronal excitability enables specific (timely) spike-patterns to be recognised and selectively responded to by efferent neural populations, therefore providing a novel spike-timing based implementation of the hypothetical ‘serial-compound’ representation suggested by temporal difference learning. I subsequently discuss more recent work, focused upon modelling those complex spike-patterns observed in cortex. Here, I describe neural features likely to contribute to the expression of such activity and subsequently present novel simulation software allowing for interactive exploration of these factors, in a more comprehensive neural model that implements both dynamical synapses and dopaminergic neuromodulation. I conclude by describing how the work presented ultimately suggests an integrated theory of autonomous learning, in which direct coupling of agent and environment supports a predictive coding mechanism, bootstrapped in early development by a more fundamental process of trial-and-error learning

Embodying a Computational Model of Hippocampal Replay for Robotic Reinforcement Learning

Hippocampal reverse replay has been speculated to play an important role in biological reinforcement learning since its discovery over a decade ago. Whilst a number of computational models have recently emerged in an attempt to understand the dynamics of hippocampal replay, there has been little progress in testing and implementing these models in real-world robotics settings. Presented first in this body of work then is a bio-inspired hippocampal CA3 network model. It runs in real-time to produce reverse replays of recent spatio-temporal sequences, represented as place cell activities, in a robotic spatial navigation task. The model is based on two very recent computational models of hippocampal reverse replay. An analysis of these models show that, in their original forms, they are each insufficient for effective performance when applied to a robot. As such, choosing particular elements from each allows for a computational model that is sufficient for application in a robotic task. Having a model of reverse replay applied successfully in a robot provides the groundwork necessary for testing the ways in which reverse replay contributes to reinforcement learning. The second portion of the work presented here builds on a previous reinforcement learning neural network model of a basic hippocampal-striatal circuit using a three-factor learning rule. By integrating reverse replays into this reinforcement learning model, results show that reverse replay, with its ability to replay the recent trajectory both in the hippocampal circuit and the striatal circuit, can speed up the learning process. In addition, for situations where the original reinforcement learning model performs poorly, such as when its time dynamics do not sufficiently store enough of the robot's behavioural history for effective learning, the reverse replay model can compensate for this by replaying the recent history. These results are inline with experimental findings showing that disruption of awake hippocampal replay events severely diminishes, but does not entirely eliminate, reinforcement learning. This work provides possible insights into the important role that reverse replays could contribute to mnemonic function, and reinforcement learning in particular; insights that could benefit the robotic, AI, and neuroscience communities. However, there is still much to be done. How reverse replays are initiated is still an ongoing research problem, for instance. Furthermore, the model presented here generates place cells heuristically, but there are computational models tackling the problem of how hippocampal cells such as place cells, but also grid cells and head direction cells, emerge. This leads to the pertinent question of asking how these models, which make assumptions about their network architectures and dynamics, could integrate with the computational models of hippocampal replay which make their own assumptions on network architectures and dynamics

Demonstrating Advantages of Neuromorphic Computation: A Pilot Study

Neuromorphic devices represent an attempt to mimic aspects of the brain's architecture and dynamics with the aim of replicating its hallmark functional capabilities in terms of computational power, robust learning and energy efficiency. We employ a single-chip prototype of the BrainScaleS 2 neuromorphic system to implement a proof-of-concept demonstration of reward-modulated spike-timing-dependent plasticity in a spiking network that learns to play the Pong video game by smooth pursuit. This system combines an electronic mixed-signal substrate for emulating neuron and synapse dynamics with an embedded digital processor for on-chip learning, which in this work also serves to simulate the virtual environment and learning agent. The analog emulation of neuronal membrane dynamics enables a 1000-fold acceleration with respect to biological real-time, with the entire chip operating on a power budget of 57mW. Compared to an equivalent simulation using state-of-the-art software, the on-chip emulation is at least one order of magnitude faster and three orders of magnitude more energy-efficient. We demonstrate how on-chip learning can mitigate the effects of fixed-pattern noise, which is unavoidable in analog substrates, while making use of temporal variability for action exploration. Learning compensates imperfections of the physical substrate, as manifested in neuronal parameter variability, by adapting synaptic weights to match respective excitability of individual neurons.Comment: Added measurements with noise in NEST simulation, add notice about journal publication. Frontiers in Neuromorphic Engineering (2019