Learning and reversal in the sub-cortical limbic system: a computational model

The basal ganglia are a group of nuclei that signal to and from the cerebral cortex. They play an important role in cognition and in the initiation and regulation of normal motor activity. A range of characteristic motor diseases such as Parkinson's and Huntington's have been associated with the degeneration and lesioning of the dopaminergic neurons that target these regions. The study of dopaminergic activity has numerous benefits from understanding how and what effects neurodegenerative diseases have on behavior to determining how the brain responds and adapts to rewards. The study is also useful in understanding what motivates agents to select actions and do the things that they do. The striatum is a major input structure of the basal ganglia and is a target structure of dopaminergic neurons which originate from the mid brain. These dopaminergic neurons release dopamine which is known to exert modulatory influences on the striatal projections. Action selection and control are involved in the dorsal regions of the striatum while the dopaminergic projections to the ventral striatum are involved in reward based learning and motivation. There are many computational models of the dorsolateral striatum and the basal ganglia nuclei which have been proposed as neural substrates for prediction, control and action selection. However, there are relatively few models which aim to describe the role of the ventral striatal nucleus accumbens and its core and shell sub divisions in motivation and reward related learning. This thesis presents a systems level computational model of the sub-cortical nuclei of the limbic system which focusses in particular, on the nucleus accumbens shell and core circuitry. It is proposed that the nucleus accumbens core plays a role in enabling reward driven motor behaviour by acquiring stimulus-response associations which are used to invigorate responding. The nucleus accumbens shell mediates the facilitation of highly rewarding behaviours as well as behavioural switching. In this model, learning is achieved by implementing isotropic sequence order learning and a third factor (ISO-3) that triggers learning at relevant moments. This third factor is modelled by phasic dopaminergic activity which enables long term potentiation to occur during the acquisition of stimulus-reward associations. When a stimulus no longer predicts reward, tonic dopaminergic activity is generated. This enables long term depression. Weak depression has been simulated in the core so that stimulus-response associations which are used to enable instrumental response are not rapidly abolished. However, comparatively strong depression is implemented in the shell so that information about the reward is quickly updated. The shell influences the facilitation of highly rewarding behaviours enabled by the core through a shell-ventral pallido-medio dorsal pathway. This pathway functions as a feed-forward switching mechanism and enables behavioural flexibility. The model presented here, is capable of acquiring associations between stimuli and rewards and simulating reversal learning. In contrast to earlier work, the reversal is modelled by the attenuation of the previously learned behaviour. This allows for the reinstatement of behaviour to recur quickly as observed in animals. The model will be tested in both open- and closed-loop experiments and compared against animal experiments

Lasting dynamic effects of the psychedelic 2,5-dimethoxy-4- iodoamphetamine ((±)-DOI) on cognitive flexibility

Psychedelic drugs can aid fast and lasting remission from various neuropsychiatric disorders, though the underlying mechanisms remain unclear. Preclinical studies suggest serotonergic psychedelics enhance neuronal plasticity, but whether neuroplastic changes can also be seen at cognitive and behavioural levels is unexplored. Here we show that a single dose of the psychedelic 2,5-dimethoxy-4-iodoamphetamine ((±)-DOI) affects structural brain plasticity and cognitive flexibility in young adult mice beyond the acute drug experience. Using ex vivo magnetic resonance imaging, we show increased volumes of several sensory and association areas one day after systemic administration of 2mgkg-1 (±)-DOI. We then demonstrate lasting effects of (±)- DOI on cognitive flexibility in a two-step probabilistic reversal learning task where 2mgkg-1 (±)-DOI improved the rate of adaptation to a novel reversal in task structure occurring one-week post-treatment. Strikingly, (±)-DOI-treated mice started learning from reward omissions, a unique strategy not typically seen in mice in this task, suggesting heightened sensitivity to previously overlooked cues. Crucially, further experiments revealed that (±)-DOI’s effects on cognitive flexibility were contingent on the timing between drug treatment and the novel reversal, as well as on the nature of the intervening experience. (±)-DOI’s facilitation of both cognitive adaptation and novel thinking strategies may contribute to the clinical benefits of psychedelic-assisted therapy, particularly in cases of perseverative behaviours and a resistance to change seen in depression, anxiety, or addiction. Furthermore, our findings highlight the crucial role of time-dependent neuroplasticity and the influence of experiential factors in shaping the therapeutic potential of psychedelic interventions for impaired cognitive flexibility

Integration of geometric and contextual inputs to hippocampal place cells.

Neurons in the rodent hippocampus fire in highly restricted portions of an environment. These place cells have receptive fields called place fields and are argued to form a representation of space. The work described in this thesis explores the different types of sensory input to these cells, how these inputs are integrated and the implications for our understanding of hippocampal processing. To this end, hippocampal pyramidal neurons were recorded from awake, behaving rats as they foraged for food in a series of different environments. By manipulating the environments to which rats were exposed the nature of the input to place cells was elucidated. The first two experiments explored the influence of geometry on place fields. A novel environment was created that facilitated an examination of how the boundaries that constituted that environment affected place field activity. It was found that the presence of boundaries was important in order to have well-defined and consistent place fields across trials. Furthermore, exposure to one environment affected the place fields recorded in a similar but different environment, suggesting that learning was occurring. The final experiment examined in greater detail the effect of learning on the place cell representation. Place cells were recorded in two neighbouring environments that were the same colour. Initially similar place cell representations were found to diverge over the course of several days and weeks such that the place cell activations in both environments became distinct. Once a distinct pattern of place cell activity was seen, the colour of the environments was changed. The learnt discrimination that was acquired in the initial environments was not transferred to the novel environment. This suggested that the information acquired by place cells was specific to a given environment. These results are incorporated into, and extend, an existing model of place field formation

The effects of Morris water maze learning on the number, morphology and molecular composition of rat hippocampal dentate gyrus synapses

spatial long-term memory formation is dependent upon the hippocampus and associated brain structures in mammals. Memory storage is believed to involve changes in the way information is exchanged between neurons, and this is principally governed by their synaptic connections. Changes can occur in the functional properties of individual synapses, but evidence suggests that morphological changes may also occur. Research described in this thesis has used the Morris water maze, a behavioural paradigm that requires rodents to form long-term memories about a spatial environment, and this learning task involves the function of the hippocampus. Electron microscopy was used to investigate the ultrastructural morphology and composition of synapses in the hippocampal dentate gyrus in several groups of animals. Three time- points were investigated, 3, 9 and 24 hours after the start of training, which also corresponded to small, intermediate and large amounts of training, as well as two different types of control, naïve and swim-only. Animals investigated 3 hours after the start of training did not show significant long term memory for the task, whereas animals investigated 9 and 24 hours after the start of learning displayed long-term memory recall when measured by the quadrant analysis test (probe trial). Hippocampal dimensions and dentate granule cell densities were similar between all animal groups. No significant changes to synaptic ultrastructural morphology were evident in the 3 hour group. In the 9 hour group, significant increases in synapse density and synapse to neuron ratio were observed, with a simultaneous decrease in the synapse mean height and average area of PSD (post-synaptie density) per synapse. No significant changes were observed in the exercise-matched swim-only controls, suggesting that the changes were related to long-term memory formation. Morphological changes were not evident in the 24 hour group, despite long term memory recall, suggesting that the morphological changes following spatial learning in the Morris water maze are transient. The total amount of synaptic membrane was not significantly different between any of the groups, suggesting that although new, smaller synapses may be formed as a result of learning, changes also occur to existing synapses, which may result in their re-categorisation or even removal. Analysis of ionotropic glutamate receptors following training proved inconclusive, particularly for NMDA receptors, but did suggest that AMP A receptors are increased in the initial stages of learning, which may be a mechanism of short-term memory storage

Context Modulated Spatial Encoding and Memory Consolidation in the Rodent Hippocampus

The recollection of daily events is inherently personal: episodic memories are defined by the recollection of one’s sense of self during a particular event, within a surrounding context. Representations of such experiences are initially encoded in the hippocampus then consolidated by their repeated reactivation in synchrony with the cortex during sleep. After consolidation, memories are less prone to interference by similar experiences. However, a day in one’s life is usually constructed from multiple episodic experiences which can span multiple contexts. Little is known about the potential interference by previous memories on the construction of novel representations when contextual features are shared. Moreover, salient episodic memories are better remembered than neutral ones in the long term. Highly rewarding, traumatic or novel experiences can lead to intrusive (e.g. Post Traumatic Stress Disorder) or extremely vivid recall (e.g. Flashbulb memories) recall, and in general longer lasting memories. This phenomenon of prioritised memory consolidation is thought to ensure the storage of relevant memories, at the detriment of less important ones, and has been shown to correlate with an overall increase in their reactivation frequency during sleep. However, the temporal dynamics of memory triage during sleep have not yet been investigated. Recording from many hippocampal neurons simultaneously in the rat, during both sleep and the exploration of three completely new environments each session, we tracked the encoding and consolidation of feature-sharing and salience modulated representations. We provide evidence for the presence of neural patterns of activity that may support generalisation with similar past experiences, as well as differentiation of the novel representation during its initial stabilisation window. Furthermore, we show that the temporal dynamics of memory triage are not uniform, and instead exhibit a cyclic (time attributed to each memory) and an amplitude (relative proportion) component

The role of experience in memory consolidation

The hippocampus is believed to play a key role in long-term consolidation during sleep. Additionally, hippocampal place cells - pyramidal neurons that fire in discrete locations in the space - have been used as a reliable behavioural correlate to study learning and memory of spatial tasks. To date, most studies investigating memory consolidation focus on recordings from neural data obtained during tasks the subjects have been previously overexposed to. While this strategy guarantees a higher stability of the spatial map encoding for that specific experience, the reality of more naturalistic settings is that both humans and other animals can encounter multiple events of diverse duration and relevance on a daily basis. Yet, it remains unclear how the brain prioritizes and successfully stores multiple novel events. To address this question, we exposed a group of rats to pairs of novel linear tracks across different days. Each day, rats were allowed to run in each track for a different fixed number of laps, and the experience was preceded and followed by a sleep session. We found that the hippocampus was able to discriminate the different spatial maps even for short exposures with unstable place fields. We also observed awake and sleep hippocampal replay of all tracks regardless of the stability of their spatial representations. However, when presented with similar experiences of different duration in the novel tracks, the hippocampus prioritised the consolidation of the longer experience if the spatial representation of the shorter one was still unstable. Finally, we found that both awake hippocampal replay and theta sequences influenced the levels of subsequent sleep replay. These results aim to add further understanding of how experience shapes the encoding of different spatial trajectories, and how offline activity contributes to the consolidation of their memory representations

The influence of dopamine on prediction, action and learning

In this thesis I explore functions of the neuromodulator dopamine in the context of autonomous learning and behaviour. I first investigate dopaminergic influence within a simulated agent-based model, demonstrating how modulation of synaptic plasticity can enable reward-mediated learning that is both adaptive and self-limiting. I describe how this mechanism is driven by the dynamics of agentenvironment interaction and consequently suggest roles for both complex spontaneous neuronal activity and specific neuroanatomy in the expression of early, exploratory behaviour. I then show how the observed response of dopamine neurons in the mammalian basal ganglia may also be modelled by similar processes involving dopaminergic neuromodulation and cortical spike-pattern representation within an architecture of counteracting excitatory and inhibitory neural pathways, reflecting gross mammalian neuroanatomy. Significantly, I demonstrate how combined modulation of synaptic plasticity and neuronal excitability enables specific (timely) spike-patterns to be recognised and selectively responded to by efferent neural populations, therefore providing a novel spike-timing based implementation of the hypothetical ‘serial-compound’ representation suggested by temporal difference learning. I subsequently discuss more recent work, focused upon modelling those complex spike-patterns observed in cortex. Here, I describe neural features likely to contribute to the expression of such activity and subsequently present novel simulation software allowing for interactive exploration of these factors, in a more comprehensive neural model that implements both dynamical synapses and dopaminergic neuromodulation. I conclude by describing how the work presented ultimately suggests an integrated theory of autonomous learning, in which direct coupling of agent and environment supports a predictive coding mechanism, bootstrapped in early development by a more fundamental process of trial-and-error learning

An investigation into the neural substrates of virtue to determine the key place of virtues in human moral development

Virtues, as described by Aristotle and Aquinas, are understood as dispositions of character to behave in habitual, specific, positive ways; virtue is a critical requirement for human flourishing. From the perspective of Aristotelian-Thomistic anthropology which offers an integrated vision of the material and the rational in the human person, I seek to identify the neural bases for the development and exercise of moral virtue. First I review current neuroscientific knowledge of the capacity of the brain to structure according to experience, to facilitate behaviours, to regulate emotional responses and support goal election. Then, having identified characteristics of moral virtue in the light of the distinctions between cardinal virtues, I propose neural substrates by mapping neuroscientific knowledge to these characteristics. I then investigate the relationship between virtue, including its neurobiological features, and human flourishing. This process allows a contemporary and evidence-based corroboration for a model of moral development based on growth in virtue as understood by Aristotle and Aquinas, and a demonstration of a biological aptitude and predisposition for the development of virtue. Conclusions are drawn with respect to science, ethics, and parenting