Embodying a Computational Model of Hippocampal Replay for Robotic Reinforcement Learning

Hippocampal reverse replay has been speculated to play an important role in biological reinforcement learning since its discovery over a decade ago. Whilst a number of computational models have recently emerged in an attempt to understand the dynamics of hippocampal replay, there has been little progress in testing and implementing these models in real-world robotics settings. Presented first in this body of work then is a bio-inspired hippocampal CA3 network model. It runs in real-time to produce reverse replays of recent spatio-temporal sequences, represented as place cell activities, in a robotic spatial navigation task. The model is based on two very recent computational models of hippocampal reverse replay. An analysis of these models show that, in their original forms, they are each insufficient for effective performance when applied to a robot. As such, choosing particular elements from each allows for a computational model that is sufficient for application in a robotic task. Having a model of reverse replay applied successfully in a robot provides the groundwork necessary for testing the ways in which reverse replay contributes to reinforcement learning. The second portion of the work presented here builds on a previous reinforcement learning neural network model of a basic hippocampal-striatal circuit using a three-factor learning rule. By integrating reverse replays into this reinforcement learning model, results show that reverse replay, with its ability to replay the recent trajectory both in the hippocampal circuit and the striatal circuit, can speed up the learning process. In addition, for situations where the original reinforcement learning model performs poorly, such as when its time dynamics do not sufficiently store enough of the robot's behavioural history for effective learning, the reverse replay model can compensate for this by replaying the recent history. These results are inline with experimental findings showing that disruption of awake hippocampal replay events severely diminishes, but does not entirely eliminate, reinforcement learning. This work provides possible insights into the important role that reverse replays could contribute to mnemonic function, and reinforcement learning in particular; insights that could benefit the robotic, AI, and neuroscience communities. However, there is still much to be done. How reverse replays are initiated is still an ongoing research problem, for instance. Furthermore, the model presented here generates place cells heuristically, but there are computational models tackling the problem of how hippocampal cells such as place cells, but also grid cells and head direction cells, emerge. This leads to the pertinent question of asking how these models, which make assumptions about their network architectures and dynamics, could integrate with the computational models of hippocampal replay which make their own assumptions on network architectures and dynamics

Biophysics-based modeling and data analysis of local field potential signal

Understanding the neurophysiological mechanisms of information processing within and across brain regions has always been a fundamental and challenging topic in neuroscience. Considerable works in the brain connectome and transcriptome have laid a profound foundation for understanding brain function by its structure. At the same time, the recent advance in recording techniques allows us to probe the nonstationary brain activity from various spatial and temporal scales. However, how to effectively build the dialogue between the anatomical structure and the dynamical brain signal still needs to be solved. To tackle the problem, we explore interpreting electrophysiology signals with mechanistic models. In chapter 2 we first segregate high-coherent brain signals into different pathways and then connect their dynamics to synaptic properties. Based on a state space model of LFP generation, we explore several preprocessing methods to bias the signal to the synaptic inputs and enhance the separatability of pathway-specific contributions. The separated sources are more reliable with the preprocessing methods, especially in highly coherent states, e.g., awake running. With reliably separated pathways, we further studied their synaptic properties and explored the local directional connections in the hippocampus. The estimated synaptic time constant and pathway connection agrees with well-established anatomical studies. In chapter 3 we explore establishing a simple model to capture the impulse response of passive neurons with detailed dendritic morphology. We validate Green’s function methods based on compartmentalized models by comparing them to numerical simulations and analytical solutions on continuous neuron membrane potentials. A parameterized model based on laminar Green’s function is further developed and helps to infer the anatomical properties, like the input current distribution and cell position, from their spatiotemporal response patterns. The effect of cell position and template are examed. Based on the model of chapter 3, we use the biophysical possible impulse response profile to regularize the source separation in the frequency domain in chapter 4. The components from different frequencies are clustered according to the same latent input distributions. The source separation in better-separated frequency bins from the same pathway helps separation in other highly contaminated frequencies. The optimization is formulated as a probabilistic model to maximize the negentropy as well as spatial likelihood. Similar to dipole approximation for EEG signals, Green’s function method provides an effective approximation to capture biologically possible spatiotemporal patterns and helps to guide the separation. We validated the method on real data with optogenetic stimulation. In chapter 5 we further separate the far-field signals from the local pathway activities according to their physiological properties. We propose a pipeline to reliably separate and automatically detect far-field signal components. Based on this, a toolbox is provided to remove the EMG artifacts and assess the cleaning performance. In the free-running animals, we show that EMG artifacts shadow the high-frequency oscillatory events detection, and EMG cleaning rescues this effect. Overall, this thesis explored multiple possibilities to incorporate neurophysiology knowledge to understand and model the electrical field potential signals.Das Verständnis der neurophysiologischen Mechanismen der Informationsverarbeitung innerhalb und zwischen Gehirnregionen war schon immer ein grundlegendes und herausforderndes Thema in den Neurowissenschaften. Weitreichende Arbeiten zum Konnektom und Transkriptom des Gehirns haben eine Grundlage für das Verständnis der Gehirnfunktion gelegt. Des Weiteren ermöglicht uns der derzeitige Fortschritt in der Aufnahmetechnik, die nicht stationäre Gehirnaktivität auf verschiedenen räumlichen und zeitlichen Skalen zu untersuchen. Wie jedoch die anatomischen Strukturen und die dynamischen Gehirnsignal effektiv zusammen wirken können, muss jedoch noch gelöst werden. Um dieses Problem anzugehen, untersuchen wir die Interpretation elektrophysiologischer Signale mit mechanistischen Modellen. In Kapitel 2 trennen wir zunächst die hochkohärenten Gehirnsignale in verschiedene Leitungsbahnen und verbinden dann die Dynamik mit synaptischen Eigenschaften. Basierend auf einem Zustandsraummodell zur Erzeugung lokaler Feldpotentiale (LFP) untersuchen wir verschiedene Vorverarbeitungsmethoden, die die Signale bestmöglich auf die synaptischen Eingangsströme ausrichten und die Trennbarkeit von leitungsbahnspezifischen Beiträgen verbessert. Die Trennung der Signalquellen ist durch das Vorverarbeitungsverfahren insbesondere während hochkohärenter Verhaltenszustände (z. B. laufen im Wachzustand) zuverlässiger. Mit zuverlässig getrennten Leitungsbahnen konnten wir die entsprechenden synaptischen Eigenschaften weiter untersuchen und die lokalen gerichteten Verbindungen im Hippocampus untersuchen. Die geschätzte synaptische Zeitkonstante und die Verbindungen der Leitungsbahnen stimmen mit etablierten anatomischen Studien überein. In Kapitel 3 untersuchen wir die Erstellung eines einfachen Modells zur Beschreibung der Impulsantwort passiver Neuronen mit detaillierter dendritischer Morphologie. Wir validieren Greensche Funktionsmethoden basierend auf kompartimentierten Modellen, indem wir sie mit numerischen Simulationen und analytischen Lösungen des kontinuierlichen Membranpotentials von Neuronen vergleichen. Ein parametrisiertes Modell, das auf der laminaren Greenschen Funktion basiert, wird weiterentwickelt. Es hilft dabei, die anatomischen Eigenschaften - die Verteilung des Eingangsstroms und die Zellposition - aus ihren raumzeitlichen Reaktionsmustern abzuleiten. Die Auswirkung der Zellposition und des Templates werden untersucht. Basierend auf dem Modell aus Kapitel 3 verwenden wir in Kapitel 4 das biophysikalisch mögliche Profil der Impulsantwort, um die Quellentrennung im Frequenzbereich festzulegen. Die Komponenten verschiedener Frequenzen werden nach derselben latenten Eingangsverteilungen geclustert. Die Quellentrennung in besser getrennten Frequenzbereichen derselben Leitungsbahn hilft bei der Quelltrennung in anderen stark kontaminierten Frequenzbereichen. Die Optimierung wird als probabilistisches Modell formuliert, um sowohl die Negentropie als auch die räumliche Wahrscheinlichkeit zu maximieren. Ähnlich wie die Dipolnäherungen für EEG-Signale bietet die Greensche Funktionsmethode eine effektive Annäherung, um biologisch mögliche raumzeitliche Muster zu erfassen, und hilft, die Quellen zu trennen. Wir haben die Methode an realen Daten mit optogenetischer Stimulation validiert. Im Kapitel 5 trennen wir weiter die Fernfeldsignale von den Signalen der lokalen Leitungsbahnen nach ihren physiologischen Eigenschaften. Wir schlagen eine Methode vor, die es erlaubt, Fernfeld-Signalkomponenten zuverlässig von lokaler Aktivitaet zu trennen und automatisch zu erkennen. Es wird eine Toolbox bereitgestellt, die EMG-Artefakte entfernt und die bereinigten Signale bewertet. In Ableitungen von freilaufenden Tieren zeigen wir, dass EMG-Artefakte die Erkennung von hochfrequenten Oszillationen beeintraechtigt, aber nach der Bereinigung des EMG-Signals erkannt werden kann. Insgesamt untersucht diese Dissertation mehrere Möglichkeiten die elektrischen Feldpotentiale neuronaler Aktivität unter Einbeziehung neurophysiologischen Wissens zu modellieren und zu verstehen

Oscillatory architecture of memory circuits

The coordinated activity between remote brain regions underlies cognition and memory function. Although neuronal oscillations have been proposed as a mechanistic substrate for the coordination of information transfer and memory consolidation during sleep, little is known about the mechanisms that support the widespread synchronization of brain regions and the relationship of neuronal dynamics with other bodily rhythms, such as breathing. During exploratory behavior, the hippocampus and the prefrontal cortex are organized by theta oscillations, known to support memory encoding and retrieval, while during sleep the same structures are dominated by slow oscillations that are believed to underlie the consolidation of recent experiences. The expression of conditioned fear and extinction memories relies on the coordinated activity between the mPFC and the basolateral amygdala (BLA), a neuronal structure encoding associative fear memories. However, to date, the mechanisms allowing this long-range network synchronization of neuronal activity between the mPFC and BLA during fear behavior remain virtually unknown. Using a combination of extracellular recordings and open- and closed-loop optogenetic manipulations, we investigated the oscillatory and coding mechanisms mediating the organization and coupling of the limbic circuit in the awake and asleep brain, as well as during memory encoding and retrieval. We found that freezing, a behavioral expression of fear, is tightly associated with an internally generated brain state that manifests in sustained 4Hz oscillatory dynamics in prefrontal-amygdala circuits. 4Hz oscillations accurately predict the onset and termination of the freezing state. These oscillations synchronize prefrontal-amygdala circuits and entrain neuronal activity to dynamically regulate the development of neuronal ensembles. This enables the precise timing of information transfer between the two structures and the expression of fear responses. Optogenetic induction of prefrontal 4Hz oscillations promotes freezing behavior and the formation of long-lasting fear memory, while closed-loop phase specific manipulations bidirectionally modulate fear expression. Our results unravel a physiological signature of fear memory and identify a novel internally generated brain state, characterized by 4Hz oscillations. This oscillation enables the temporal coordination and information transfer in the prefrontal-amygdala circuit via a phase-specific coding mechanism, facilitating the encoding and expression of fear memory. In the search for the origin of this oscillation, we focused our attention on breathing, the most fundamental and ubiquitous rhythmic activity in life. Using large-scale extracellular recordings from a number of structures, including the medial prefrontal cortex, hippocampus, thalamus, amygdala and nucleus accumbens in mice we identified and characterized the entrainment by breathing of a host of network dynamics across the limbic circuit. We established that fear-related 4Hz oscillations are a state-specific manifestation of this cortical entrainment by the respiratory rhythm. We characterized the translaminar and transregional profile of this entrainment and demonstrated a causal role of breathing in synchronizing neuronal activity and network dynamics between these structures in a variety of behavioral scenarios in the awake and sleep state. We further revealed a dual mechanism of respiratory entrainment, in the form of an intracerebral corollary discharge that acts jointly with an olfactory reafference to coordinate limbic network dynamics, such as hippocampal ripples and cortical UP and DOWN states, involved in memory consolidation. Respiration provides a perennial stream of rhythmic input to the brain. In addition to its role as the condicio sine qua non for life, here we provide evidence that breathing rhythm acts as a global pacemaker for the brain, providing a reference signal that enables the integration of exteroceptive and interoceptive inputs with the internally generated dynamics of the hippocampus and the neocortex. Our results highlight breathing, a perennial rhythmic input to the brain, as an oscillatory scaffold for the functional coordination of the limbic circuit, enabling the segregation and integration of information flow across neuronal networks

Diffusion and Perfusion MRI in Paediatric Posterior Fossa Tumours

Brain tumours in children frequently occur in the posterior fossa. Most undergo surgical resection, after which up to 25% develop cerebellar mutism syndrome (CMS), characterised by mutism, emotional lability and cerebellar motor signs; these typically improve over several months. This thesis examines the application of diffusion (dMRI) and arterial spin labelling (ASL) perfusion MRI in children with posterior fossa tumours. dMRI enables non-invasive in vivo investigation of brain microstructure and connectivity by a computational process known as tractography. The results of a unique survey of British neurosurgeons’ attitudes towards tractography are presented, demonstrating its widespread adoption and numerous limitations. State-of-the-art modelling of dMRI data combined with tractography is used to probe the anatomy of cerebellofrontal tracts in healthy children, revealing the first evidence of a topographic organization of projections to the frontal cortex at the superior cerebellar peduncle. Retrospective review of a large institutional series shows that CMS remains the most common complication of posterior fossa tumour resection, and that surgical approach does not influence surgical morbidity in this cohort. A prospective case-control study of children with posterior fossa tumours treated at Great Ormond Street Hospital is reported, in which children underwent longitudinal MR imaging at three timepoints. A region-of-interest based approach did not reveal any differences in dMRI metrics with respect to CMS status. However, the candidate also conducted an analysis of a separate retrospective cohort of medulloblastoma patients at Stanford University using an automated tractography pipeline. This demonstrated, in unprecedented spatiotemporal detail, a fine-grained evolution of changes in cerebellar white matter tracts in children with CMS. ASL studies in the prospective cohort showed that following tumour resection, increases in cortical cerebral blood flow were seen alongside reductions in blood arrival time, and these effects were modulated by clinical features of hydrocephalus and CMS. The results contained in this thesis are discussed in the context of the current understanding of CMS, and the novel anatomical insights presented provide a foundation for future research into the condition

Two photon interrogation of hippocampal subregions CA1 and CA3 during spatial behaviour

The hippocampus is crucial for spatial navigation and episodic memory formation. Hippocampal place cells exhibit spatially selective activity within an environment and form the neural basis of a cognitive map of space which supports these mnemonic functions. Hebb’s (1949) postulate regarding the creation of cell assemblies is seen as the pre-eminent model of learning in neural systems. Investigating changes to the hippocampal representation of space during an animal’s exploration of its environment provides an opportunity to observe Hebbian learning at the population and single cell level. When exploring new environments animals form spatial memories that are updated with experience and retrieved upon re-exposure to the same environment, but how this is achieved by different subnetworks in hippocampal CA1 and CA3, and how these circuits encode distinct memories of similar objects and events remains unclear. To test these ideas, we developed an experimental strategy and detailed protocols for simultaneously recording from CA1 and CA3 populations with 2P imaging. We also developed a novel all-optical protocol to simultaneously activate and record from ensembles of CA3 neurons. We used these approaches to show that targeted activation of CA3 neurons results in an increasing excitatory amplification seen only in CA3 cells when stimulating other CA3 cells, and not in CA1, perhaps reflecting the greater number of recurrent connections in CA3. To probe hippocampal spatial representations, we titrated input to the network by morphing VR environments during spatial navigation to assess the local CA3 as well as downstream CA1 responses. To this end, we found CA1 and CA3 neural population responses behave nonlinearly, consistent with attractor dynamics associated with the two stored representations. We interpret our findings as supporting classic theories of Hebbian learning and as the beginning of uncovering the relationship between hippocampal neural circuit activity and the computations implemented by their dynamics. Establishing this relationship is paramount to demystifying the neural underpinnings of cognition

Disease progression and genetic risk factors in the primary tauopathies

The primary tauopathies are a group of progressive neurodegenerative diseases within the frontotemporal lobar degeneration spectrum (FTLD) characterised by the accumulation of misfolded, hyperphosphorylated microtubule-associated tau protein (MAPT) within neurons and glial cells. They can be classified according to the underlying ratio of three-repeat (3R) to four-repeat (4R) tau and include Pick’s disease (PiD), which is the only 3R tauopathy, and the 4R tauopathies the most common of which are progressive supranuclear palsy (PSP) and corticobasal degeneration (CBD). There are no disease modifying therapies currently available, with research complicated by the wide variability in clinical presentations for each underlying pathology, with presentations often overlapping, as well as the frequent occurrence of atypical presentations that may mimic other non-FTLD pathologies. Although progress has been made in understanding the genetic contribution to disease risk in the more common 4R tauopathies (PSP and CBD), very little is known about the genetics of the 3R tauopathy PiD. There are two broad aims to this thesis; firstly, to use data-driven generative models of disease progression to try and more accurately stage and subtype patients presenting with PSP and corticobasal syndrome (CBS, the most common presentation of CBD), and secondly to identify genetic drivers of disease risk and progression in PiD. Given the rarity of these disorders, as part of this PhD I had to assemble two large cohorts through international collaboration, the 4R tau imaging cohort and the Pick’s disease International Consortium (PIC), to build large enough sample sizes to enable the required analyses. In Chapter 3 I use a probabilistic event-based modelling (EBM) approach applied to structural MRI data to determine the sequence of brain atrophy changes in clinically diagnosed PSP - Richardson syndrome (PSP-RS). The sequence of atrophy predicted by the model broadly mirrors the sequential spread of tau pathology in PSP post-mortem staging studies, and has potential utility to stratify PSP patients on entry into clinical trials based on disease stage, as well as track disease progression. To better characterise the spatiotemporal heterogeneity of the 4R tauopathies, I go on to use Subtype and Stage Inference (SuStaIn), an unsupervised machine algorithm, to identify population subgroups with distinct patterns of atrophy in PSP (Chapter 4) and CBS (Chapter 5). The SuStaIn model provides data-driven evidence for the existence of two spatiotemporal subtypes of atrophy in clinically diagnosed PSP, giving insights into the relationship between pathology and clinical syndrome. In CBS I identify two distinct imaging subtypes that are differentially associated with underlying pathology, and potentially a third subtype that if confirmed in a larger dataset may allow the differentiation of CBD from both PSP and AD pathology using a baseline MRI scan. In Chapter 6 I investigate the association between the MAPT H1/H2 haplotype and PiD, showing for the first time that the H2 haplotype, known to be strongly protective against developing PSP or CBD, is associated with an increased risk of PiD. This is an important finding and has implications for the future development of MAPT isoform-specific therapeutic strategies for the primary tauopathies. In Chapter 7 I perform the first genome wide association study (GWAS) in PiD, identifying five genomic loci that are nominally associated with risk of disease. The top two loci implicate perturbed GABAergic signalling (KCTD8) and dysregulation of the ubiquitin proteosome system (TRIM22) in the pathogenesis of PiD. In the final chapter (Chapter 8) I investigate the genetic determinants of survival in PiD, by carrying out a Cox proportional hazards genome wide survival study (GWSS). I identify a genome-wide significant association with survival on chromosome 3, within the NLGN1 gene. which encodes a synaptic scaffolding protein located at the neuronal pre-synaptic membrane. Loss of synaptic integrity with resulting dysregulation of synaptic transmission leading to increased pathological tau accumulation is a plausible mechanism though which NLGN1 dysfunction could impact on survival in PiD

Visual Cortex

The neurosciences have experienced tremendous and wonderful progress in many areas, and the spectrum encompassing the neurosciences is expansive. Suffice it to mention a few classical fields: electrophysiology, genetics, physics, computer sciences, and more recently, social and marketing neurosciences. Of course, this large growth resulted in the production of many books. Perhaps the visual system and the visual cortex were in the vanguard because most animals do not produce their own light and offer thus the invaluable advantage of allowing investigators to conduct experiments in full control of the stimulus. In addition, the fascinating evolution of scientific techniques, the immense productivity of recent research, and the ensuing literature make it virtually impossible to publish in a single volume all worthwhile work accomplished throughout the scientific world. The days when a single individual, as Diderot, could undertake the production of an encyclopedia are gone forever. Indeed most approaches to studying the nervous system are valid and neuroscientists produce an almost astronomical number of interesting data accompanied by extremely worthy hypotheses which in turn generate new ventures in search of brain functions. Yet, it is fully justified to make an encore and to publish a book dedicated to visual cortex and beyond. Many reasons validate a book assembling chapters written by active researchers. Each has the opportunity to bind together data and explore original ideas whose fate will not fall into the hands of uncompromising reviewers of traditional journals. This book focuses on the cerebral cortex with a large emphasis on vision. Yet it offers the reader diverse approaches employed to investigate the brain, for instance, computer simulation, cellular responses, or rivalry between various targets and goal directed actions. This volume thus covers a large spectrum of research even though it is impossible to include all topics in the extremely diverse field of neurosciences