The Cognitive Architecture of Spatial Navigation: Hippocampal and Striatal Contributions

Spatial navigation can serve as a model system in cognitive neuroscience, in which specific neural representations, learning rules, and control strategies can be inferred from the vast experimental literature that exists across many species, including humans. Here, we review this literature, focusing on the contributions of hippocampal and striatal systems, and attempt to outline a minimal cognitive architecture that is consistent with the experimental literature and that synthesizes previous related computational modeling. The resulting architecture includes striatal reinforcement learning based on egocentric representations of sensory states and actions, incidental Hebbian association of sensory information with allocentric state representations in the hippocampus, and arbitration of the outputs of both systems based on confidence/uncertainty in medial prefrontal cortex. We discuss the relationship between this architecture and learning in model-free and model-based systems, episodic memory, imagery, and planning, including some open questions and directions for further experiments

Striatal and hippocampal contributions to flexible navigation in rats and humans.

The hippocampus has been firmly established as playing a crucial role in flexible navigation. Recent evidence suggests that dorsal striatum may also play an important role in such goal-directed behaviour in both rodents and humans. Across recent studies, activity in the caudate nucleus has been linked to forward planning and adaptation to changes in the environment. In particular, several human neuroimaging studies have found the caudate nucleus tracks information traditionally associated with that by the hippocampus. In this brief review, we examine this evidence and argue the dorsal striatum encodes the transition structure of the environment during flexible, goal-directed behaviour. We highlight that future research should explore the following: (1) Investigate neural responses during spatial navigation via a biophysically plausible framework explained by reinforcement learning models and (2) Observe the interaction between cortical areas and both the dorsal striatum and hippocampus during flexible navigation

Functional MRI investigations of overlapping spatial memories and flexible decision-making in humans

Thesis (Ph.D.)--Boston UniversityResearch in rodents and computational modeling work suggest a critical role for the hippocampus in representing overlapping memories. This thesis tested predictions that the hippocampus is important in humans for remembering overlapping spatial events, and that flexible navigation of spatial routes is supported by key prefrontal and striatal structures operating in conjunction with the hippocampus. The three experiments described in this dissertation used functional magnetic resonance imaging (fMRI) in healthy young people to examine brain activity during context-dependent navigation of virtual maze environments. Experiment 1 tested whether humans recruit the hippocampus and orbitofrontal cortex to successfully retrieve well-learned overlapping spatial routes. Participants navigated familiar virtual maze environments during fMRI scanning. Brain activity for flexible retrieval of overlapping spatial memories was contrasted with activity for retrieval of distinct non-overlapping memories. Results demonstrate the hippocampus is more strongly recruited for planning and retrieval of overlapping routes than non-overlapping routes, and the orbitofrontal cortex is recruited specifically for context-dependent navigational decisions. Experiment 2 examined whether the hippocampus, orbitofrontal cortex, and striatum interact cooperatively to support flexible navigation of overlapping routes. Using a functional connectivity analysis of fMRI data, we compared interactions between these structures during virtual navigation of overlapping and non-overlapping mazes. Results demonstrate the hippocampus interacts with the caudate more strongly for navigating overlapping than non-overlapping routes. Both structures cooperate with the orbitofrontal cortex specifically during context-dependent decision points, suggesting the orbitofrontal cortex mediates translation of contextual information into the flexible selection of behavior. Experiment 3 examined whether the hippocampus and caudate contribute to forming context-dependent memories. fMRI activity for learning new virtual mazes which overlap with familiar routes was compared with activity for learning completely distinct routes. Results demonstrate both the hippocampus and caudate are preferentially recruited for learning mazes which overlap with existing route memories. Furthermore, both areas update their responses to familiar route memories which become context-dependent, suggesting complementary roles in both learning and updating overlapping representations. Together, these studies demonstrate that navigational decisions based on overlapping representations rely on a network incorporating hippocampal function with the evaluation and selection of behavior in the prefrontal cortex and striatum

The well-worn route revisited: Striatal and hippocampal system contributions to route learning in human navigation

Parallel spatial memory systems theory posits that there are two types of memory system. One is a flexible, cognitive mapping system subserved by the hippocampal formation, and the other is a system centred on the striatum based on reinforcement learning principles where specific stimuli are associated with rewarded actions (O’Keefe & Nadel, 1978; White & McDonald, 2002). More recently, Khamassi & Humphries (2012) have argued that the division between model-based and model-free spatial learning is a better predictor of whether hippocampal or striatal systems will be recruited, with hippocampal systems associated with model-based responding and striatal systems with model-free responding. Model-free decision-making occurs when responding is based on average reward history associated with a particular cue-action pairing, whereas model-based decision-making allows knowledge of outcomes from previous learning history to be represented. We sought to test these theories by asking participants (N = 24) to navigate within a virtual environment through a previously learned, 9-junction route with distinctive landmarks at each junction, while undergoing functional magnetic resonance imaging. In critical conflict probe trials, a landmark was presented out of sequence such that following the usual sequence of actions would generate an opposite response to following the learned individual landmark-action association, now out of sequence. Participants that made sequence-based responses had higher parahippocampal activations relative to participants that made responses based on the individual landmark-action association, a result that would be predicted by the need to recruit model-based systems to make a sequence-based response. Parallel spatial memory systems theory would not predict hippocampal formation recruitment for either response in the conflict probe, because no cognitive mapping is required when following a prescribed route. In longer probe trials where participants were able to plan a sequence of responses, striatal systems were recruited (caudate and putamen) suggesting a role for striatum in action chunking

Neural basis of route-planning and goal-coding during flexible navigation

Animals and humans are remarkable in their ability to flexibly adapt to changes in their surroundings. Navigational flexibility may take many forms and in this thesis we investigate its neural and behavioral underpinnings using a variety of methods and tasks tailored to each specific research aim. These methods include functional resonance magnetic imaging (fMRI), freely moving virtual reality, desktop virtual reality, large-scale online testing, and computational modelling. First, we reanalysed previously collected rodent data in the lab to better under- stand behavioural bias that may occur during goal-directed navigation tasks. Based on finding some biases we designed a new approach of simulating results on maze configurations prior to data collection to select the ideal mazes for our task. In a parallel line of methods development, we designed a freely moving navigation task using large-scale wireless virtual reality in a 10x10 space. We compared human behaviour to that of a select number of reinforcement learning agents to investigate the feasibility of computational modelling approaches to freely moving behaviour. Second, we further developed our new approach of simulating results on maze configuration to design a novel spatial navigation task used in a parallel experiment in both rats and humans. We report the human findings using desktop virtual reality and fMRI. We identified a network of regions including hippocampal, caudate nu- cleus, and lateral orbitofrontal cortex involvement in learning hidden goal locations. We also identified a positive correlation between Euclidean goal distance and brain activity in the caudate nucleus during ongoing navigation. Third, we developed a large online testing paradigm to investigate the role of home environment on wayfinding ability. We extended previous reports that street network complexity is beneficial in improving wayfinding ability as measured using a previously reported virtual navigation game, Sea Hero Quest, as well as in a novel virtual navigation game, City Hero Quest. We also report results of a navigational strategies questionnaire that highlights differences of growing up inside and outside cities in the United States and how this relates to wayfinding ability. Fourth, we investigate route planning in a group of expert navigators, licensed London taxi drivers. We designed a novel mental route planning task, probing 120 different routes throughout the extensive street network of London. We find hip- pocampal and retrosplenial involvement in route planning. We also identify the frontopolar cortex as one of several brain regions parametrically modulated by plan- ning demand. Lastly, I summarize the findings from these studies and how they all come to provide different insights into our remarkable ability to flexibly adapt to naviga- tional challenges in our environment

Hippocampal predictive maps of an uncertain world

Humans and other animals can solve a wide variety of decision-making problems with remarkable flexibility. This flexibility is thought to derive from an internal model of the world, or ‘cognitive map’, used to predict the future and plan actions accordingly. A recent theoretical proposal suggests that the hippocampus houses a representation of long-run state expectancies. These “successor representations” (SRs) occupy a middle ground between model-free and model-based reinforcement learning strategies. However, it is not clear whether SRs can explain hippocampal contributions to spatial and model-based behaviour, nor how a putative hippocampal SR might interface with striatal learning mechanisms. More generally, it is not clear how the predictive map should encode uncertainty, and how an uncertainty-augmented predictive map modifies our experimental predictions for animal behaviour. In the first part of this thesis, I investigated whether viewing the hippocampus as an SR can explain experiments contrasting hippocampal and dorsolateral striatal contributions to behaviour in spatial and non-spatial tasks. To do this, I modelled the hippocampus as an SR and DLS as model-free reinforcement learning, combining their outputs via their relative reliability as a proxy for uncertainty. Current SR models do not formally address uncertainty. Therefore I extended the learning of SRs by temporal differences to include managing uncertainty in new observations versus existing knowledge. I generalise this approach to a multi-task setting using a Bayesian nonparametric switching Kalman Filter, allowing the model to learn and maintain multiple task-specific SR maps and infer which one to use at any moment based on the observations. I show that this Bayesian SR model captures animal behaviour in tasks which require contextual memory and generalisation. In conclusion, I consider how the hippocampal contribution to behaviour can be considered as a predictive map when adapted to take account of uncertainty and combined with other behavioural controllers

Computational Properties of the Hippocampus Increase the Efficiency of Goal-Directed Foraging through Hierarchical Reinforcement Learning

Sherpa Romeo green journal; open accessThe mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals’ ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a flexible MBRL mechanism for spatial navigation that is computationally efficient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We find that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efficient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deficits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efficiently solve navigation and other difficult tasks.Ye

The Mixed Instrumental Controller: Using Value of Information to Combine Habitual Choice and Mental Simulation

Instrumental behavior depends on both goal-directed and habitual mechanisms of choice. Normative views cast these mechanisms in terms of model-free and model-based methods of reinforcement learning, respectively. An influential proposal hypothesizes that model-free and model-based mechanisms coexist and compete in the brain according to their relative uncertainty. In this paper we propose a novel view in which a single Mixed Instrumental Controller produces both goal-directed and habitual behavior by flexibly balancing and combining model-based and model-free computations. The Mixed Instrumental Controller performs a cost-benefits analysis to decide whether to chose an action immediately based on the available “cached” value of actions (linked to model-free mechanisms) or to improve value estimation by mentally simulating the expected outcome values (linked to model-based mechanisms). Since mental simulation entails cognitive effort and increases the reward delay, it is activated only when the associated “Value of Information” exceeds its costs. The model proposes a method to compute the Value of Information, based on the uncertainty of action values and on the distance of alternative cached action values. Overall, the model by default chooses on the basis of lighter model-free estimates, and integrates them with costly model-based predictions only when useful. Mental simulation uses a sampling method to produce reward expectancies, which are used to update the cached value of one or more actions; in turn, this updated value is used for the choice. The key predictions of the model are tested in different settings of a double T-maze scenario. Results are discussed in relation with neurobiological evidence on the hippocampus – ventral striatum circuit in rodents, which has been linked to goal-directed spatial navigation