Neural basis of route-planning and goal-coding during flexible navigation

Animals and humans are remarkable in their ability to flexibly adapt to changes in their surroundings. Navigational flexibility may take many forms and in this thesis we investigate its neural and behavioral underpinnings using a variety of methods and tasks tailored to each specific research aim. These methods include functional resonance magnetic imaging (fMRI), freely moving virtual reality, desktop virtual reality, large-scale online testing, and computational modelling. First, we reanalysed previously collected rodent data in the lab to better under- stand behavioural bias that may occur during goal-directed navigation tasks. Based on finding some biases we designed a new approach of simulating results on maze configurations prior to data collection to select the ideal mazes for our task. In a parallel line of methods development, we designed a freely moving navigation task using large-scale wireless virtual reality in a 10x10 space. We compared human behaviour to that of a select number of reinforcement learning agents to investigate the feasibility of computational modelling approaches to freely moving behaviour. Second, we further developed our new approach of simulating results on maze configuration to design a novel spatial navigation task used in a parallel experiment in both rats and humans. We report the human findings using desktop virtual reality and fMRI. We identified a network of regions including hippocampal, caudate nu- cleus, and lateral orbitofrontal cortex involvement in learning hidden goal locations. We also identified a positive correlation between Euclidean goal distance and brain activity in the caudate nucleus during ongoing navigation. Third, we developed a large online testing paradigm to investigate the role of home environment on wayfinding ability. We extended previous reports that street network complexity is beneficial in improving wayfinding ability as measured using a previously reported virtual navigation game, Sea Hero Quest, as well as in a novel virtual navigation game, City Hero Quest. We also report results of a navigational strategies questionnaire that highlights differences of growing up inside and outside cities in the United States and how this relates to wayfinding ability. Fourth, we investigate route planning in a group of expert navigators, licensed London taxi drivers. We designed a novel mental route planning task, probing 120 different routes throughout the extensive street network of London. We find hip- pocampal and retrosplenial involvement in route planning. We also identify the frontopolar cortex as one of several brain regions parametrically modulated by plan- ning demand. Lastly, I summarize the findings from these studies and how they all come to provide different insights into our remarkable ability to flexibly adapt to naviga- tional challenges in our environment

A computational model of cortical-striatal mediation of speed-accuracy tradeoff and habit formation emerging from anatomical gradients in dopamine physiology and reinforcement learning

Decision making – committing to a single action from a plethora of viable alternatives – is a necessity for all motile creatures, each moving a single body to many possible destinations. Some decisions are better than others. For example, to a rat deciding between one path that will bring it to a piece of cheese and another that will bring it to the jaws of a cat, there is a clear reason for the rat to prefer one choice over the other. Two criteria for adjusting decision making for optimal outcome are to make decisions as accurately as possible – choose the course of action most likely to result in the preferred outcome – but also to decide as fast as possible. Because these criteria often conflict, decision making has an inherent “speed-accuracy tradeoff”. Presented here is a computational neural model of decision making, which incorporates neurobiological design principles that optimize this tradeoff via reward-guided transfers of control between two sensory processing systems with different speed/accuracy characteristics. This model incorporates anatomical and physiological evidence that dopamine, the key neurotransmitter in reinforcement learning, has varying effects in different sub-regions of the basal ganglia, a subcortical structure that interfaces with the neocortex to control behavior. Based on the observed differences between these sub-regions, the model proposes that gradual adaptations of synaptic links by reinforcement learning signals lead to rapid changes in the speed and accuracy of decision making, by assigning control of behavior to alternative cortical representations. Chapter one draws conceptual links from experimental data to the design of the proposed model. Chapter two applies the model to speed-accuracy tradeoffs and habit formation by simulating forced-choice paradigms. Several robust behavioral phenomena are replicated. By isolating reinforcement learning factors that control the speed and depth of habit formation, the model can help explain why all substances that strongly and synergistically affect such factors share a high potential for habit formation, or habit abatement. To illustrate such potential applications of the current model, chapter three investigates effects of varying model parameters in accord with the known neurochemical effects of some major habit-forming substances, such as cocaine and ethanol

Role of the hippocampus in goal representation : Insights from behavioural and electrophysiological approaches

The hippocampus plays an important role in spatial cognition, as supported by the location-specific firing of hippocampal place cells. In random foraging tasks, each place cell fires at a specific position (‘place field’) while other hippocampal pyramidal neurons remain silent. A recent study evidenced a reliable extra-field activity in most CA1 place cells of rats waiting for reward delivery in an uncued goal zone. While the location-specific activity of place cells is thought to underlie a flexible representation of space, the nature of this goal-related signal remains unclear. To test whether hippocampal goal-related activity reflects a representation of goal location or a reward-related signal, we designed a two-goal navigation task in which rats were free to choose between two uncued spatial goals to receive a reward. The magnitude of reward associated to each goal zone was modulated, therefore changing the goal value. We recorded CA1 and CA3 unit activity from rats performing this task. Behaviourally, rats were able to remember each goal location and flexibly adapt their choices to goal values. Electrophysiological data showed that a large majority of CA1-CA3 place and silent cells expressed goal-related activity. This activity was independent from goal value and rats’ behavioural choices. Importantly, a large proportion of cells expressed a goal-related activity at one goal zone only. Altogether, our findings suggest that the hippocampus processes and stores relevant information about the spatial characteristics of the goal. This goal representation could be used in cooperation with structures involved in decision-making to optimise goal-directed navigation

Spatial learning and navigation in the rat:a biomimetic model

Animals behave in different ways depending on the specific task they are required to solve. In certain cases, if a cue marks the goal location, they can rely on simple stimulusresponse associations. In contrast, other tasks require the animal to be endowed with a representation of space. Such a representation (i.e. cognitive map) allows the animal to locate itself within a known environment and perform complex target-directed behaviour. In order to efficiently perform, the animal not only should be able to exhibit these types of behaviour, but it should be able to select which behaviour is the most appropriate at any given task conditions. Neurophysiological and behavioural experiments provide important information on how such processes may take place in the rodent's brain. Specifically, place- and orientation sensitive cells in the rat Hippocampus have been interpreted as a neural substrate for spatial abilities related to the theory of the cognitive map proposed in the late 1940s by Tolman. Moreover, recent dissociation experiments using selectively located lesions, as well as pharmacological studies have shown that different brain regions may be involved in different types of behaviour. Accordingly, one memory system involving the hippocampus and the ventral striatum would be responsible for cognitive navigation, while navigation based on stimulus-response associations would be mediated by the dorsolateral striatum. Based on these studies, the aim of this work is to develop a neural network model of the spatial abilities of the rat. The model, based on functional properties and anatomical inter-connections of the brain areas involved in spatial learning should be able to establish a distributed representation of space composed of place-sensitive units. Such a representation takes into account both internal and external sensory information, and the model reproduces physiological properties of place cells such as changes in their directional dependence. Moreover, the spatial representation may be used to perform cognitive navigation. Modelled place cells drive an extra-hippocampal population of action-coding cells, allowing the establishment of place-response associations. These associations encoded in synaptic connections between place- and action-cells are modified by means of reinforcement learning. In a similar way, simple sensory input can be used to establish stimulus-response associations. These associations are encoded in a different set of action cells which corresponds to a different neural substrate encoding for non-cognitive navigation strategies (i.e. taxon or praxic). Both cognitive and non-cognitive navigation strategies compete for action control to determine the actual behaviour of the agent. Tests of the performance of the model show that it is able to establish a representation of space, and modelled place cells reproduce some physiological properties of their biological counterparts. Furthermore, the model reproduces goal-based behaviour based on both cognitive and non-cognitive strategies as well as behaviour in conflicting situations reported in experimental studies in animals

Hippocampal predictive maps of an uncertain world

Humans and other animals can solve a wide variety of decision-making problems with remarkable flexibility. This flexibility is thought to derive from an internal model of the world, or ‘cognitive map’, used to predict the future and plan actions accordingly. A recent theoretical proposal suggests that the hippocampus houses a representation of long-run state expectancies. These “successor representations” (SRs) occupy a middle ground between model-free and model-based reinforcement learning strategies. However, it is not clear whether SRs can explain hippocampal contributions to spatial and model-based behaviour, nor how a putative hippocampal SR might interface with striatal learning mechanisms. More generally, it is not clear how the predictive map should encode uncertainty, and how an uncertainty-augmented predictive map modifies our experimental predictions for animal behaviour. In the first part of this thesis, I investigated whether viewing the hippocampus as an SR can explain experiments contrasting hippocampal and dorsolateral striatal contributions to behaviour in spatial and non-spatial tasks. To do this, I modelled the hippocampus as an SR and DLS as model-free reinforcement learning, combining their outputs via their relative reliability as a proxy for uncertainty. Current SR models do not formally address uncertainty. Therefore I extended the learning of SRs by temporal differences to include managing uncertainty in new observations versus existing knowledge. I generalise this approach to a multi-task setting using a Bayesian nonparametric switching Kalman Filter, allowing the model to learn and maintain multiple task-specific SR maps and infer which one to use at any moment based on the observations. I show that this Bayesian SR model captures animal behaviour in tasks which require contextual memory and generalisation. In conclusion, I consider how the hippocampal contribution to behaviour can be considered as a predictive map when adapted to take account of uncertainty and combined with other behavioural controllers

A biologically inspired meta-control navigation system for the Psikharpax rat robot

A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e. g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment-recognized as new contexts-and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics

Acute Stress Exposure and Expression of Instrumentally Conditioned Financial Preferences: An fMRI Study

Recent research suggests acute stress exposure is associated with increased habit-based over goal-oriented decision making (e.g., Schwabe & Wolf, 2011). The current study examined whether acute stress promotes the expression of simple financial preferences “overtrained” to the point of habit in the face of a changing environment where said preferences were later rendered non-optimal. Over three days participants (N = 28) learned to discriminate between visual stimuli probabilistically associated with monetary gains or losses and made decisions between stimuli with real financial outcomes. On the fourth day after exposure to either an acute stressor or control procedure participants performed the same tasks during fMRI scanning, including a related task in which monetary values associated with the same stimuli were altered. Choice and fMRI data, psychophysiological measures and salivary cortisol were collected. Participants in both groups successfully made optimal decisions between stimuli on Days 1 to 3 (reaching asymptote on Day 2). During fMRI scanning after stimuli values were altered stressed participants made significantly more decisions consistent with original stimuli values, although these decisions were now financially detrimental, than did non-stressed participants. Thus, stressed participants made decisions more consistent with their overtrained (i.e., habit-based) preferences. In the control group, differential levels of BOLD activation, relative to stimulus valence, were observed in regions associated with goal-directed (i.e., caudate and prefrontal cortex) and habit-based (i.e., putamen) behaviors during both overtrained and novel stimulus-outcome pairings. In the acute stress group, similar differential BOLD activation was limited to the putamen and was only observed for overtrained pairings. During the decision-making portion of the task, increased BOLD activation was observed in the dorsal anterior cingulate cortex and insula for incorrect relative to correct responses in both groups. Further, alterations in dorsolateral prefrontal and entorhinal cortex suggest some stress-related impairment of executive control of memory. The current study adds to research that demonstrates a dual-process of decision-making and the propensity to resort to habitual behavior after exposure to acute stress. Further, these findings suggest stress-induced neural changes take place during both the learning and recall of reward-related information used in decision-making