Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement Learning

Reinforcement learning (RL) provides an influential characterization of the brain's mechanisms for learning to make advantageous choices. An important problem, though, is how complex tasks can be represented in a way that enables efficient learning. We consider this problem through the lens of spatial navigation, examining how two of the brain's location representations—hippocampal place cells and entorhinal grid cells—are adapted to serve as basis functions for approximating value over space for RL. Although much previous work has focused on these systems' roles in combining upstream sensory cues to track location, revisiting these representations with a focus on how they support this downstream decision function offers complementary insights into their characteristics. Rather than localization, the key problem in learning is generalization between past and present situations, which may not match perfectly. Accordingly, although neural populations collectively offer a precise representation of position, our simulations of navigational tasks verify the suggestion that RL gains efficiency from the more diffuse tuning of individual neurons, which allows learning about rewards to generalize over longer distances given fewer training experiences. However, work on generalization in RL suggests the underlying representation should respect the environment's layout. In particular, although it is often assumed that neurons track location in Euclidean coordinates (that a place cell's activity declines “as the crow flies” away from its peak), the relevant metric for value is geodesic: the distance along a path, around any obstacles. We formalize this intuition and present simulations showing how Euclidean, but not geodesic, representations can interfere with RL by generalizing inappropriately across barriers. Our proposal that place and grid responses should be modulated by geodesic distances suggests novel predictions about how obstacles should affect spatial firing fields, which provides a new viewpoint on data concerning both spatial codes

DTD 5 ARTICLE IN PRESS 2005 Special issue

A model of STDP based on spatially and temporally local information: Derivation and combination with gated decay

Special Issue

We investigated the importance of hippocampal theta oscillations and the significance of phase differences of theta modulation in the cortical regions that are involved in goal-directed spatial navigation. Our models used representations of entorhinal cortex layer III (ECIII), hippocampus and prefrontal cortex (PFC) to guide movements of a virtual rat in a virtual environment. The model encoded representations of the environment through long-term potentiation of excitatory recurrent connections between sequentially spiking place cells in ECIII and CA3. This encoding required buffering of place cell activity, which was achieved by a short-term memory (STM) in EC that was regulated by theta modulation and allowed synchronized reactivation with encoding phases in ECIII and CA3. Inhibition at a specific theta phase deactivated the oldest item in the buffer when new input was presented to a full STM buffer. A 1808 phase difference separated retrieval and encoding in ECIII and CA3, which enabled us to simulate data on theta phase precession of place cells. Retrieval of known paths was elicited in ECIII by input at the retrieval phase from PFC working memory for goal location, requiring strict theta phase relationships with PFC. Known locations adjacent to the virtual rat were retrieved in CA3. Together, input from ECIII and CA3 activated predictive spiking in cells in CA1 for the next desired place on a shortest path to a goal. Consistent with data, place cell activity in CA1 and CA3 showed smaller place fields than in ECIII