Skip to main content
Article thumbnail
Location of Repository

The Influence of Markov Decision Process Structure on the Possible Strategic Use of Working Memory and Episodic Memory

By Eric A. Zilli and Michael E. Hasselmo


Researchers use a variety of behavioral tasks to analyze the effect of biological manipulations on memory function. This research will benefit from a systematic mathematical method for analyzing memory demands in behavioral tasks. In the framework of reinforcement learning theory, these tasks can be mathematically described as partially-observable Markov decision processes. While a wealth of evidence collected over the past 15 years relates the basal ganglia to the reinforcement learning framework, only recently has much attention been paid to including psychological concepts such as working memory or episodic memory in these models. This paper presents an analysis that provides a quantitative description of memory states sufficient for correct choices at specific decision points. Using information from the mathematical structure of the task descriptions, we derive measures that indicate whether working memory (for one or more cues) or episodic memory can provide strategically useful information to an agent. In particular, the analysis determines which observed states must be maintained in or retrieved from memory to perform these specific tasks. We demonstrate the analysis on three simplified tasks as well as eight more complex memory tasks drawn from the animal and human literature (two alternation tasks, two sequence disambiguation tasks, two non-matching tasks, the 2-back task, and the 1-2-AX task). The results of these analyses agree with results from quantitative simulations of the task reported in previous publications and provide simple indications of the memory demands of the tasks which can require far less computation than a full simulation of the task. This may provide a basis for a quantitative behavioral stoichiometry of memory tasks

Topics: Research Article
Publisher: Public Library of Science
OAI identifier:
Provided by: PubMed Central

Suggested articles


  1. (1998). A role for NMDA-receptor channels in working memory.
  2. (1993). A spiking network model of short-term active memory.
  3. (1982). A survey of Partially Observable Markov Decision Processes.
  4. (2007). Bilinearity, rules, and prefrontal cortex.
  5. (1995). Cellular basis of working memory.
  6. (2007). Context learning in the rodent hippocampus.
  7. (2002). Critical role of the hippocampus in memory for sequences of events.
  8. (1996). Ensemble codes involving hippocampal neurons are at risk during delayed performance tests.
  9. (2002). Episodic memory: From mind to brain.
  10. (2007). Exploring the role of context-dependent hippocampal activity in spatial alternation behavior.
  11. (1976). Finite Markov chains.
  12. (2006). First-in-first-out item replacement in a model of short-term memory based on persistent spiking.
  13. (2001). From Conditioning to Conscious Recollection.
  14. (2007). Gradual changes in hippocampal activity support remembering the order of events.
  15. (2006). Gradual translocation of spatial correlates of neural firing in the hippocampus toward prospective reward locations.
  16. (2005). Hippocampal mechanisms for the context-dependent retrieval of episodes.
  17. (2000). Hippocampal neurons encode information about different types of memory episodes occurring in the same location.
  18. (2006). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia.
  19. (1990). Measurement of visual sustained attention in schizophrenia using signal detection analysis and a newly developed computerized CPT task.
  20. (2001). Medial temporal and prefrontal contributions to working memory tasks with novel and familiar stimuli.
  21. (1995). Memory in the cerebral cortex.
  22. (2004). Memory systems of the brain: a brief history and current perspective.
  23. (2008). Modeling the role of working memory and episodic memory in behavioral tasks.
  24. (1996). Neural mechanisms of visual working memory in prefrontal cortex of the macaque.
  25. (1998). Planning and acting in partially observable stochastic domains.
  26. (1998). Reinforcement Learning: An Introduction.
  27. (1968). Solutions to some functional equations and their applications to characterization of probability distributions.
  28. (2007). Spatial representations of hippocampal CA1 neurons are modulated by behavioral context in a hippocampus-dependent memory task.
  29. (1997). Temporal dynamics of brain activation during a working memory task.
  30. (2002). The hippocampus and disambiguation of overlapping sequences.
  31. (2005). The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychol Rev
  32. (1994). Tulving E
  33. (2007). Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia.
  34. (2000). Working memory for letters, shapes, and locations: fMRI evidence against stimulus-based regional organization in human prefrontal cortex.
  35. (1974). Working memory.

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.