The Influence of Markov Decision Process Structure on the Possible Strategic Use of Working Memory and Episodic Memory

Researchers use a variety of behavioral tasks to analyze the effect of biological manipulations on memory function. This research will benefit from a systematic mathematical method for analyzing memory demands in behavioral tasks. In the framework of reinforcement learning theory, these tasks can be mathematically described as partially-observable Markov decision processes. While a wealth of evidence collected over the past 15 years relates the basal ganglia to the reinforcement learning framework, only recently has much attention been paid to including psychological concepts such as working memory or episodic memory in these models. This paper presents an analysis that provides a quantitative description of memory states sufficient for correct choices at specific decision points. Using information from the mathematical structure of the task descriptions, we derive measures that indicate whether working memory (for one or more cues) or episodic memory can provide strategically useful information to an agent. In particular, the analysis determines which observed states must be maintained in or retrieved from memory to perform these specific tasks. We demonstrate the analysis on three simplified tasks as well as eight more complex memory tasks drawn from the animal and human literature (two alternation tasks, two sequence disambiguation tasks, two non-matching tasks, the 2-back task, and the 1-2-AX task). The results of these analyses agree with results from quantitative simulations of the task reported in previous publications and provide simple indications of the memory demands of the tasks which can require far less computation than a full simulation of the task. This may provide a basis for a quantitative behavioral stoichiometry of memory tasks

Spike-Timing Theory of Working Memory

Working memory (WM) is the part of the brain's memory system that provides temporary storage and manipulation of information necessary for cognition. Although WM has limited capacity at any given time, it has vast memory content in the sense that it acts on the brain's nearly infinite repertoire of lifetime long-term memories. Using simulations, we show that large memory content and WM functionality emerge spontaneously if we take the spike-timing nature of neuronal processing into account. Here, memories are represented by extensively overlapping groups of neurons that exhibit stereotypical time-locked spatiotemporal spike-timing patterns, called polychronous patterns; and synapses forming such polychronous neuronal groups (PNGs) are subject to associative synaptic plasticity in the form of both long-term and short-term spike-timing dependent plasticity. While long-term potentiation is essential in PNG formation, we show how short-term plasticity can temporarily strengthen the synapses of selected PNGs and lead to an increase in the spontaneous reactivation rate of these PNGs. This increased reactivation rate, consistent with in vivo recordings during WM tasks, results in high interspike interval variability and irregular, yet systematically changing, elevated firing rate profiles within the neurons of the selected PNGs. Additionally, our theory explains the relationship between such slowly changing firing rates and precisely timed spikes, and it reveals a novel relationship between WM and the perception of time on the order of seconds

L-lysine as adjunctive treatment in patients with schizophrenia: a single-blinded, randomized, cross-over pilot study

<p>Abstract</p> <p>Background</p> <p>Accumulating evidence suggests that the brain's nitric oxide (NO) signalling system may be involved in the pathophysiology of schizophrenia and could thus constitute a novel treatment target. The study was designed to investigate the benefit of L-lysine, an amino acid that interferes with NO production, as an add-on treatment for schizophrenia.</p> <p>Methods</p> <p>L-lysine, 6 g/day, was administered to 10 patients with schizophrenia as an adjunctive to their conventional antipsychotic medication. The study was designed as a single-blinded, cross-over study where patients were randomly assigned to initial treatment with either L-lysine or placebo and screened at baseline, after four weeks when treatment was crossed over, and after eight weeks.</p> <p>Results</p> <p>L-lysine treatment caused a significant increase in blood concentration of L-lysine and was well tolerated. A significant decrease in positive symptom severity, measured by the Positive And Negative Syndrome Scale (PANSS), was detected. A certain decrease in score was also observed during placebo treatment and the effects on PANSS could not unequivocally be assigned to the L-lysine treatment. Furthermore, performance on the Wisconsin Card Sorting Test was significantly improved compared to baseline, an effect probably biased by training. Subjective reports from three of the patients indicated decreased symptom severity and enhanced cognitive functioning.</p> <p>Conclusions</p> <p>Four-week L-lysine treatment of 6 g/day caused a significant increase in blood concentration of L-lysine that was well tolerated. Patients showed a significant decrease in positive symptoms as assessed by PANSS in addition to self-reported symptom improvement by three patients. The NO-signalling pathway is an interesting, potentially new treatment target for schizophrenia; however, the effects of L-lysine need further evaluation to decide the amino acid's potentially beneficial effects on symptom severity in schizophrenia.</p> <p>Trial registration</p> <p><a href="http://www.clinicaltrials.gov/ct2/show/NCT00996242">NCT00996242</a></p

Functional magnetic resonance imaging (fMRI) of attention processes in presumed obligate carriers of schizophrenia: preliminary findings

<p>Abstract</p> <p>Background</p> <p>Presumed obligate carriers (POCs) are the first-degree relatives of people with schizophrenia who, although do not exhibit the disorder, are in direct lineage of it. Thus, this subpopulation of first-degree relatives could provide very important information with regard to the investigation of endophenotypes for schizophrenia that could clarify the often contradictory findings in schizophrenia high-risk populations. To date, despite the extant literature on schizophrenia endophenotypes, we are only aware of one other study that examined the neural mechanisms that underlie cognitive abnormalities in this group. The aim of this study was to investigate whether a more homogeneous group of relatives, such as POCs, have neural abnormalities that may be related to schizophrenia.</p> <p>Methods</p> <p>We used functional magnetic resonance imaging (fMRI) to collect blood oxygenated level dependent (BOLD) response data in six POCs and eight unrelated healthy controls while performing under conditions of sustained, selective and divided attention.</p> <p>Results</p> <p>The POCs indicated alterations in a widely distributed network of regions involved in attention processes, such as the prefrontal and temporal (including the parahippocampal gyrus) cortices, in addition to the anterior cingulate gyrus. More specifically, a general reduction in BOLD response was found in these areas compared to the healthy participants during attention processes.</p> <p>Conclusion</p> <p>These preliminary findings of decreased activity in POCs indicate that this more homogeneous population of unaffected relatives share similar neural abnormalities with people with schizophrenia, suggesting that reduced BOLD activity in the attention network may be an intermediate marker for schizophrenia.</p

Dysfunctional GABAergic inhibition in the prefrontal cortex leading to "psychotic" hyperactivation

<p>Abstract</p> <p>Background</p> <p>The GABAergic system in the brain seems to be dysfunctional in various psychiatric disorders. Many studies have suggested so far that, in schizophrenia patients, GABAergic inhibition is selectively but consistently reduced in the prefrontal cortex (PFC).</p> <p>Results</p> <p>This study used a computational model of the PFC to investigate the dynamics of the PFC circuit with and without chandelier cells and other GABAergic interneurons. The inhibition by GABAergic interneurons other than chandelier cells effectively regulated the PFC activity with rather low or modest levels of dopaminergic neurotransmission. This activity of the PFC is associated with normal cognitive functions and has an inverted-U shaped profile of dopaminergic modulation. In contrast, the chandelier cell-type inhibition affected only the PFC circuit dynamics in hyperdopaminergic conditions. Reduction of chandelier cell-type inhibition resulted in bistable dynamics of the PFC circuit, in which the upper stable state is associated with a hyperactive mode. When both types of inhibition were reduced, this hyperactive mode and the conventional inverted-U mode merged.</p> <p>Conclusion</p> <p>The results of our simulation suggest that, in schizophrenia, a reduction of GABAergic inhibition increases vulnerability to psychosis by (i) producing the hyperactive mode of the PFC with hyperdopaminergic neurotransmission by dysfunctional chandelier cells and (ii) increasing the probability of the transition to the hyperactive mode from the conventional inverted-U mode by dysfunctional GABAergic interneurons.</p

Visuospatial working memory in children and adolescents with 22q11.2 deletion syndrome; an fMRI study

22q11.2 deletion syndrome (22q11DS) is a genetic disorder associated with a microdeletion of chromosome 22q11. In addition to high rates of neuropsychiatric disorders such as schizophrenia and attention deficit hyperactivity disorder, children with 22q11DS have a specific neuropsychological profile with particular deficits in visuospatial and working memory. However, the neurobiological substrate underlying these deficits is poorly understood. We investigated brain function during a visuospatial working memory (SWM) task in eight children with 22q11DS and 13 healthy controls, using fMRI. Both groups showed task-related activation in dorsolateral prefrontal cortex (DLPFC) and bilateral parietal association cortices. Controls activated parietal and occipital regions significantly more than those with 22q11DS but there was no significant between-group difference in DLPFC. In addition, while controls had a significant age-related increase in the activation of posterior brain regions and an age-related decrease in anterior regions, the 22q11DS children showed the opposite pattern. Genetically determined differences in the development of specific brain systems may underpin the cognitive deficits in 22q11DS, and may contribute to the later development of neuropsychiatric disorders

Coherence and recurrency: maintenance, control and integration in working memory

Working memory (WM), including a ‘central executive’, is used to guide behavior by internal goals or intentions. We suggest that WM is best described as a set of three interdependent functions which are implemented in the prefrontal cortex (PFC). These functions are maintenance, control of attention and integration. A model for the maintenance function is presented, and we will argue that this model can be extended to incorporate the other functions as well. Maintenance is the capacity to briefly maintain information in the absence of corresponding input, and even in the face of distracting information. We will argue that maintenance is based on recurrent loops between PFC and posterior parts of the brain, and probably within PFC as well. In these loops information can be held temporarily in an active form. We show that a model based on these structural ideas is capable of maintaining a limited number of neural patterns. Not the size, but the coherence of patterns (i.e., a chunking principle based on synchronous firing of interconnected cell assemblies) determines the maintenance capacity. A mechanism that optimizes coherent pattern segregation, also poses a limit to the number of assemblies (about four) that can concurrently reverberate. Top-down attentional control (in perception, action and memory retrieval) can be modelled by the modulation and re-entry of top-down information to posterior parts of the brain. Hierarchically organized modules in PFC create the possibility for information integration. We argue that large-scale multimodal integration of information creates an ‘episodic buffer’, and may even suffice for implementing a central executive

Dopamine, affordance and active inference.

The role of dopamine in behaviour and decision-making is often cast in terms of reinforcement learning and optimal decision theory. Here, we present an alternative view that frames the physiology of dopamine in terms of Bayes-optimal behaviour. In this account, dopamine controls the precision or salience of (external or internal) cues that engender action. In other words, dopamine balances bottom-up sensory information and top-down prior beliefs when making hierarchical inferences (predictions) about cues that have affordance. In this paper, we focus on the consequences of changing tonic levels of dopamine firing using simulations of cued sequential movements. Crucially, the predictions driving movements are based upon a hierarchical generative model that infers the context in which movements are made. This means that we can confuse agents by changing the context (order) in which cues are presented. These simulations provide a (Bayes-optimal) model of contextual uncertainty and set switching that can be quantified in terms of behavioural and electrophysiological responses. Furthermore, one can simulate dopaminergic lesions (by changing the precision of prediction errors) to produce pathological behaviours that are reminiscent of those seen in neurological disorders such as Parkinson's disease. We use these simulations to demonstrate how a single functional role for dopamine at the synaptic level can manifest in different ways at the behavioural level

Stress, the Hippocampus and the Hypothalamic-Pituitary-Adrenal Axis: Implications for the development of psychotic disorders

The experience of stress is commonly implicated in models of the onset of psychotic disorders. However, prospective studies investigating associations between biological markers of stress and the emergence of psychotic disorders are limited and inconclusive. One biological system proposed as the link between the psychological experience of stress and the development of psychosis is the Hypothalamic-Pituitary-Adrenal (HPA) axis. This paper summarizes and discusses evidence supporting a role for HPA-axis dysfunction in the early phase of schizophrenia and related disorders. METHOD A selective review of psychiatric and psychological research on stress, coping, HPA-axis, the hippocampus and psychotic disorders was performed, with a particular focus on the relationship between HPA-axis dysfunction and the onset of psychotic disorders. RESULTS Individual strands of past research have suggested that the HPA-axis is dysfunctional in at least some individuals with established psychotic disorders; that the hippocampus is an area of the brain that appears to be implicated in the onset and maintenance of psychotic disorders; and that an increase in the experience of stress precedes the onset of a psychotic episode in some individuals. Models of the onset and maintenance of psychotic disorders that link these individual strands of research and strategies for examining these models are proposed in this paper. CONCLUSIONS The current literature provides some evidence that the onset of psychotic disorders may be associated with a higher rate of stress and changes to the hippocampus. It is suggested that future research should investigate whether a relationship exists between psychological stress, HPA-axis functioning and the hippocampus in the onset of these disorders. Longitudinal assessment of these factors in young people at 'ultra' high risk of psychosis and first-episode psychosis cohorts may enhance understanding of the possible interaction between them in the early phases of illness