What is value – accumulated reward or evidence?

Why are you reading this abstract? In some sense, your answer will cast the exercise as valuable – but what is value? In what follows, we suggest that value is evidence or, more exactly, log Bayesian evidence. This implies that a sufficient explanation for valuable behaviour is the accumulation of evidence for internal models of our world. This contrasts with normative models of optimal control and reinforcement learning, which assume the existence of a value function that explains behaviour, where (somewhat tautologically) behaviour maximises value. In this paper, we consider an alternative formulation – active inference – that replaces policies in normative models with prior beliefs about (future) states agents should occupy. This enables optimal behaviour to be cast purely in terms of inference: where agents sample their sensorium to maximise the evidence for their generative model of hidden states in the world – and minimise their uncertainty about those states. Crucially, this formulation resolves the tautology inherent in normative models and allows one to consider how prior beliefs are themselves optimised in a hierarchical setting. We illustrate these points by showing that any optimal policy can be specified with prior beliefs in the context of Bayesian inference. We then show how these prior beliefs are themselves prescribed by an imperative to minimise uncertainty. This formulation explains the saccadic eye movements required to read this text and defines the value of the visual sensations you are soliciting

Empirical Bayes for DCM: a group inversion scheme

This technical note considers a simple but important methodological issue in estimating effective connectivity; namely, how do we integrate measurements from multiple subjects to infer functional brain architectures that are conserved over subjects. We offer a solution to this problem that rests on a generalisation of random effects analyses to Bayesian inference about nonlinear models of electrophysiological time-series data. Specifically, we present an empirical Bayesian scheme for group or hierarchical models, in the setting of dynamic causal modelling (DCM). Recent developments in approximate Bayesian inference for hierarchical models enable the efficient estimation of group effects in DCM studies of multiple trials, sessions or subjects. This approach estimates second (e.g., between-subject) level parameters based posterior estimates from the first (e.g., within-subject) level. Here, we use empirical priors from the second level to iteratively optimise posterior densities over parameters at the first level. The motivation for this iterative application is to finesse the local minima problem inherent in the (first level) inversion of nonlinear and ill-posed models. Effectively, the empirical priors shrink the first level parameter estimates towards the global maximum, to provide more robust and efficient estimates of within (and between-subject) effects. This paper describes the inversion scheme using a worked example based upon simulated electrophysiological responses. In the subsequent paper, we will assess its robustness and reproducibility using an empirical example

VIRTUAL REALITY IN WAKING AND DREAMING CONSCIOUSNESS

This article explores the notion that the brain is genetically endowed with an innate virtual reality generator that – through experience-dependent plasticity –becomes a generative or predictive model of the world. This model, which is most clearly revealed in rapid eye movement (REM) sleep dreaming, may provide the theatre for conscious experience. Functional neuroimaging evidence for brain activations that are time-locked to rapid eye movements endorses the view that waking consciousness emerges from REM sleep – and dreaming lays the foundations for waking perception. In this view, the brain is equipped with a virtual model of the world that generates predictions of its sensations. This model is continually updated and entrained by sensory prediction errors in wakefulness to ensure veridical perception, but not in dreaming. In contrast, dreaming plays an essential role in maintaining and enhancing the capacity to model the world by minimizing model complexity and thereby maximizing both statistical and thermodynamic efficiency. This perspective suggests that consciousness corresponds to the embodied process of inference, realized through the generation of virtual realities (in both sleep and wakefulness). In short, our premise or hypothesis is that the waking brain engages with the sensorium to predict the causes of sensations, while in sleep the brain's generative model is actively refined so that it generates more efficient predictions during waking. We review the evidence in support of this hypothesis – evidence that grounds consciousness in biophysical computations whose neuronal and neurochemical infrastructure has been disclosed by sleep research

Simultaneous Learning and Filtering without Delusions: A Bayes-Optimal Derivation of Combining Predictive Inference and AdaptiveFiltering

Predictive coding appears to be one of the fundamental working principles of brain processing. Amongst other aspects, brains often predict the sensory consequences of their own actions. Predictive coding resembles Kalman filtering, where incoming sensory information is filtered to produce prediction errors for subsequent adaptation and learning. However, to generate prediction errors given motor commands, a suitable temporal forward model is required to generate predictions. While in engineering applications, it is usually assumed that this forward model is known, the brain has to learn it. When filtering sensory input and learning from the residual signal in parallel, a fundamental problem arises: the system can enter a delusional loop when filtering the sensory information using an overly trusted forward model. In this case, learning stalls before accurate convergence because uncertainty about the forward model is not properly accommodated. We present a Bayes-optimal solution to this generic and pernicious problem for the case of linear forward models, which we call Predictive Inference and Adaptive Filtering (PIAF). PIAF filters incoming sensory information and learns the forward model simultaneously. We show that PIAF is formally related to Kalman filtering and to the Recursive Least Squares linear approximation method, but combines these procedures in a Bayes optimal fashion. Numerical evaluations confirm that the delusional loop is precluded and that the learning of the forward model is more than ten-times faster when compared to a naive combination of Kalman filtering and Recursive Least Squares

The anatomy of choice:active inference and agency

This paper considers agency in the setting of embodied or active inference. In brief, we associate a sense of agency with prior beliefs about action and ask what sorts of beliefs underlie optimal behavior. In particular, we consider prior beliefs that action minimizes the KullbackLeibler (KL) divergence between desired states and attainable states in the future. This allows one to formulate bounded rationality as approximate Bayesian inference that optimizes a free energy bound on model evidence. We show that constructs like expected utility, exploration bonuses, softmax choice rules and optimism bias emerge as natural consequences of this formulation. Previous accounts of active inference have focused on predictive coding and Bayesian filtering schemes for minimizing free energy. Here, we consider variational Bayes as an alternative scheme that provides formal constraints on the computational anatomy of inference and actionconstraints that are remarkably consistent with neuroanatomy. Furthermore, this scheme contextualizes optimal decision theory and economic (utilitarian) formulations as pure inference problems. For example, expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature (of softmax functions and quantal response equilibria) has a unique and Bayes-optimal solutionthat minimizes free energy. This sensitivity corresponds to the precision of beliefs about behavior, such that attainable goals are afforded a higher precision or confidence. In turn, this means that optimal behavior entails a representation of confidence about outcomes that are under an agent's control

Resting-state coupling between core regions within the central-executive and salience networks contributes to working memory performance

Previous studies investigated the distinct roles played by different cognitive regions and suggested that the patterns of connectivity of these regions are associated with working memory. However, the specific causal mechanism through which the neuronal circuits that involve these brain regions contribute to working memory is still unclear. Here, in a large sample of healthy young adults, we first identified the core working memory regions by linking working memory accuracy to resting-state functional connectivity with the bilateral dorsolateral prefrontal cortex (a principal region in the central-executive network). Then a spectral dynamic causal modeling analysis was performed to quantify the effective connectivity between these regions. Finally, the effective connectivity was correlated with working memory accuracy to characterize the relationship between these connections and working memory performance. We found that the functional connections between the bilateral dorsolateral prefrontal cortex and the dorsal anterior cingulate cortex and between the right dorsolateral prefrontal cortex and the left orbital fronto-insular cortex were correlated with working memory accuracy. Furthermore, the effective connectivity from the dorsal anterior cingulate cortex to the bilateral dorsolateral prefrontal cortex and from the right dorsolateral prefrontal cortex to the left orbital fronto-insular cortex could predict individual differences in working memory. Because the dorsal anterior cingulate cortex and orbital fronto-insular cortex are core regions of the salience network, we inferred that the inter- and causal-connectivity between core regions within the central-executive and salience networks is functionally relevant for working memory performance. In summary, the current study identified the dorsolateral prefrontal cortex-related resting-state effective connectivity underlying working memory and suggests that individual differences in cognitive ability could be characterized by resting-state effective connectivity