Active inference, evidence accumulation, and the urn task

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology

Reply to: Systematic Overestimation of Reflection Impulsivity in the Information Sampling Task

To the Editor: Impulsivity, a psychological construct comprising both motor and cognitive factors, is thought to underlie important interindividual differences in health and disease (1). In particular, reflection impulsivity, which refers to the tendency to gather and evaluate information before decision making (2), has been implicated in many psychiatric and neurological disorders (3, 4, 5). One of the standard tasks for measuring reflection impulsivity in healthy and clinical populations is the Information Sampling Task (IST), designed by Clark et al. (3) and included in the widely used Cambridge Neuropsychological Test Automated Battery (CANTAB) (6). In this CANTAB version of the IST, participants sample a variable amount of information about an uncertain outcome before making a decision. The amount of information sampled before the decision gives a measure of participants’ reflection impulsivity. In this correspondence, we show that the calculation of the IST’s main outcome measure, P(correct), is based on incorrect statistical inference, resulting in systematic overestimation of participants’ reflection impulsivity and potentially inflated type II error rates. This might affect the results of numerous recent psychopharmacological, neuropsychological, and psychiatric publications that have used the IST (4, 5, 7).This work was supported by a Strategic Research Initiative Grant (to CM) from the University of Melbourne, Australia, and the National Health and Medical Research Council of Australia (Grant No. APP1021973 to MY

All Thinking is 'Wishful' Thinking

Motivation to engage in any epistemic behavior can be decomposed into two basic types that emerge in various guises across different disciplines and areas of study. The first basic dimension refers to a desire to approach versus avoid nonspecific certainty, which has epistemic value. It describes a need for an unambiguous, precise answer to a question, regardless of that answer’s specific content. Second basic dimension refers to a desire to approach versus avoid specific certainty, which has instrumental value. It concerns a need for the specific content of one’s beliefs and prior preferences. Together, they explain diverse epistemic behaviors, such as seeking, avoiding, and biasing new information and revising and updating, versus protecting, one’s beliefs, when confronted with new evidence. The relative strength of these motivational components determines the form of (Bayes optimal) epistemic behavior that follows

Evidence for surprise minimization over value maximization in choice behavior

Classical economic models are predicated on the idea that the ultimate aim of choice is to maximize utility or reward. In contrast, an alternative perspective highlights the fact that adaptive behavior requires agents' to model their environment and minimize surprise about the states they frequent. We propose that choice behavior can be more accurately accounted for by surprise minimization compared to reward or utility maximization alone. Minimizing surprise makes a prediction at variance with expected utility models; namely, that in addition to attaining valuable states, agents attempt to maximize the entropy over outcomes and thus 'keep their options open'. We tested this prediction using a simple binary choice paradigm and show that human decision-making is better explained by surprise minimization compared to utility maximization. Furthermore, we replicated this entropy-seeking behavior in a control task with no explicit utilities. These findings highlight a limitation of purely economic motivations in explaining choice behavior and instead emphasize the importance of belief-based motivations

The ease and sureness of a decision: evidence accumulation of conflict and uncertainty.

The likelihood of an outcome (uncertainty or sureness) and the similarity between choices (conflict or ease of a decision) are often critical to decision-making. We often ask ourselves: how likely are we to win or lose? And how different is this option's likelihood from the other? Uncertainty is a characteristic of the stimulus and conflict between stimuli, but these dissociable processes are often confounded. Here, applying a novel hierarchical drift diffusion approach, we study their interaction using a sequential learning task in healthy volunteers and pathological groups characterized by compulsive behaviours, by posing it as an evidence accumulation problem. The variables, Conflict (difficult or easy; difference between reward probabilities of the stimuli) and Uncertainty (low, medium or high; inverse U-shaped probability-uncertainty function) were then used to extract threshold ('a', amount of evidence accumulated before making a decision) and drift rate ('v', information processing speed) parameters. Critically, when a decision was both difficult (high conflict) and uncertain, relative to other conditions, healthy volunteers unexpectedly accumulated less evidence with lower decision thresholds and accuracy rates at chance levels. In contrast, patients with obsessive-compulsive disorder had slower processing speeds during these difficult uncertain decisions; yet, despite this more cautious approach, performed suboptimally with poorer accuracy relative to healthy volunteers below that of chance level. Thus, faced with a difficult uncertain decision, healthy controls are capable of rapid possibly random decisions, displaying almost a willingness to 'walk away', whereas those with obsessive compulsive disorder become more deliberative and cautious but despite appearing to learn the differential contingencies, still perform poorly. These observations might underlie disordered behaviours characterized by pathological uncertainty or doubt despite compulsive checking with impaired performance. In contrast, alcohol-dependent subjects show a different pattern relative to healthy controls with difficulties in adjusting their behavioural patterns with slower drift rates or processing speed despite decisions being easy or low conflict. We emphasize the multidimensional nature of compulsive behaviours and the utility of computational models in detecting subtle underlying processes relative to behavioural measures. These observations have implications for targeted behavioural interventions for specific cognitive impairments across psychiatric disorders

Bayesian Brains and the Rényi Divergence

Under the Bayesian brain hypothesis, behavioral variations can be attributed to different priors over generative model parameters. This provides a formal explanation for why individuals exhibit inconsistent behavioral preferences when confronted with similar choices. For example, greedy preferences are a consequence of confident (or precise) beliefs over certain outcomes. Here, we offer an alternative account of behavioral variability using Rényi divergences and their associated variational bounds. Rényi bounds are analogous to the variational free energy (or evidence lower bound) and can be derived under the same assumptions. Importantly, these bounds provide a formal way to establish behavioral differences through an α parameter, given fixed priors. This rests on changes in α that alter the bound (on a continuous scale), inducing different posterior estimates and consequent variations in behavior. Thus, it looks as if individuals have different priors and have reached different conclusions. More specifically, α→0+ optimization constrains the variational posterior to be positive whenever the true posterior is positive. This leads to mass-covering variational estimates and increased variability in choice behavior. Furthermore, α→+∞ optimization constrains the variational posterior to be zero whenever the true posterior is zero. This leads to mass-seeking variational posteriors and greedy preferences. We exemplify this formulation through simulations of the multiarmed bandit task. We note that these α parameterizations may be especially relevant (i.e., shape preferences) when the true posterior is not in the same family of distributions as the assumed (simpler) approximate density, which may be the case in many real-world scenarios. The ensuing departure from vanilla variational inference provides a potentially useful explanation for differences in behavioral preferences of biological (or artificial) agents under the assumption that the brain performs variational Bayesian inference