Goals and information processing in human decisions

We do not make decisions in the void. Every day, we act in awareness of our context, adjusting our objectives according to the situations we find. Operating effectively under multiple goals is fundamental for appropriate learning and decision-making, and deficiencies in this capacity can be at the core of mental disorders such as anxiety, depression, or post-traumatic stress disorder. In this thesis, I present studies I conducted to investigate how goals impact different stages of the decision process, from simple perceptual choices to subjective value preferences. Previous studies have described how animals assess alternatives and integrate evidence to make decisions. Most of the time, the focus of this work has been on simplified scenarios with single goals. In this thesis, my experiments tackle the issue of how people adjust information processing in tasks that demand more than one objective. Through various manipulations of the behavioural goals, such as decision framing, I show that (i) attention and evidence accumulation, (ii) brain representations, and (iii) decision confidence were all affected by context changes. Using behavioural testing, computational models, and neuroimaging I show that goals have a crucial role in evidence integration and the allocation of visual attention. My findings indicate that brain patterns adapt to enhance goal-relevant information during learning and the valuation of alternatives. Finally, I report the presence of goal-dependent asymmetries in the generation of decision confidence, overweighting the evidence of the most-relevant option to fulfil the goal. In conclusion, I show how the entire process is highly flexible and serves the behavioural demands. These findings support the reinterpretation of some perspectives, such as reported biases and irrationalities in decisions, as attributes of adaptive processing towards goal fulfilment

Value-Based Decision Making and Learning as Algorithms Computed by the Nervous System

How do we do what we do? Casting light on this essential question, the blossoming perspective of computational cognitive neuroscience gives rise to the present exposition of the nervous system and its phenomena of value-based decision making and learning. As justified herein by not only theory but also simulation against empirical data, human decision making and learning are framed mathematically in the explicit terms of two fundamental classes of algorithms--namely, sequential sampling and reinforcement learning. These counterparts are complementary in their coverage of the dynamics of unified neural, mental, and behavioral processes at different temporal scales. Novel variants of models based on such algorithms are introduced here to account for findings from experiments including measurements of both behavior and the brain in human participants. In principle, formal dynamical models of decision making hold the potential to represent fundamental computations underpinning value-based (i.e., preferential) decisions in addition to perceptual decisions. Sequential-sampling models such as the race model and the drift-diffusion model that are grounded in simplicity, analytical tractability, and optimality remain popular, but some of their more recent counterparts have instead been designed with an aim for more feasibility as architectures to be implemented by actual neural systems. In Chapter 2, connectionist models are proposed at an intermediate level of analysis that bridges mental phenomena and underlying neurophysiological mechanisms. Several such models drawing elements from the established race, drift-diffusion, feedforward-inhibition, divisive-normalization, and competing-accumulator models were tested with respect to fitting empirical data from human participants making choices between foods on the basis of hedonic value rather than a traditional perceptual attribute. Even when considering performance at emulating behavior alone, more neurally plausible models were set apart from more normative race or drift-diffusion models both quantitatively and qualitatively despite remaining parsimonious. To best capture the paradigm, a novel six-parameter computational model was formulated with features including hierarchical levels of competition via mutual inhibition as well as a static approximation of attentional modulation, which promotes "winner-take-all" processing. Moreover, a meta-analysis encompassing several related experiments validated the robustness of model-predicted trends in humans' value-based choices and concomitant reaction times. These findings have yet further implications for analysis of neurophysiological data in accordance with computational modeling, which is also discussed in this new light. Decision making in any brain is imperfect and costly in terms of time and energy. Operating under such constraints, an organism could be in a position to improve performance if an opportunity arose to exploit informative patterns in the environment being searched. Such an improvement of performance could entail both faster and more accurate (i.e., reward-maximizing) decisions. Chapter 3 investigated the extent to which human participants could learn to take advantage of immediate patterns in the spatial arrangement of serially presented foods such that a region of space would consistently be associated with greater subjective value. Eye movements leading up to choices demonstrated rapidly induced biases in the selective allocation of visual fixation and attention that were accompanied by both faster and more accurate choices of desired goods as implicit learning occurred. However, for the control condition with its spatially balanced reward environment, these subjects exhibited preexisting lateralized biases for eye and hand movements (i.e., leftward and rightward, respectively) that could act in opposition not only to each other but also to the orienting biases elicited by the experimental manipulation, producing an asymmetry between the left and right hemifields with respect to performance. Potentially owing at least in part to learned cultural conventions (e.g., reading from left to right), the findings herein particularly revealed an intrinsic leftward bias underlying initial saccades in the midst of more immediate feedback-directed processes for which spatial biases can be learned flexibly to optimize oculomotor and manual control in value-based decision making. The present study thus replicates general findings of learned attentional biases in a novel context with inherently rewarding stimuli and goes on to further elucidate the interactions between endogenous and exogenous biases. Prediction-error signals consistent with formal models of "reinforcement learning" (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. For Chapter 4, we created a novel paradigm optimized to dissociate the subtypes of reward-prediction errors that function as the key computational signatures of two distinct classes of RL models--namely, "actor/critic" models and action-value-learning models (e.g., the Q-learning model). The state-value-prediction error (SVPE), which is independent of actions, is a hallmark of the actor/critic architecture, whereas the action-value-prediction error (AVPE) is the distinguishing feature of action-value-learning algorithms. To test for the presence of these prediction-error signals in the brain, we scanned human participants with a high-resolution functional magnetic-resonance imaging (fMRI) protocol optimized to enable measurement of neural activity in the dopaminergic midbrain as well as the striatal areas to which it projects. In keeping with the actor/critic model, the SVPE signal was detected in the substantia nigra. The SVPE was also clearly present in both the ventral striatum and the dorsal striatum. However, alongside these purely state-value-based computations we also found evidence for AVPE signals throughout the striatum. These high-resolution fMRI findings suggest that model-free aspects of reward learning in humans can be explained algorithmically with RL in terms of an actor/critic mechanism operating in parallel with a system for more direct action-value learning.</p

Neurocomputational models of corticostriatal interactions in action selection

Schema theory is a framework based on the idea that behaviour in many areas depends on abstractions over instances called schemas, which work in a cooperative or sequential fashion, but also compete with each other for activation. Cooper & Shallice (2000) provide an implementation of schema-theory with their model that simulates how routine actions works in healthy and neurologically-impaired populations. While schema theory is helpful in representing functional interactions in the action-perception cycle, it has no commitment to a specific neural implementation. Redgrave et al.’s (2001) model of the basal ganglia is, in principle, compatible with a device that regulates the competition among schemas, carrying out action selection. This thesis is mainly concerned with improving the neurobiological plausibility of the schema theoretic account of action selection without sacrificing its theoretical underpinning. We therefore start by combining an implementation of schema-theory with a reparametrised version of the original basal ganglia model, building the model from the ground up. The model simulates two widely used neuropsychological tasks, the Wisconsin Card Sorting Test (WCST), and the Brixton Task (BRX). In order to validate the model, we then present a study with 25 younger and 25 over-60 individuals performing the WCST and BRX, and we simulate their performance using the schema-theoretic basal ganglia model. Experimental results indicate a dissociation between loss of representation (present in older adults) and perseveration of response (absent in older adults) in the WCST, and the model fits adequately simulate these findings while grounding the interpretation of parameters to the neurobiology of aging. We subsequently present a further study with 50 participants, 14 of whom have an ADHD diagnosis, performing the WCST under an untimed and a timed condition, and we then use our model to fit response time. Results indicate that impulsivity traits, but not inattention ones, predict a slower tail of responses in the untimed task and an increased number of missed responses and variability across subtasks. Using the model, we show that these results can be produced by variation of a combination of two parameters representing basal ganglia activity and top-down excitation. We conclude with recommendations on how to improve and extend the model

Development of cue integration with reward-mediated learning

This thesis will first introduce in more detail the Bayesian theory and its use in integrating multiple information sources. I will briefly talk about models and their relation to the dynamics of an environment, and how to combine multiple alternative models. Following that I will discuss the experimental findings on multisensory integration in humans and animals. I start with psychophysical results on various forms of tasks and setups, that show that the brain uses and combines information from multiple cues. Specifically, the discussion will focus on the finding that humans integrate this information in a way that is close to the theoretical optimal performance. Special emphasis will be put on results about the developmental aspects of cue integration, highlighting experiments that could show that children do not perform similar to the Bayesian predictions. This section also includes a short summary of experiments on how subjects handle multiple alternative environmental dynamics. I will also talk about neurobiological findings of cells receiving input from multiple receptors both in dedicated brain areas but also primary sensory areas. I will proceed with an overview of existing theories and computational models of multisensory integration. This will be followed by a discussion on reinforcement learning (RL). First I will talk about the original theory including the two different main approaches model-free and model-based reinforcement learning. The important variables will be introduced as well as different algorithmic implementations. Secondly, a short review on the mapping of those theories onto brain and behaviour will be given. I mention the most in uential papers that showed correlations between the activity in certain brain regions with RL variables, most prominently between dopaminergic neurons and temporal difference errors. I will try to motivate, why I think that this theory can help to explain the development of near-optimal cue integration in humans. The next main chapter will introduce our model that learns to solve the task of audio-visual orienting. Many of the results in this section have been published in [Weisswange et al. 2009b,Weisswange et al. 2011]. The model agent starts without any knowledge of the environment and acts based on predictions of rewards, which will be adapted according to the reward signaling the quality of the performed action. I will show that after training this model performs similarly to the prediction of a Bayesian observer. The model can also deal with more complex environments in which it has to deal with multiple possible underlying generating models (perform causal inference). In these experiments I use di#erent formulations of Bayesian observers for comparison with our model, and find that it is most similar to the fully optimal observer doing model averaging. Additional experiments using various alterations to the environment show the ability of the model to react to changes in the input statistics without explicitly representing probability distributions. I will close the chapter with a discussion on the benefits and shortcomings of the model. The thesis continues whith a report on an application of the learning algorithm introduced before to two real world cue integration tasks on a robotic head. For these tasks our system outperforms a commonly used approximation to Bayesian inference, reliability weighted averaging. The approximation is handy because of its computational simplicity, because it relies on certain assumptions that are usually controlled for in a laboratory setting, but these are often not true for real world data. This chapter is based on the paper [Karaoguz et al. 2011]. Our second modeling approach tries to address the neuronal substrates of the learning process for cue integration. I again use a reward based training scheme, but this time implemented as a modulation of synaptic plasticity mechanisms in a recurrent network of binary threshold neurons. I start the chapter with an additional introduction section to discuss recurrent networks and especially the various forms of neuronal plasticity that I will use in the model. The performance on a task similar to that of chapter 3 will be presented together with an analysis of the in uence of different plasticity mechanisms on it. Again benefits and shortcomings and the general potential of the method will be discussed. I will close the thesis with a general conclusion and some ideas about possible future work

Neurocomputational models of corticostriatal interactions in action selection

Schema theory is a framework based on the idea that behaviour in many areas depends on abstractions over instances called schemas, which work in a cooperative or sequential fashion, but also compete with each other for activation. Cooper & Shallice (2000) provide an implementation of schema-theory with their model that simulates how routine actions works in healthy and neurologically-impaired populations. While schema theory is helpful in representing functional interactions in the action-perception cycle, it has no commitment to a specific neural implementation. Redgrave et al.’s (2001) model of the basal ganglia is, in principle, compatible with a device that regulates the competition among schemas, carrying out action selection. This thesis is mainly concerned with improving the neurobiological plausibility of the schema theoretic account of action selection without sacrificing its theoretical underpinning. We therefore start by combining an implementation of schema-theory with a reparametrised version of the original basal ganglia model, building the model from the ground up. The model simulates two widely used neuropsychological tasks, the Wisconsin Card Sorting Test (WCST), and the Brixton Task (BRX). In order to validate the model, we then present a study with 25 younger and 25 over-60 individuals performing the WCST and BRX, and we simulate their performance using the schema-theoretic basal ganglia model. Experimental results indicate a dissociation between loss of representation (present in older adults) and perseveration of response (absent in older adults) in the WCST, and the model fits adequately simulate these findings while grounding the interpretation of parameters to the neurobiology of aging. We subsequently present a further study with 50 participants, 14 of whom have an ADHD diagnosis, performing the WCST under an untimed and a timed condition, and we then use our model to fit response time. Results indicate that impulsivity traits, but not inattention ones, predict a slower tail of responses in the untimed task and an increased number of missed responses and variability across subtasks. Using the model, we show that these results can be produced by variation of a combination of two parameters representing basal ganglia activity and top-down excitation. We conclude with recommendations on how to improve and extend the model

Industrial Robotics

This book covers a wide range of topics relating to advanced industrial robotics, sensors and automation technologies. Although being highly technical and complex in nature, the papers presented in this book represent some of the latest cutting edge technologies and advancements in industrial robotics technology. This book covers topics such as networking, properties of manipulators, forward and inverse robot arm kinematics, motion path-planning, machine vision and many other practical topics too numerous to list here. The authors and editor of this book wish to inspire people, especially young ones, to get involved with robotic and mechatronic engineering technology and to develop new and exciting practical applications, perhaps using the ideas and concepts presented herein