The Mixed Instrumental Controller: Using Value of Information to Combine Habitual Choice and Mental Simulation

Instrumental behavior depends on both goal-directed and habitual mechanisms of choice. Normative views cast these mechanisms in terms of model-free and model-based methods of reinforcement learning, respectively. An influential proposal hypothesizes that model-free and model-based mechanisms coexist and compete in the brain according to their relative uncertainty. In this paper we propose a novel view in which a single Mixed Instrumental Controller produces both goal-directed and habitual behavior by flexibly balancing and combining model-based and model-free computations. The Mixed Instrumental Controller performs a cost-benefits analysis to decide whether to chose an action immediately based on the available “cached” value of actions (linked to model-free mechanisms) or to improve value estimation by mentally simulating the expected outcome values (linked to model-based mechanisms). Since mental simulation entails cognitive effort and increases the reward delay, it is activated only when the associated “Value of Information” exceeds its costs. The model proposes a method to compute the Value of Information, based on the uncertainty of action values and on the distance of alternative cached action values. Overall, the model by default chooses on the basis of lighter model-free estimates, and integrates them with costly model-based predictions only when useful. Mental simulation uses a sampling method to produce reward expectancies, which are used to update the cached value of one or more actions; in turn, this updated value is used for the choice. The key predictions of the model are tested in different settings of a double T-maze scenario. Results are discussed in relation with neurobiological evidence on the hippocampus – ventral striatum circuit in rodents, which has been linked to goal-directed spatial navigation

On the development of intention understanding for joint action tasks

Our everyday, common sense ability to discern the intentions of others’ from their motions is fundamental for a successful cooperation in joint action tasks. In this paper we address in a modeling study the question of how the ability to understand complex goal-directed action sequences may develop during learning and practice. The model architecture reflects recent neurophysiological findings that suggest the existence of chains of mirror neurons associated with specific goals. These chains may be activated by external events to simulate the consequences of observed actions. Using the mathematical framework of dynamical neural fields to model the dynamics of different neural populations representing goals, action means and contextual cues, we show that such chains may develop based on a local, Hebbian learning rule. We validate the functionality of the learned model in a joint action task in which an observer robot infers the intention of a partner to chose a complementary action sequence.Fundação para a Ciência e a Tecnologia (FCT)European Commission (EC

Lichen Striatus Post-COVID-19 Infection: Clinical and Dermoscopic Presentations

N/

Spatial navigation, episodic memory, episodic future thinking, and theory of mind in children with autism spectrum disorder: evidence for impairments in mental simulation?

This study explored spatial navigation alongside several other cognitive abilities that are thought to share common underlying neurocognitive mechanisms (e.g., the capacity for self-projection, scene construction, or mental simulation), and which we hypothesized may be impaired in autism spectrum disorder (ASD). Twenty intellectually high-functioning children with ASD (with a mean age of ~8 years) were compared to 20 sex, age, IQ, and language ability matched typically developing children on a series of tasks to assess spatial navigation, episodic memory, episodic future thinking (also known as episodic foresight or prospection), theory of mind (ToM), relational memory, and central coherence. This is the first study to explore these abilities concurrently within the same sample. Spatial navigation was assessed using the “memory island” task, which involves finding objects within a realistic, computer simulated, three-dimensional environment. Episodic memory and episodic future thinking were assessed using a past and future event description task. ToM was assessed using the “animations” task, in which children were asked to describe the interactions between two animated triangles. Relational memory was assessed using a recognition task involving memory for items (line drawings), patterned backgrounds, or combinations of items and backgrounds. Central coherence was assessed by exploring differences in performance across segmented and unsegmented versions of block design. Children with ASD were found to show impairments in spatial navigation, episodic memory, episodic future thinking, and central coherence, but not ToM or relational memory. Among children with ASD, spatial navigation was found to be significantly negatively related to the number of repetitive behaviors. In other words, children who showed more repetitive behaviors showed poorer spatial navigation. The theoretical and practical implications of the results are discussed

Neuronal Chains for Actions in the Parietal Lobe: A Computational Model

The inferior part of the parietal lobe (IPL) is known to play a very important role in sensorimotor integration. Neurons in this region code goal-related motor acts performed with the mouth, with the hand and with the arm. It has been demonstrated that most IPL motor neurons coding a specific motor act (e.g., grasping) show markedly different activation patterns according to the final goal of the action sequence in which the act is embedded (grasping for eating or grasping for placing). Some of these neurons (parietal mirror neurons) show a similar selectivity also during the observation of the same action sequences when executed by others. Thus, it appears that the neuronal response occurring during the execution and the observation of a specific grasping act codes not only the executed motor act, but also the agent's final goal (intention)

How to characterize the function of a brain region

Many brain regions have been defined, but a comprehensive formalization of each region’s function in relation to human behavior is still lacking. Current knowledge comes from various fields, which have diverse conceptions of ‘functions’. We briefly review these fields and outline how the heterogeneity of associations could be harnessed to disclose the computational function of any region. Aggregating activation data from neuroimaging studies allows us to characterize the functional engagement of a region across a range of experimental conditions. Furthermore, large-sample data can disclose covariation between brain region features and ecological behavioral phenotyping. Combining these two approaches opens a new perspective to determine the behavioral associations of a brain region, and hence its function and broader role within large-scale functional networks

Finding minimal action sequences with a simple evaluation of actions

Animals are able to discover the minimal number of actions that achieves an outcome (the minimal action sequence). In most accounts of this, actions are associated with a measure of behavior that is higher for actions that lead to the outcome with a shorter action sequence, and learning mechanisms find the actions associated with the highest measure. In this sense, previous accounts focus on more than the simple binary signal of “was the outcome achieved?”; they focus on “how well was the outcome achieved?” However, such mechanisms may not govern all types of behavioral development. In particular, in the process of action discovery (Redgrave and Gurney, 2006), actions are reinforced if they simply lead to a salient outcome because biological reinforcement signals occur too quickly to evaluate the consequences of an action beyond an indication of the outcome’s occurrence. Thus, action discovery mechanisms focus on the simple evaluation of “was the outcome achieved?” and not “how well was the outcome achieved?” Notwithstanding this impoverishment of information, can the process of action discovery find the minimal action sequence? We address this question by implementing computational mechanisms, referred to in this paper as no-cost learning rules, in which each action that leads to the outcome is associated with the same measure of behavior. No-cost rules focus on “was the outcome achieved?” and are consistent with action discovery. No-cost rules discover the minimal action sequence in simulated tasks and execute it for a substantial amount of time. Extensive training, however, results in extraneous actions, suggesting that a separate process (which has been proposed in action discovery) must attenuate learning if no-cost rules participate in behavioral development. We describe how no-cost rules develop behavior, what happens when attenuation is disrupted, and relate the new mechanisms to wider computational and biological context

Sentence Processing: Linking Language to Motor Chains

A growing body of evidence in cognitive science and neuroscience points towards the existence of a deep interconnection between cognition, perception and action. According to this embodied perspective language is grounded in the sensorimotor system and language understanding is based on a mental simulation process (Jeannerod, 2007; Gallese, 2008; Barsalou, 2009). This means that during action words and sentence comprehension the same perception, action, and emotion mechanisms implied during interaction with objects are recruited. Among the neural underpinnings of this simulation process an important role is played by a sensorimotor matching system known as the mirror neuron system (Rizzolatti and Craighero, 2004). Despite a growing number of studies, the precise dynamics underlying the relation between language and action are not yet well understood. In fact, experimental studies are not always coherent as some report that language processing interferes with action execution while others find facilitation. In this work we present a detailed neural network model capable of reproducing experimentally observed influences of the processing of action-related sentences on the execution of motor sequences. The proposed model is based on three main points. The first is that the processing of action-related sentences causes the resonance of motor and mirror neurons encoding the corresponding actions. The second is that there exists a varying degree of crosstalk between neuronal populations depending on whether they encode the same motor act, the same effector or the same action-goal. The third is the fact that neuronal populations’ internal dynamics, which results from the combination of multiple processes taking place at different time scales, can facilitate or interfere with successive activations of the same or of partially overlapping pools