Learning action-oriented models through active inference

Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms

Understanding the Cognitive Impact of Emerging Web Technologies: A Research Focus Area for Embodied, Extended and Distributed Approaches to Cognition

Alongside existing research into the social, political and economic impacts of the Web, there is also a need to explore the effects of the Web on our cognitive profile. This is particularly so as the range of interactive opportunities we have with the Web expands under the influence of a range of emerging technologies. Embodied, extended and distributed approaches to cognition are relevant to understanding the potential cognitive impact of these new technologies because of the emphasis they place on extra-neural and extra-corporeal factors in the shaping of our cognitive capabilities at both an individual and collective level. The current paper outlines a number of areas where embodied, extended and distributed approaches to cognition are useful in understanding the impact of emerging Web technologies on future forms of both human and machine intelligence

Lessons and new directions for extended cognition from social and personality psychology

This paper aims to expand the range of empirical work relevant to the extended cognition debates. First, I trace the historical development of the person-situation debate in social and personality psychology and the extended cognition debate in the philosophy of mind. Next, I highlight some instructive similarities between the two and consider possible objections to my comparison. I then argue that the resolution of the person-situation debate in terms of interactionism lends support for an analogously interactionist conception of extended cognition. I argue that this interactionism might necessitate a shift away from the dominant agent-artifact paradigm toward an agent–agent paradigm. If this is right, then social and personality psychology—the discipline(s) that developed from the person-situation debate—opens a whole new range of empirical considerations for extended cognition theorists which align with Clark & Chalmers original vision of agents themselves as spread into the world

The Extended Mind and Network-Enabled Cognition

In thinking about the transformative potential of network technologies with respect to human cognition, it is common to see network resources as playing a largely assistive or augmentative role. In this paper we propose a somewhat more radical vision. We suggest that the informational and technological elements of a network system can, at times, constitute part of the material supervenience base for a human agent’s mental states and processes. This thesis (called the thesis of network-enabled cognition) draws its inspiration from the notion of the extended mind that has been propounded in the philosophical and cognitive science literature. Our basic claim is that network systems can do more than just augment cognition; they can also constitute part of the physical machinery that makes mind and cognition mechanistically possible. In evaluating this hypothesis, we identify a number of issues that seem to undermine the extent to which contemporary network systems, most notably the World Wide Web, can legitimately feature as part of an environmentally-extended cognitive system. Specific problems include the reliability and resilience of network-enabled devices, the accessibility of online information content, and the extent to which network-derived information is treated in the same way as information retrieved from biological memory. We argue that these apparent shortfalls do not necessarily merit the wholesale rejection of the network-enabled cognition thesis; rather, they point to the limits of the current state-of-the-art and identify the targets of many ongoing research initiatives in the network and information sciences. In addition to highlighting the importance of current research and technology development efforts, the thesis of network-enabled cognition also suggests a number of areas for future research. These include the formation and maintenance of online trust relationships, the subjective assessment of information credibility and the long-term impact of network access on human psychological and cognitive functioning. The nascent discipline of web science is, we suggest, suitably placed to begin an exploration of these issues

Is futsal a donor sport for football?: Exploiting complementarity for early diversification in talent development

Introduction: In team sports like football, athlete development emerges through the continuous experience and practice of varied activities under variations in task and environmental constraints. Such variations in environmental and task constraints provide variable practice opportunities and experiences that promote an enrichment of the learning process through enhanced transfer, and the discovery of individual capabilities through diverse, functional play activities. Objectives: In this commentary, we discuss theoretical insights that suggest how the sport of futsal can provide a useful basis for supporting the transfer of skills to performance in association football. Conclusions: The complementary nature of the two sports can be exploited for skill acquisition in early diversification through emphasising selected performance–based affordances, behavioral correspondence between sports, and self-evident advances towards task goals. By taking up futsal at an early stage, future football players will have the opportunity to explore futsal tactical behaviors that will enrich their developing perceptual-motor landscape. Practial Implications: To ensure a complementary transfer of capabilities between the sports, coaching interventions should highlight informational constraints to improve the coupling of perception and action in players in futsal and association football and promote the utilization of relevant affordances available in practice task designs

Consciosusness in Cognitive Architectures. A Principled Analysis of RCS, Soar and ACT-R

This report analyses the aplicability of the principles of consciousness developed in the ASys project to three of the most relevant cognitive architectures. This is done in relation to their aplicability to build integrated control systems and studying their support for general mechanisms of real-time consciousness.\ud To analyse these architectures the ASys Framework is employed. This is a conceptual framework based on an extension for cognitive autonomous systems of the General Systems Theory (GST).\ud A general qualitative evaluation criteria for cognitive architectures is established based upon: a) requirements for a cognitive architecture, b) the theoretical framework based on the GST and c) core design principles for integrated cognitive conscious control systems

An Examination into the Putative Mechanisms Underlying Human Sensorimotor Learning and Decision Making

Sensorimotor learning can be defined as a process by which an organism benefits from its experience, such that its future behaviour is better adapted to its environment. Humans are sensorimotor learners par excellence, and neurologically intact adults possess an incredible repertoire of skilled behaviours. Nevertheless, despite the topic fascinating scientists for centuries, there remains a lack of understanding about how humans truly learn. There is a need to better understand sensorimotor learning mechanisms in order to develop treatments for individuals with movement problems, improve training regimes (e.g. surgery) and accelerate motor learning in tasks such as handwriting in children and stroke rehabilitation. This thesis set out to improve our understanding of sensorimotor learning processes and develop methodologies and tools that enable other scientists to tackle these research questions using the power of recent developments in computer science (particularly immersive technologies). Errors in sensorimotor learning are the specific focus of the experimental chapters of this thesis, where the goal is to address our understanding of error perception and correction in motor learning and provide a computational understanding of how we process different types of error to inform subsequent behaviour. A brief summary of the approaches employed, and tools developed over the course of this thesis are presented below. Chapter 1 of this thesis provides a concise overview of the literature on human sensorimotor learning. It introduces the concept of internal models of human interactions with the environment, constructed and refined by the brain in the learning process. Highlighted in this chapter are potential mechanisms for promoting learning (e.g. error augmentation, motor variability) and outstanding challenges for the field (e.g. redundancy, credit assignment). In Chapter 2 a computational model based on information acquisition is developed. The model suggests that disruptive forces applied to human movements during training could improve learning because they allow the learner to sample more information from their environment. Chapter 3 investigates whether sensorimotor learning can be accelerated through forcing participants to explore (and thus acquire more information) a novel workspace. The results imply that exploration may be a necessary component of learning but manipulating it in this way is not sufficient to accelerate learning. This work serves to highlight the critical role of error correction in learning. The process of conducting the experimental work in Chapters 2 and 3 highlighted the need for an application programme interface that would allow researchers to rapidly deploy experiments that allow one to examine learning in a controlled but ecologically relevant manner. Virtual reality systems (that measure human interactions with computer generated worlds) provide a powerful tool for exploring sensorimotor learning and their use in the study of human behaviour is now more feasible due to recent technological advances. To this end, Chapter 4 reports the development of the Unity Experiment Framework - a new tool to assist in the development of virtual reality experiments in the Unity game engine. Chapter 5 builds on the findings from Chapters 2 & 3 on learning by addressing the specific contributions of visual error. It utilises the Unity Experiment Framework to explore whether visually increasing the error signal in a novel aiming task can accelerate motor learning. A novel aiming task is developed which requires participants to learn the mapping between rotations of the handheld virtual reality controllers and the movement of a cursor in Cartesian space. The results show that the visual disturbance does not accelerate the learning of skilled movements, implying a crucial role for mechanical forces, or physical error correction, which is consistent with the findings reported in Chapter 2. Uncontrolled manifold analysis provides insight into how the variability in selected solutions related to learning and performance, as the task deliberately allowed a variety of solutions from a redundant parameter space. Chapter 6 extends the scope of this thesis by examining how error information from the sensorimotor system influences higher order action selection processes. Chapter 5 highlighted the loose definition of “error” in sensorimotor learning and here, the goal was to advance our understanding of error learning by discriminating between different sources of error to better understand their contributions to future behaviour. This issue is illustrated through the example of a tennis player who, on a given point, has the options of selecting a backhand or forehand shot available to her. If the shot is ineffective (and produces an error signal), to optimise future behaviour, the brain needs to rapidly determine whether the error was due to poor shot selection, or whether the correct shot was selected but just poorly executed. To examine these questions, a novel ‘action bandit’ task was developed where participants made reaching movements towards targets, with each target having distinct probabilities of execution and selection error. The results revealed a significant selection bias towards a target that produced a higher frequency of execution errors (rather than a target associated with more selection error) despite no difference in expected value. This behaviour may be explained by a gating mechanism, where learning from the lack of reward is discounted following sensorimotor errors. However, execution errors also increase uncertainty about the appropriateness of a selected choice and the need to reduce uncertainty could equally account for these results. Subsequent experiments test these competing hypotheses and show this putative gating mechanism can be dynamically regulated though coupling of selections and execution errors. Development of models of these processes highlighted the dynamics of the mechanisms that drive the behaviour. In Chapter 7, the motor component of the task was removed to examine whether this effect is not unique to execution errors, but a feature of any two-stage decision-making process with, multiple error types which are presumed to be dissociated. These observations highlight the complex role error plays in learning and suggest the credit assignment process is guided and modulated by internal models of the task at hand. Finally, Chapter 8 closes this thesis with a summary of the key findings and arising from this work in the context of the literature on motor learning and decision making. It is noted here that this thesis sought to cover two broad research topics of motor learning and decision making that have, until recently, been studied by separate groups of researchers, with very little overlap in literature. A key goal of this programme of research was to contribute towards bringing together these hitherto disparate fields by focussing on breadth to establish common ground. As the experimental work developed, it became clear that the processing of error required a multi-pronged approach. Within each experimental chapter, the focus on error was accordingly narrowed and definitions refined. This culminated in developing and testing how individuals discriminate between errors in the sensorimotor and cognitive domains, thus presenting a framework for understanding how motor learning and decision making interact

Uncertainty in perception and the Hierarchical Gaussian Filter

In its full sense, perception rests on an agent's model of how its sensory input comes about and the inferences it draws based on this model. These inferences are necessarily uncertain. Here, we illustrate how the Hierarchical Gaussian Filter (HGF) offers a principled and generic way to deal with the several forms that uncertainty in perception takes. The HGF is a recent derivation of one-step update equations from Bayesian principles that rests on a hierarchical generative model of the environment and its (in)stability. It is computationally highly efficient, allows for online estimates of hidden states, and has found numerous applications to experimental data from human subjects. In this paper, we generalize previous descriptions of the HGF and its account of perceptual uncertainty. First, we explicitly formulate the extension of the HGF's hierarchy to any number of levels; second, we discuss how various forms of uncertainty are accommodated by the minimization of variational free energy as encoded in the update equations; third, we combine the HGF with decision models and demonstrate the inversion of this combination; finally, we report a simulation study that compared four optimization methods for inverting the HGF/decision model combination at different noise levels. These four methods (Nelder-Mead simplex algorithm, Gaussian process-based global optimization, variational Bayes and Markov chain Monte Carlo sampling) all performed well even under considerable noise, with variational Bayes offering the best combination of efficiency and informativeness of inference. Our results demonstrate that the HGF provides a principled, flexible, and efficient-but at the same time intuitive-framework for the resolution of perceptual uncertainty in behaving agents

Free-energy and the brain

If one formulates Helmholtz's ideas about perception in terms of modern-day theories one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts. Using constructs from statistical physics it can be shown that the problems of inferring what cause our sensory input and learning causal regularities in the sensorium can be resolved using exactly the same principles. Furthermore, inference and learning can proceed in a biologically plausible fashion. The ensuing scheme rests on Empirical Bayes and hierarchical models of how sensory information is generated. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of the brain's organisation and responses.In this paper, we suggest that these perceptual processes are just one emergent property of systems that conform to a free-energy principle. The free-energy considered here represents a bound on the surprise inherent in any exchange with the environment, under expectations encoded by its state or configuration. A system can minimise free-energy by changing its configuration to change the way it samples the environment, or to change its expectations. These changes correspond to action and perception respectively and lead to an adaptive exchange with the environment that is characteristic of biological systems. This treatment implies that the system's state and structure encode an implicit and probabilistic model of the environment. We will look at models entailed by the brain and how minimisation of free-energy can explain its dynamics and structure