100,827 research outputs found
Learning action-oriented models through active inference
Converging theories suggest that organisms learn and exploit probabilistic models of their environment. However, it remains unclear how such models can be learned in practice. The open-ended complexity of natural environments means that it is generally infeasible for organisms to model their environment comprehensively. Alternatively, action-oriented models attempt to encode a parsimonious representation of adaptive agent-environment interactions. One approach to learning action-oriented models is to learn online in the presence of goal-directed behaviours. This constrains an agent to behaviourally relevant trajectories, reducing the diversity of the data a model need account for. Unfortunately, this approach can cause models to prematurely converge to sub-optimal solutions, through a process we refer to as a bad-bootstrap. Here, we exploit the normative framework of active inference to show that efficient action-oriented models can be learned by balancing goal-oriented and epistemic (information-seeking) behaviours in a principled manner. We illustrate our approach using a simple agent-based model of bacterial chemotaxis. We first demonstrate that learning via goal-directed behaviour indeed constrains models to behaviorally relevant aspects of the environment, but that this approach is prone to sub-optimal convergence. We then demonstrate that epistemic behaviours facilitate the construction of accurate and comprehensive models, but that these models are not tailored to any specific behavioural niche and are therefore less efficient in their use of data. Finally, we show that active inference agents learn models that are parsimonious, tailored to action, and which avoid bad bootstraps and sub-optimal convergence. Critically, our results indicate that models learned through active inference can support adaptive behaviour in spite of, and indeed because of, their departure from veridical representations of the environment. Our approach provides a principled method for learning adaptive models from limited interactions with an environment, highlighting a route to sample efficient learning algorithms
Goal-Directed Planning for Habituated Agents by Active Inference Using a Variational Recurrent Neural Network
It is crucial to ask how agents can achieve goals by generating action plans
using only partial models of the world acquired through habituated
sensory-motor experiences. Although many existing robotics studies use a
forward model framework, there are generalization issues with high degrees of
freedom. The current study shows that the predictive coding (PC) and active
inference (AIF) frameworks, which employ a generative model, can develop better
generalization by learning a prior distribution in a low dimensional latent
state space representing probabilistic structures extracted from well
habituated sensory-motor trajectories. In our proposed model, learning is
carried out by inferring optimal latent variables as well as synaptic weights
for maximizing the evidence lower bound, while goal-directed planning is
accomplished by inferring latent variables for maximizing the estimated lower
bound. Our proposed model was evaluated with both simple and complex robotic
tasks in simulation, which demonstrated sufficient generalization in learning
with limited training data by setting an intermediate value for a
regularization coefficient. Furthermore, comparative simulation results show
that the proposed model outperforms a conventional forward model in
goal-directed planning, due to the learned prior confining the search of motor
plans within the range of habituated trajectories.Comment: 30 pages, 19 figure
Propagation Networks for Model-Based Control Under Partial Observation
There has been an increasing interest in learning dynamics simulators for
model-based control. Compared with off-the-shelf physics engines, a learnable
simulator can quickly adapt to unseen objects, scenes, and tasks. However,
existing models like interaction networks only work for fully observable
systems; they also only consider pairwise interactions within a single time
step, both restricting their use in practical systems. We introduce Propagation
Networks (PropNet), a differentiable, learnable dynamics model that handles
partially observable scenarios and enables instantaneous propagation of signals
beyond pairwise interactions. Experiments show that our propagation networks
not only outperform current learnable physics engines in forward simulation,
but also achieve superior performance on various control tasks. Compared with
existing model-free deep reinforcement learning algorithms, model-based control
with propagation networks is more accurate, efficient, and generalizable to
new, partially observable scenes and tasks.Comment: Accepted to ICRA 2019. Project Page: http://propnet.csail.mit.edu
Video: https://youtu.be/ZAxHXegkz4
Goal-Directed Behavior under Variational Predictive Coding: Dynamic Organization of Visual Attention and Working Memory
Mental simulation is a critical cognitive function for goal-directed behavior
because it is essential for assessing actions and their consequences. When a
self-generated or externally specified goal is given, a sequence of actions
that is most likely to attain that goal is selected among other candidates via
mental simulation. Therefore, better mental simulation leads to better
goal-directed action planning. However, developing a mental simulation model is
challenging because it requires knowledge of self and the environment. The
current paper studies how adequate goal-directed action plans of robots can be
mentally generated by dynamically organizing top-down visual attention and
visual working memory. For this purpose, we propose a neural network model
based on variational Bayes predictive coding, where goal-directed action
planning is formulated by Bayesian inference of latent intentional space. Our
experimental results showed that cognitively meaningful competencies, such as
autonomous top-down attention to the robot end effector (its hand) as well as
dynamic organization of occlusion-free visual working memory, emerged.
Furthermore, our analysis of comparative experiments indicated that
introduction of visual working memory and the inference mechanism using
variational Bayes predictive coding significantly improve the performance in
planning adequate goal-directed actions
On Offline Evaluation of Vision-based Driving Models
Autonomous driving models should ideally be evaluated by deploying them on a
fleet of physical vehicles in the real world. Unfortunately, this approach is
not practical for the vast majority of researchers. An attractive alternative
is to evaluate models offline, on a pre-collected validation dataset with
ground truth annotation. In this paper, we investigate the relation between
various online and offline metrics for evaluation of autonomous driving models.
We find that offline prediction error is not necessarily correlated with
driving quality, and two models with identical prediction error can differ
dramatically in their driving performance. We show that the correlation of
offline evaluation with driving quality can be significantly improved by
selecting an appropriate validation dataset and suitable offline metrics. The
supplementary video can be viewed at
https://www.youtube.com/watch?v=P8K8Z-iF0cYComment: Published at the ECCV 2018 conferenc
Remembering Forward: Neural Correlates of Memory and Prediction in Human Motor Adaptation
We used functional MR imaging (FMRI), a robotic manipulandum and systems identification techniques to examine neural correlates of predictive compensation for spring-like loads during goal-directed wrist movements in neurologically-intact humans. Although load changed unpredictably from one trial to the next, subjects nevertheless used sensorimotor memories from recent movements to predict and compensate upcoming loads. Prediction enabled subjects to adapt performance so that the task was accomplished with minimum effort. Population analyses of functional images revealed a distributed, bilateral network of cortical and subcortical activity supporting predictive load compensation during visual target capture. Cortical regions – including prefrontal, parietal and hippocampal cortices – exhibited trial-by-trial fluctuations in BOLD signal consistent with the storage and recall of sensorimotor memories or “states” important for spatial working memory. Bilateral activations in associative regions of the striatum demonstrated temporal correlation with the magnitude of kinematic performance error (a signal that could drive reward-optimizing reinforcement learning and the prospective scaling of previously learned motor programs). BOLD signal correlations with load prediction were observed in the cerebellar cortex and red nuclei (consistent with the idea that these structures generate adaptive fusimotor signals facilitating cancelation of expected proprioceptive feedback, as required for conditional feedback adjustments to ongoing motor commands and feedback error learning). Analysis of single subject images revealed that predictive activity was at least as likely to be observed in more than one of these neural systems as in just one. We conclude therefore that motor adaptation is mediated by predictive compensations supported by multiple, distributed, cortical and subcortical structures
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids
Real-life control tasks involve matters of various substances---rigid or soft
bodies, liquid, gas---each with distinct physical behaviors. This poses
challenges to traditional rigid-body physics engines. Particle-based simulators
have been developed to model the dynamics of these complex scenes; however,
relying on approximation techniques, their simulation often deviates from
real-world physics, especially in the long term. In this paper, we propose to
learn a particle-based simulator for complex control tasks. Combining learning
with particle-based systems brings in two major benefits: first, the learned
simulator, just like other particle-based systems, acts widely on objects of
different materials; second, the particle-based representation poses strong
inductive bias for learning: particles of the same type have the same dynamics
within. This enables the model to quickly adapt to new environments of unknown
dynamics within a few observations. We demonstrate robots achieving complex
manipulation tasks using the learned simulator, such as manipulating fluids and
deformable foam, with experiments both in simulation and in the real world. Our
study helps lay the foundation for robot learning of dynamic scenes with
particle-based representations.Comment: Accepted to ICLR 2019. Project Page: http://dpi.csail.mit.edu Video:
https://www.youtube.com/watch?v=FrPpP7aW3L
- …