Probabilistic modeling of planar pushing

Thesis: S.M., Massachusetts Institute of Technology, Department of Mechanical Engineering, 2018.Cataloged from PDF version of thesis.Includes bibliographical references (pages 65-68).This work studies the problem of data-driven modeling and stochastic filtering of complex dynamical systems. The main contributions are GP-SUM, a filtering algorithm tailored to systems expressed as Gaussian processes (GP), and the probabilistic modeling of planar pushing by combining input-dependent GPs and GP-SUM. The main advantages of GP-SUM for filtering are that it does not rely on linearizations or unimodal Gaussian approximations of the belief. Moreover, it can be seen as a combination of a sampling-based filter and a probabilistic Bayes filter as GP-SUM operates by sampling the state distribution and propagating each sample through the dynamic system and observation models. Effective sampling and accurate probabilistic propagation are possible by relying on the GP form of the system, and a Gaussian mixture form of the belief. In this thesis we show that GP-SUM outperforms several GP-Bayes and Particle Filters on a standard benchmark. To characterize the dynamics of pushing, we use input-dependent GPs to learn the motion of the pushed object after a short time step. With this approach we show that we can learn accurate data-driven models that outperform analytical models after less than 100 samples and saturate in performance with less than 1000 samples. We validate the results against a collected dataset of repeated trajectories, and use the learned models to study questions such as the nature of the variability in pushing, and the validity of the quasi-static assumption. Finally, we illustrate how our learned model for pushing can be combined with GP-SUM, and demonstrate that we can predict heteroscedasticity, i.e., different amounts of uncertainty, and multi-modality when naturally occurring in pushing.by Maria Bauza Villalonga.S.M

A probabilistic data-driven model for planar pushing

This paper presents a data-driven approach to model planar pushing interaction to predict both the most likely outcome of a push and its expected variability. The learned models rely on a variation of Gaussian processes with input-dependent noise called Variational Heteroscedastic Gaussian processes (VHGP) [1] that capture the mean and variance of a stochastic function. We show that we can learn accurate models that outperform analytical models after less than 100 samples and saturate in performance with less than 1000 samples. We validate the results against a collected dataset of repeated trajectories, and use the learned models to study questions such as the nature of the variability in pushing, and the validity of the quasi-static assumption

GP-SUM. Gaussian Process Filtering of non-Gaussian Beliefs

This work studies the problem of stochastic dynamic filtering and state propagation with complex beliefs. The main contribution is GP-SUM, a filtering algorithm tailored to dynamic systems and observation models expressed as Gaussian Processes (GP), and to states represented as a weighted Sum of Gaussians. The key attribute of GP-SUM is that it does not rely on linearizations of the dynamic or observation models, or on unimodal Gaussian approximations of the belief, hence enables tracking complex state distributions. The algorithm can be seen as a combination of a sampling-based filter with a probabilistic Bayes filter. On the one hand, GP-SUM operates by sampling the state distribution and propagating each sample through the dynamic system and observation models. On the other hand, it achieves effective sampling and accurate probabilistic propagation by relying on the GP form of the system, and the sum-of-Gaussian form of the belief. We show that GP-SUM outperforms several GP-Bayes and Particle Filters on a standard benchmark. We also demonstrate its use in a pushing task, predicting with experimental accuracy the naturally occurring non-Gaussian distributions

Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations

This paper presents a novel regrasp control policy that makes use of tactile sensing to plan local grasp adjustments.Our approach determines regrasp actions by virtually searching for local transformations of tactile measurements that improve the quality of the grasp.First, we construct a tactile-based grasp quality metricusing a deep convolutional neural network trained on over2800 grasps. The quality of each grasp, a continuous value between 0 and 1, is determined experimentally by measuring its resistance to external perturbations. Second, we simulate the tactile imprints associated with robot motions relative to the initial grasp by performing rigid-body transformations of the given tactile measurements. The newly generated tactile imprints are evaluated with the learned grasp quality network and the regrasp action is chosen to maximize the grasp quality.Results show that the grasp quality network can predict the outcome of grasps with an average accuracy of 85%on known objects and 75%on novel objects. The regrasp control policy improves the success rate of grasp actions by an average relative increase of 70%on a test set of 8 objects. We provide a video summarizing our approach at https://youtu.be/gjn7DmfpwDk

Graph element networks: Adaptive, structured computation and memory

© 36th International Conference on Machine Learning, ICML 2019. All rights reserved. We explore the use of graph neural networks (GNNs) to model spatial processes in which there is no a priori graphical structure. Similar to finite element analysis, we assign nodes of a GNN to spatial locations and use a computational process defined on the graph to model the relationship between an initial function defined over a space and a resulting function in the same space. We use GNNs as a computational substrate, and show that the locations of the nodes in space as well as their connectivity can be optimized to focus on the most complex parts of the space. Moreover, this representational strategy allows the learned input-output relationship to generalize over the size of the underlying space and run the same model at different levels of precision, trading computation for accuracy. We demonstrate this method on a traditional PDE problem, a physical prediction problem from robotics, and learning to predict scene images from novel viewpoints

Combining physical simulators and object-based networks for control

Physics engines play an important role in robot planning and control; however, many real-world control problems involve complex contact dynamics that cannot be characterized analytically. Most physics engines therefore employ approximations that lead to a loss in precision. In this paper, we propose a hybrid dynamics model, simulator-augmented interaction networks (SAIN), combining a physics engine with an object-based neural network for dynamics modeling. Compared with existing models that are purely analytical or purely data-driven, our hybrid model captures the dynamics of interacting objects in a more accurate and data-efficient manner. Experiments both in simulation and on a real robot suggest that it also leads to better performance when used in complex control tasks. Finally, we show that our model generalizes to novel environments with varying object shapes and materials.NSF (nos. 1420316, 1523767, and 1723381)AFOSR (Grant FA9550- 17-1-0165)ONR MURI (N00014-16-1-2007

Omnipush: accurate, diverse, real-world dataset of pushing dynamics with RGB-D video

Pushing is a fundamental robotic skill. Existing work has shown how to exploit models of pushing to achieve a variety of tasks, including grasping under uncertainty, in-hand manipulation and clearing clutter. Such models, however, are approximate, which limits their applicability. Learning-based methods can reason directly from raw sensory data with accuracy, and have the potential to generalize to a wider diversity of scenarios. However, developing and testing such methods requires rich-enough datasets. In this paper we introduce Omnipush, a dataset with high variety of planar pushing behavior.In particular, we provide 250 pushes for each of 250 objects, all recorded with RGB-D and a high precision tracking system. The objects are constructed so as to systematically explore key factors that affect pushing-The shape of the object and its mass distribution-which have not been broadly explored in previous datasets, and allow to study generalization in model learning. Omnipush includes a benchmark for meta-learning dynamic models, which requires algorithms that make good predictions and estimate their own uncertainty. We also provide an RGB video prediction benchmark and propose other relevant tasks that can be suited with this dataset. Data and code are available at https://web.mit.edu/mcube/omnipush-dataset/