Enabling Motion Planning and Execution for Tasks Involving Deformation and Uncertainty

A number of outstanding problems in robotic motion and manipulation involve tasks where degrees of freedom (DoF), be they part of the robot, an object being manipulated, or the surrounding environment, cannot be accurately controlled by the actuators of the robot alone. Rather, they are also controlled by physical properties or interactions - contact, robot dynamics, actuator behavior - that are influenced by the actuators of the robot. In particular, we focus on two important areas of poorly controlled robotic manipulation: motion planning for deformable objects and in deformable environments; and manipulation with uncertainty. Many everyday tasks we wish robots to perform, such as cooking and cleaning, require the robot to manipulate deformable objects. The limitations of real robotic actuators and sensors result in uncertainty that we must address to reliably perform fine manipulation. Notably, both areas share a common principle: contact, which is usually prohibited in motion planners, is not only sometimes unavoidable, but often necessary to accurately complete the task at hand. We make four contributions that enable robot manipulation in these poorly controlled tasks: First, an efficient discretized representation of elastic deformable objects and cost function that assess a ``cost of deformation\u27 for a specific configuration of a deformable object that enables deformable object manipulation tasks to be performed without physical simulation. Second, a method using active learning and inverse-optimal control to build these discretized representations from expert demonstrations. Third, a motion planner and policy-based execution approach to manipulation with uncertainty which incorporates contact with the environment and compliance of the robot to generate motion policies which are then adapted during execution to reflect actual robot behavior. Fourth, work towards the development of an efficient path quality metric for paths executed with actuation uncertainty that can be used inside a motion planner or trajectory optimizer

Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation

Understanding and manipulating deformable objects (e.g., ropes and fabrics) is an essential yet challenging task with broad applications. Difficulties come from complex states and dynamics, diverse configurations and high-dimensional action space of deformable objects. Besides, the manipulation tasks usually require multiple steps to accomplish, and greedy policies may easily lead to local optimal states. Existing studies usually tackle this problem using reinforcement learning or imitating expert demonstrations, with limitations in modeling complex states or requiring hand-crafted expert policies. In this paper, we study deformable object manipulation using dense visual affordance, with generalization towards diverse states, and propose a novel kind of foresightful dense affordance, which avoids local optima by estimating states' values for long-term manipulation. We propose a framework for learning this representation, with novel designs such as multi-stage stable learning and efficient self-supervised data collection without experts. Experiments demonstrate the superiority of our proposed foresightful dense affordance. Project page: https://hyperplane-lab.github.io/DeformableAffordanc

Detection and Physical Interaction with Deformable Linear Objects

Deformable linear objects (e.g., cables, ropes, and threads) commonly appear in our everyday lives. However, perception of these objects and the study of physical interaction with them is still a growing area. There have already been successful methods to model and track deformable linear objects. However, the number of methods that can automatically extract the initial conditions in non-trivial situations for these methods has been limited, and they have been introduced to the community only recently. On the other hand, while physical interaction with these objects has been done with ground manipulators, there have not been any studies on physical interaction and manipulation of the deformable linear object with aerial robots. This workshop describes our recent work on detecting deformable linear objects, which uses the segmentation output of the existing methods to provide the initialization required by the tracking methods automatically. It works with crossings and can fill the gaps and occlusions in the segmentation and output the model desirable for physical interaction and simulation. Then we present our work on using the method for tasks such as routing and manipulation with the ground and aerial robots. We discuss our feasibility analysis on extending the physical interaction with these objects to aerial manipulation applications.Comment: Presented at ICRA 2022 2nd Workshop on Representing and Manipulating Deformable Objects (https://deformable-workshop.github.io/icra2022/

Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation

Manipulation of deformable objects, such as ropes and cloth, is an important but challenging problem in robotics. We present a learning-based system where a robot takes as input a sequence of images of a human manipulating a rope from an initial to goal configuration, and outputs a sequence of actions that can reproduce the human demonstration, using only monocular images as input. To perform this task, the robot learns a pixel-level inverse dynamics model of rope manipulation directly from images in a self-supervised manner, using about 60K interactions with the rope collected autonomously by the robot. The human demonstration provides a high-level plan of what to do and the low-level inverse model is used to execute the plan. We show that by combining the high and low-level plans, the robot can successfully manipulate a rope into a variety of target shapes using only a sequence of human-provided images for direction.Comment: 8 pages, accepted to International Conference on Robotics and Automation (ICRA) 201

Modélisations et stratégie de prise pour la manipulation d'objets déformables

Dexterous manipulation is an important issue in robotics research in which few works have tackled deformable object manipulation. New applications in surgery, food industry or in service robotics require mastering the grasping and manipulation of deformable objects. This thesis focuses on deformable object manipulation by anthropomorphic mechanical graspers such as multi-fingered articulated hands. This task requires a great expertise in mechanical modeling and control: interaction modeling, tactile and vision perception, force / position control of finger movements to ensure stable grasping. The work presented in this thesis focuses on modeling the grasping of deformable objects. To this end, we used a discretization by non-linear mass-spring systems to model deformable bodies in large displacements and deformations while having a low computational cost. To predict the interaction forces between robot hand and deformable object, we proposed an original approach based on a visco-elasto-plastic rheological model to evaluate tangential contact forces and describe the transition between the sticking and slipping modes. The contact forces are evaluated at nodes as function of the relative movements between the fingertips and the surface mesh facets of the manipulated object. Another contribution of this thesis is the use of this model in the planning of 3D deformable object manipulation tasks. This planning consists in determining the optimal configuration of the hand for grasping the objects as well as the paths to track and the efforts to be applied by the fingers to control the deformation of the object while ensuring the stability of the operation. The experimental validation of this work has been carried out on two robotic platforms: a Barrett hand embedded on a Adept S1700D ® manipulator and a Shadow hand embedded on a Kuka LWR4+® manipulator.La manipulation dextre est un sujet important dans la recherche en robotique et dans lequel peu de travaux ont abordé la manipulation d'objets déformables. De nouvelles applications en chirurgie, en industrie agroalimentaire ou encore dans les services à la personne nécessitent la maîtrise de la saisie et la manipulation d'objets déformables. Cette thèse s’intéresse à la manipulation d’objets déformables par des préhenseurs mécaniques anthropomorphiques tels que des mains articulées à plusieurs doigts. Cette tâche requière une grande expertise en modélisation mécanique et en commande : modélisation des interactions, perception tactile et par vision, contrôle des mouvements des doigts en position et en force pour assurer la stabilité de la saisie. Les travaux présentés dans cette thèse se focalisent sur la modélisation de la saisie d'objets déformables. Pour cela, nous avons utilisé une discrétisation par des systèmes masses-ressorts non-linéaires pour modéliser des corps déformables en grands déplacements et déformations tout en ayant un coût calculatoire faible. Afin de prédire les forces d’interactions entre main robotique et objet déformable, nous avons proposé une approche originale basée sur un modèle rhéologique visco-élasto-plastique pour évaluer les forces tangentielles de contact et décrire la transition entre les modes d’adhérence et de glissement. Les forces de contact sont évaluées aux points nodaux en fonction des mouvements relatifs entre les bouts des doigts et les facettes du maillage de la surface de l’objet manipulé. Une autre contribution de cette thèse consiste à utiliser de cette modélisation dans la planification des tâches de manipulation d’objets déformables 3D. Cette planification consiste à déterminer la configuration optimale de la main pour la saisie de l’objet ainsi que les trajectoires à suivre et les efforts à appliquer par les doigts pour contrôler la déformation de l’objet tout en assurant la stabilité de l’opération. La validation expérimentale de ces travaux a été réalisée sur deux plateformes robotiques : une main Barrett embarquée sur un bras manipulateur Adept S1700D et une main Shadow embarquée sur un bras manipulateur Kuka LWR4+