Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation

Manipulation of deformable objects, such as ropes and cloth, is an important but challenging problem in robotics. We present a learning-based system where a robot takes as input a sequence of images of a human manipulating a rope from an initial to goal configuration, and outputs a sequence of actions that can reproduce the human demonstration, using only monocular images as input. To perform this task, the robot learns a pixel-level inverse dynamics model of rope manipulation directly from images in a self-supervised manner, using about 60K interactions with the rope collected autonomously by the robot. The human demonstration provides a high-level plan of what to do and the low-level inverse model is used to execute the plan. We show that by combining the high and low-level plans, the robot can successfully manipulate a rope into a variety of target shapes using only a sequence of human-provided images for direction.Comment: 8 pages, accepted to International Conference on Robotics and Automation (ICRA) 201

Cable Manipulation with a Tactile-Reactive Gripper

Cables are complex, high dimensional, and dynamic objects. Standard approaches to manipulate them often rely on conservative strategies that involve long series of very slow and incremental deformations, or various mechanical fixtures such as clamps, pins or rings. We are interested in manipulating freely moving cables, in real time, with a pair of robotic grippers, and with no added mechanical constraints. The main contribution of this paper is a perception and control framework that moves in that direction, and uses real-time tactile feedback to accomplish the task of following a dangling cable. The approach relies on a vision-based tactile sensor, GelSight, that estimates the pose of the cable in the grip, and the friction forces during cable sliding. We achieve the behavior by combining two tactile-based controllers: 1) Cable grip controller, where a PD controller combined with a leaky integrator regulates the gripping force to maintain the frictional sliding forces close to a suitable value; and 2) Cable pose controller, where an LQR controller based on a learned linear model of the cable sliding dynamics keeps the cable centered and aligned on the fingertips to prevent the cable from falling from the grip. This behavior is possible by a reactive gripper fitted with GelSight-based high-resolution tactile sensors. The robot can follow one meter of cable in random configurations within 2-3 hand regrasps, adapting to cables of different materials and thicknesses. We demonstrate a robot grasping a headphone cable, sliding the fingers to the jack connector, and inserting it. To the best of our knowledge, this is the first implementation of real-time cable following without the aid of mechanical fixtures.Comment: Accepted to RSS 202

Homotopic Path Planning on Manifolds for Cabled Mobile Robots

We present two path planning algorithms for mobile robots that are connected by cable to a fixed base. Our algorithms efficiently compute the shortest path and control strategy that lead the robot to the target location considering cable length and obstacle interactions. First, we focus on cable-obstacle collisions. We introduce and formally analyze algorithms that build and search an overlapped configuration space manifold. Next, we present an extension that considers cable-robot collisions. All algorithms are experimentally validated using a real robot

Modeling and grasping of thin deformable objects

Deformable modeling of thin shell-like and other objects have potential application in robot grasping, medical robotics, home robots, and so on. The ability to manipulate electrical and optical cables, rubber toys, plastic bottles, ropes, biological tissues, and organs is an important feature of robot intelligence. However, grasping of deformable objects has remained an underdeveloped research area. When a robot hand applies force to grasp a soft object, deformation will result in the enlarging of the finger contact regions and the rotation of the contact normals, which in turn will result in a changing wrench space. The varying geometry can be determined by either solving a high order differential equation or minimizing potential energy. Efficient and accurate modeling of deformations is crucial for grasp analysis. It helps us predict whether a grasp will be successful from its finger placement and exerted force, and subsequently helps us design a grasping strategy. The first part of this thesis extends the linear and nonlinear shell theories to describe extensional, shearing, and bending strains in terms of geometric invariants including the principal curvatures and vectors, and the related directional and covariant derivatives. To our knowledge, this is the first non-parametric formulation of thin shell strains. A computational procedure for the strain energy is then offered for general parametric shells. In practice, a shell deformation is conveniently represented by a subdivision surface. We compare the results via potential energy minimization over a couple of benchmark problems with their analytical solutions and the results generated by two commercial softwares ABAQUS and ANSYS. Our method achieves a convergence rate an order of magnitude higher. Experimental validation involves regular and freeform shell-like objects (of various materials) grasped by a robot hand, with the results compared against scanned 3-D data (accuracy 0.127mm). Grasped objects often undergo sizable shape changes, for which a much higher modeling accuracy can be achieved using the nonlinear elasticity theory than its linear counterpart. The second part numerically studies two-finger grasping of deformable curve-like objects under frictional contacts. The action is like squeezing. Deformation is modeled by a degenerate version of the thin shell theory. Several differences from rigid body grasping are shown. First, under a squeeze, the friction cone at each finger contact rotates in a direction that depends on the deformable object\u27s global geometry, which implies that modeling is necessary for grasp prediction. Second, the magnitude of the grasping force has to be above certain threshold to achieve equilibrium. Third, the set of feasible finger placements may increase significantly compared to that for a rigid object of the same shape. Finally, the ability to resist disturbance is bounded in the sense that increasing the magnitude of an external force may result in the breaking of the grasp

Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems

The goal of this research is to enable a robotic system to manipulate clothing and other highly non-rigid objects using an RGBD sensor. The focus of this thesis is to define and test various algorithms / models that are used to solve parts of the laundry process (i.e. handling, classifying, sorting, unfolding, and folding). First, a system is presented for automatically extracting and classifying items in a pile of laundry. Using only visual sensors, the robot identifies and extracts items sequentially from the pile. When an item is removed and isolated, a model is captured of the shape and appearance of the object, which is then compared against a dataset of known items. The contributions of this part of the laundry process are a novel method for extracting articles of clothing from a pile of laundry, a novel method of classifying clothing using interactive perception, and a multi-layer approach termed L-M-H, more specifically L-C-S-H for clothing classification. This thesis describes two different approaches to classify clothing into categories. The first approach relies upon silhouettes, edges, and other low-level image measurements of the articles of clothing. Experiments from the first approach demonstrate the ability of the system to efficiently classify and label into one of six categories (pants, shorts, short-sleeve shirt, long-sleeve shirt, socks, or underwear). These results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction. The second approach relies upon color, texture, shape, and edge information from 2D and 3D data within a local and global perspective. The multi-layer approach compartmentalizes the problem into a high (H) layer, multiple mid-level (characteristics(C), selection masks(S)) layers, and a low (L) layer. This approach produces \u27local\u27 solutions to solve the global classification problem. Experiments demonstrate the ability of the system to efficiently classify each article of clothing into one of seven categories (pants, shorts, shirts, socks, dresses, cloths, or jackets). The results presented in this paper show that, on average, the classification rates improve by +27.47% for three categories, +17.90% for four categories, and +10.35% for seven categories over the baseline system, using support vector machines. Second, an algorithm is presented for automatically unfolding a piece of clothing. A piece of cloth is pulled in different directions at various points of the cloth in order to flatten the cloth. The features of the cloth are extracted and calculated to determine a valid location and orientation in which to interact with it. The features include the peak region, corner locations, and continuity / discontinuity of the cloth. In this thesis, a two-stage algorithm is presented, introducing a novel solution to the unfolding / flattening problem using interactive perception. Simulations using 3D simulation software, and experiments with robot hardware demonstrate the ability of the algorithm to flatten pieces of laundry using different starting configurations. These results show that, at most, the algorithm flattens out a piece of cloth from 11.1% to 95.6% of the canonical configuration. Third, an energy minimization algorithm is presented that is designed to estimate the configuration of a deformable object. This approach utilizes an RGBD image to calculate feature correspondence (using SURF features), depth values, and boundary locations. Input from a Kinect sensor is used to segment the deformable surface from the background using an alpha-beta swap algorithm. Using this segmentation, the system creates an initial mesh model without prior information of the surface geometry, and it reinitializes the configuration of the mesh model after a loss of input data. This approach is able to handle in-plane rotation, out-of-plane rotation, and varying changes in translation and scale. Results display the proposed algorithm over a dataset consisting of seven shirts, two pairs of shorts, two posters, and a pair of pants. The current approach is compared using a simulated shirt model in order to calculate the mean square error of the distance from the vertices on the mesh model to the ground truth, provided by the simulation model

Flexible Object Manipulation

Flexible objects are a challenge to manipulate. Their motions are hard to predict, and the high number of degrees of freedom makes sensing, control, and planning difficult. Additionally, they have more complex friction and contact issues than rigid bodies, and they may stretch and compress. In this thesis, I explore two major types of flexible materials: cloth and string. For rigid bodies, one of the most basic problems in manipulation is the development of immobilizing grasps. The same problem exists for flexible objects. I have shown that a simple polygonal piece of cloth can be fully immobilized by grasping all convex vertices and no more than one third of the concave vertices. I also explored simple manipulation methods that make use of gravity to reduce the number of fingers necessary for grasping. I have built a system for folding a T-shirt using a 4 DOF arm and a fixed-length iron bar which simulates two fingers. The main goal with string manipulation has been to tie knots without the use of any sensing. I have developed single-piece fixtures capable of tying knots in fishing line, solder, and wire, along with a more complex track-based system for autonomously tying a knot in steel wire. I have also developed a series of different fixtures that use compressed air to tie knots in string. Additionally, I have designed four-piece fixtures, which demonstrate a way to fully enclose a knot during the insertion process, while guaranteeing that extraction will always succeed