Robot Learning for Manipulation of Deformable Linear Objects

Deformable Object Manipulation (DOM) is a challenging problem in robotics. Until recently there has been limited research on the subject, with most robotic manipulation methods being developed for rigid objects. Part of the challenge in DOM is that non-rigid objects require solutions capable of generalizing to changes in shape and mechanical properties. Recently, Machine Learning (ML) has been proven successful in other fields where generalization is important such as computer vision, thus encouraging the application of ML to robotics as well. Notably, Reinforcement Learning (RL) has shown promise in finding control policies for manipulation of rigid objects. However, RL requires large amounts of data that are better satisfied in simulation while deformable objects are inherently more difficult to model and simulate. This thesis presents ReForm, a simulation sandbox for robotic manipulation of Deformable Linear Objects (DLOs) such as cables, ropes, and wires. DLO manipulation is an interesting problem for a variety of applications throughout manufacturing, agriculture, and medicine. Currently, this sandbox includes six shape control tasks, which are classified as explicit when a precise shape is to be achieved, or implicit when the deformation is just a consequence of a more abstract goal, e.g. wrapping a DLO around another object. The proposed simulation environments aim to facilitate comparison and reproducibility of robot learning research. To that end, an RL algorithm is tested on each simulated task providing initial benchmarking results. ReForm is one of three concurrent frameworks to first support DOM problems. This thesis also addresses the problem of DLO state representation for an explicit shape control problem. Moreover, the effects of elastoplastic properties on the RL reward definition are investigated. From a control perspective, DLOs with these properties are particularly challenging to manipulate due to their nonlinear behavior, acting elastic up to a yield point after which they become permanently deformed. A low-dimensional representation from discrete differential geometry is proposed, offering more descriptive shape information than a simple point-cloud while avoiding the need for curve fitting. Empirical results show that this representation leads to a better goal description in the presence of elastoplasticity, preventing the RL algorithm from converging to local minima which correspond to incorrect shapes of the DLO

Digital Fabrication Approaches for the Design and Development of Shape-Changing Displays

Interactive shape-changing displays enable dynamic representations of data and information through physically reconfigurable geometry. The actuated physical deformations of these displays can be utilised in a wide range of new application areas, such as dynamic landscape and topographical modelling, architectural design, physical telepresence and object manipulation. Traditionally, shape-changing displays have a high development cost in mechanical complexity, technical skills and time/finances required for fabrication. There is still a limited number of robust shape-changing displays that go beyond one-off prototypes. Specifically, there is limited focus on low-cost/accessible design and development approaches involving digital fabrication (e.g. 3D printing). To address this challenge, this thesis presents accessible digital fabrication approaches that support the development of shape-changing displays with a range of application examples – such as physical terrain modelling and interior design artefacts. Both laser cutting and 3D printing methods have been explored to ensure generalisability and accessibility for a range of potential users. The first design-led content generation explorations show that novice users, from the general public, can successfully design and present their own application ideas using the physical animation features of the display. By engaging with domain experts in designing shape-changing content to represent data specific to their work domains the thesis was able to demonstrate the utility of shape-changing displays beyond novel systems and describe practical use-case scenarios and applications through rapid prototyping methods. This thesis then demonstrates new ways of designing and building shape-changing displays that goes beyond current implementation examples available (e.g. pin arrays and continuous surface shape-changing displays). To achieve this, the thesis demonstrates how laser cutting and 3D printing can be utilised to rapidly fabricate deformable surfaces for shape-changing displays with embedded electronics. This thesis is concluded with a discussion of research implications and future direction for this work

New Geometric Data Structures for Collision Detection

We present new geometric data structures for collision detection and more, including: Inner Sphere Trees - the first data structure to compute the peneration volume efficiently. Protosphere - an new algorithm to compute space filling sphere packings for arbitrary objects. Kinetic AABBs - a bounding volume hierarchy that is optimal in the number of updates when the objects deform. Kinetic Separation-List - an algorithm that is able to perform continuous collision detection for complex deformable objects in real-time. Moreover, we present applications of these new approaches to hand animation, real-time collision avoidance in dynamic environments for robots and haptic rendering, including a user study that exploits the influence of the degrees of freedom in complex haptic interactions. Last but not least, we present a new benchmarking suite for both, peformance and quality benchmarks, and a theoretic analysis of the running-time of bounding volume-based collision detection algorithms

Robust interactive simulation of deformable solids with detailed geometry using corotational FEM

This thesis focuses on the interactive simulation of highly detailed deformable solids modelled with the Corotational Finite Element Method. Starting from continuum mechanics we derive the discrete equations of motion and present a simulation scheme with support for user-in-the-loop interaction, geometric constraints and contact treatment. The interplay between accuracy and computational cost is discussed in depth, and practical approximations are analyzed with an emphasis on robustness and efficiency, as required by interactive simulation. The first part of the thesis focuses on deformable material discretization using the Finite Element Method with simplex elements and a corotational linear constitutive model, and presents our contributions to the solution of widely reported robustness problems in case of large stretch deformations and finite element degeneration. First,we introduce a stress differential approximation for quasi-implicit corotational linear FEM that improves its results for large deformations and closely matches the fullyimplicit solution with minor computational overhead. Next, we address the problem ofrobustness and realism in simulations involving element degeneration, and show that existing methods have previously unreported flaws that seriously threaten robustness and physical plausibility in interactive applications. We propose a new continuous-time approach, degeneration-aware polar decomposition, that avoids such flaws and yields robust degeneration recovery. In the second part we focus on geometry representation and contact determination for deformable solids with highly detailed surfaces. Given a high resolution closed surface mesh we automatically build a coarse embedding tetrahedralization and a partitioned representation of the collision geometry in a preprocess. During simulation, our proposed contact determination algorithm finds all intersecting pairs of deformed triangles using a memory-efficient barycentric bounding volume hierarchy, connects them into potentially disjoint intersection curves and performs a topological flood process on the exact intersection surfaces to discover a minimal set of contact points. A novel contact normal definition is used to find contact point correspondences suitable for contact treatment.Aquesta tesi tracta sobre la simulació interactiva de sòlids deformables amb superfícies detallades, modelats amb el Mètode dels Elements Finits (FEM) Corotacionals. A partir de la mecànica del continuu derivem les equacions del moviment discretes i presentem un esquema de simulació amb suport per a interacció d'usuari, restriccions geomètriques i tractament de contactes. Aprofundim en la interrelació entre precisió i cost de computació, i analitzem aproximacions pràctiques fent èmfasi en la robustesa i l'eficiència necessàries per a la simulació interactiva. La primera part de la tesi es centra en la discretització del material deformable mitjançant el Mètode dels Elements Finits amb elements de tipus s'implex i un model constituent basat en elasticitat linial corotacional, i presenta les nostres contribucions a la solució de problemes de robustesa àmpliament coneguts que apareixen en cas de sobreelongament i degeneració dels elements finits. Primer introduïm una aproximació dels diferencials d'estress per a FEM linial corotacional amb integració quasi-implícita que en millora els resultats per a deformacions grans i s'apropa a la solució implícita amb un baix cost computacional. A continuació tractem el problema de la robustesa i el realisme en simulacions que inclouen degeneració d'elements finits, i mostrem que els mètodes existents presenten inconvenients que posen en perill la robustesa plausibilitat de la simulació en aplicacions interactives. Proposem un enfocament nou basat en temps continuu, la descomposició polar amb coneixement de degeneració, que evita els inconvenients esmentats i permet corregir la degeneració de forma robusta. A la segona part de la tesi ens centrem en la representació de geometria i la determinació de contactes per a sòlids deformables amb superfícies detallades. A partir d'una malla de superfície tancada construím una tetraedralització englobant de forma automàtica en un preprocés, i particionem la geometria de colisió. Proposem un algorisme de detecció de contactes que troba tots els parells de triangles deformats que intersecten mitjançant una jerarquia de volums englobants en coordenades baricèntriques, els connecta en corbes d'intersecció potencialment disjuntes i realitza un procés d'inundació topològica sobre les superfícies d'intersecció exactes per tal de descobrir un conjunt mínim de punts de contacte. Usem una definició nova de la normal de contacte per tal de calcular correspondències entre punts de contacte útils per al seu tractament.Postprint (published version

Active recognition and pose estimation of rigid and deformable objects in 3D space

Object recognition and pose estimation is a fundamental problem in computer vision and of utmost importance in robotic applications. Object recognition refers to the problem of recognizing certain object instances, or categorizing objects into specific classes. Pose estimation deals with estimating the exact position of the object in 3D space, usually expressed in Euler angles. There are generally two types of objects that require special care when designing solutions to the aforementioned problems: rigid and deformable. Dealing with deformable objects has been a much harder problem, and usually solutions that apply to rigid objects, fail when used for deformable objects due to the inherent assumptions made during the design. In this thesis we deal with object categorization, instance recognition and pose estimation of both rigid and deformable objects. In particular, we are interested in a special type of deformable objects, clothes. We tackle the problem of autonomously recognizing and unfolding articles of clothing using a dual manipulator. This problem consists of grasping an article from a random point, recognizing it and then bringing it into an unfolded state by a dual arm robot. We propose a data-driven method for clothes recognition from depth images using Random Decision Forests. We also propose a method for unfolding an article of clothing after estimating and grasping two key-points, using Hough Forests. Both methods are implemented into a POMDP framework allowing the robot to interact optimally with the garments, taking into account uncertainty in the recognition and point estimation process. This active recognition and unfolding makes our system very robust to noisy observations. Our methods were tested on regular-sized clothes using a dual-arm manipulator. Our systems perform better in both accuracy and speed compared to state-of-the-art approaches. In order to take advantage of the robotic manipulator and increase the accuracy of our system, we developed a novel approach to address generic active vision problems, called Active Random Forests. While state of the art focuses on best viewing parameters selection based on single view classifiers, we propose a multi-view classifier where the decision mechanism of optimally changing viewing parameters is inherent to the classification process. This has many advantages: a) the classifier exploits the entire set of captured images and does not simply aggregate probabilistically per view hypotheses; b) actions are based on learnt disambiguating features from all views and are optimally selected using the powerful voting scheme of Random Forests and c) the classifier can take into account the costs of actions. The proposed framework was applied to the same task of autonomously unfolding clothes by a robot, addressing the problem of best viewpoint selection in classification, grasp point and pose estimation of garments. We show great performance improvement compared to state of the art methods and our previous POMDP formulation. Moving from deformable to rigid objects while keeping our interest to domestic robotic applications, we focus on object instance recognition and 3D pose estimation of household objects. We are particularly interested in realistic scenes that are very crowded and objects can be perceived under severe occlusions. Single shot-based 6D pose estimators with manually designed features are still unable to tackle such difficult scenarios for a variety of objects, motivating the research towards unsupervised feature learning and next-best-view estimation. We present a complete framework for both single shot-based 6D object pose estimation and next-best-view prediction based on Hough Forests, the state of the art object pose estimator that performs classification and regression jointly. Rather than using manually designed features we propose an unsupervised feature learnt from depth-invariant patches using a Sparse Autoencoder. Furthermore, taking advantage of the clustering performed in the leaf nodes of Hough Forests, we learn to estimate the reduction of uncertainty in other views, formulating the problem of selecting the next-best-view. To further improve 6D object pose estimation, we propose an improved joint registration and hypotheses verification module as a final refinement step to reject false detections. We provide two additional challenging datasets inspired from realistic scenarios to extensively evaluate the state of the art and our framework. One is related to domestic environments and the other depicts a bin-picking scenario mostly found in industrial settings. We show that our framework significantly outperforms state of the art both on public and on our datasets. Unsupervised feature learning, although efficient, might produce sub-optimal features for our particular tast. Therefore in our last work, we leverage the power of Convolutional Neural Networks to tackled the problem of estimating the pose of rigid objects by an end-to-end deep regression network. To improve the moderate performance of the standard regression objective function, we introduce the Siamese Regression Network. For a given image pair, we enforce a similarity measure between the representation of the sample images in the feature and pose space respectively, that is shown to boost regression performance. Furthermore, we argue that our pose-guided feature learning using our Siamese Regression Network generates more discriminative features that outperform the state of the art. Last, our feature learning formulation provides the ability of learning features that can perform under severe occlusions, demonstrating high performance on our novel hand-object dataset. Concluding, this work is a research on the area of object detection and pose estimation in 3D space, on a variety of object types. Furthermore we investigate how accuracy can be further improved by applying active vision techniques to optimally move the camera view to minimize the detection error.Open Acces

Data-driven robotic manipulation of cloth-like deformable objects : the present, challenges and future prospects

Manipulating cloth-like deformable objects (CDOs) is a long-standing problem in the robotics community. CDOs are flexible (non-rigid) objects that do not show a detectable level of compression strength while two points on the article are pushed towards each other and include objects such as ropes (1D), fabrics (2D) and bags (3D). In general, CDOs’ many degrees of freedom (DoF) introduce severe self-occlusion and complex state–action dynamics as significant obstacles to perception and manipulation systems. These challenges exacerbate existing issues of modern robotic control methods such as imitation learning (IL) and reinforcement learning (RL). This review focuses on the application details of data-driven control methods on four major task families in this domain: cloth shaping, knot tying/untying, dressing and bag manipulation. Furthermore, we identify specific inductive biases in these four domains that present challenges for more general IL and RL algorithms.Publisher PDFPeer reviewe