Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors amid challenging cases of clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after only a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.eduComment: To appear at the International Conference On Intelligent Robots and Systems (IROS) 2018. Project webpage: http://vpg.cs.princeton.edu Summary video: https://youtu.be/-OkyX7Zlhi

Parallel-Jaw Gripper and Grasp Co-Optimization for Sets of Planar Objects

We propose a framework for optimizing a planar parallel-jaw gripper for use with multiple objects. While optimizing general-purpose grippers and contact locations for grasps are both well studied, co-optimizing grasps and the gripper geometry to execute them receives less attention. As such, our framework synthesizes grippers optimized to stably grasp sets of polygonal objects. Given a fixed number of contacts and their assignments to object faces and gripper jaws, our framework optimizes contact locations along these faces, gripper pose for each grasp, and gripper shape. Our key insights are to pose shape and contact constraints in frames fixed to the gripper jaws, and to leverage the linearity of constraints in our grasp stability and gripper shape models via an augmented Lagrangian formulation. Together, these enable a tractable nonlinear program implementation. We apply our method to several examples. The first illustrative problem shows the discovery of a geometrically simple solution where possible. In another, space is constrained, forcing multiple objects to be contacted by the same features as each other. Finally a toolset-grasping example shows that our framework applies to complex, real-world objects. We provide a physical experiment of the toolset grasps.Comment: 2023 IEEE IROS conferenc

Analysis and Observations from the First Amazon Picking Challenge

This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge

On alternative uses of structural compliance for the development of adaptive robot grippers and hands

Adaptive robot hands are typically created by introducing structural compliance either in their joints (e.g., implementation of flexures joints) or in their finger-pads. In this paper, we present a series of alternative uses of structural compliance for the development of simple, adaptive, compliant and/or under-actuated robot grippers and hands that can efficiently and robustly execute a variety of grasping and dexterous, in-hand manipulation tasks. The proposed designs utilize only one actuator per finger to control multiple degrees of freedom and they retain the superior grasping capabilities of the adaptive grasping mechanisms even under significant object pose or other environmental uncertainties. More specifically, in this work, we introduce, discuss, and evaluate: (a) a design of pre-shaped, compliant robot fingers that adapts/conforms to the object geometry, (b) a hyper-adaptive finger-pad design that maximizes the area of the contact patches between the hand and the object, maximizing also grasp stability, and (c) a design that executes compliance adjustable manipulation tasks that can be predetermined by tuning the in-series compliance of the tendon routing system and by appropriately selecting the imposed tendon loads. The grippers are experimentally tested and their efficiency is validated using three different types of tests: (i) grasping tests that involve different everyday objects, (ii) grasp quality tests that estimate the contact area between the grippers and the objects grasped, and (iii) dexterous, in-hand manipulation experiments to evaluate the manipulation capabilities of the Compliance Adjustable Manipulation (CAM) hand. The devices employ mechanical adaptability to facilitate and simplify the efficient execution of robust grasping and dexterous, in-hand manipulation tasks

The Fractal Hand-II: Reviving a Classic Mechanism for Contemporary Grasping Challenges

This paper, and its companion, propose a new fractal robotic gripper, drawing inspiration from the century-old Fractal Vise. The unusual synergistic properties allow it to passively conform to diverse objects using only one actuator. Designed to be easily integrated with prevailing parallel jaw grippers, it alleviates the complexities tied to perception and grasp planning, especially when dealing with unpredictable object poses and geometries. We build on the foundational principles of the Fractal Vise to a broader class of gripping mechanisms, and also address the limitations that had led to its obscurity. Two Fractal Fingers, coupled by a closing actuator, can form an adaptive and synergistic Fractal Hand. We articulate a design methodology for low cost, easy to fabricate, large workspace, and compliant Fractal Fingers. The companion paper delves into the kinematics and grasping properties of a specific class of Fractal Fingers and Hands.Comment: This paper is prepared for ICRA 202

AntGrip -- Boosting parallel plate gripper performance inspired by the internal hairs of ant mandibles

Ants use their mandibles - effectively a two-finger gripper - for a wide range of grasping activities. Here we investigate whether mimicking the internal hairs found on ant mandibles can improve performance of a two-finger parallel plate robot gripper. With bin picking applications in mind, the gripper fingers are long and slim, with interchangeable soft gripping pads that can be hairy or hairless. A total of 2400 video-documented experiments have been conducted, comparing hairless to hairy pads with different hair patterns. Simply by adding hairs, the grasp success rate was increased by at least 29%, and the number of objects that remain securely gripped during manipulation more than doubled. This result not only advances the state of the art in grasping technology, but also provides novel insight into the mechanical role of mandible hairs in ant biology

Multimodal Grasp Planner for Hybrid Grippers in Cluttered Scenes

Grasping a variety of objects is still an open problem in robotics, especially for cluttered scenarios. Multimodal grasping has been recognized as a promising strategy to improve the manipulation capabilities of a robotic system. This work presents a novel grasp planning algorithm for hybrid grippers that allows for multiple grasping modalities. In particular, the planner manages two-finger grasps, single or double suction grasps, and magnetic grasps. Grasps for different modalities are geometrically computed based on the cuboid and the material properties of the objects in the clutter. The presented framework is modular and can leverage any 6D pose estimation or material segmentation network as far as they satisfy the required interface. Furthermore, the planner can be applied to any (hybrid) gripper, provided the gripper clearance, finger width, and suction diameter. The approach is fast and has a low computational burden, as it uses geometric computations for grasp synthesis and selection. The performance of the system has been assessed with an experimental campaign in three manipulation scenarios of increasing difficulty using the objects of the YCB dataset and the DLR hybrid-compliant gripper