Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching

This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.eduComment: Project webpage: http://arc.cs.princeton.edu Summary video: https://youtu.be/6fG7zwGfIk

Construction Scene Point Cloud Acquisition, Object Finding and Clutter Removal in Real Time

Within industrial construction, piping can constitute up to 50% of the cost of a typical project. It has been shown that across the activities involved in pipe fabrication, pipe fitting has the highest impact on the critical path. The pipe fitter is responsible for interpreting the isometric drawing and then performing the tack welds on piping components so that the assembly complies with the design. Three main problems in doing this task are identified as: (1) reading and interpreting the isometric drawing is challenging and error prone for spatially complicated assemblies, (2) in assemblies with tight allowable tolerance, a number of iterations will take place to fit the pipes with compliance to the design. These iterations (rework) will remain unrecorded in the production process, and (3) no continuous measurement tool exists to let the fitter check his/her work in progress against the design information and acceptance specifications. Addressing these problems could substantially improve pipe fitters’ productivity. The objective of this research is to develop a software package integrating a threefold solution to simplify complex tasks involved in pipe fabrication: (1) making design information easier to understand, with the use of a tablet, 3D imaging device and an application software, (2) providing visual feedback on the correctness of fabrication between the design intent and the as-built state, and (3) providing frequent feedback on fabrication using a step-by-step assembly and control framework. The step-by-step framework will reduce the number of required iterations for the pipe fitter. A number of challenges were encountered in order to provide a framework to make real time, visual and frequent feedback. For frequent and visual feedback, a real time 3D data acquisition tool with an acceptable level of accuracy should be adopted. This is due to the speed of fabrication in an industrial facility. The second challenge is to find the object of interest in real time, once a point cloud is acquired, and finally, once the object is found, to optimally remove points that are considered as clutter to improve the visual feedback for the pipe fitters. To address the requirement for a reliable and real time acquisition tool, Chapter 3 explores the capabilities and limitations of low cost range cameras. A commercially available 3D imaging tool was utilized to measure its performance for real time point cloud acquisition. The device was used to inspect two pipe spools altered in size. The acquired point clouds were super-imposed on the BIM (Building Information Model) model of the pipe spools to measure the accuracy of the device. Chapter 4 adapts and examines a real time and automatic object finding algorithm to measure its performance with respect to construction challenges. Then, a K-Nearest Neighbor (KNN) algorithm was employed to classify points as being clutter or corresponding to the object of interest. Chapter 5 investigates the effect of the threshold value “K” in the K-Nearest Neighbor algorithm and optimizing its value for an improved visual feedback. As a result of the work described in this thesis, along with the work of two other master students and a co-op student, a software package was designed and developed. The software package takes advantage of the investigated real time point cloud acquisition device. While the object finding algorithm proved to be effective, a 3-point matching algorithm was used, as it was more intuitive for the users and took less time. The KNN algorithm was utilized to remove clutter points to provide more accurate visual feedback more accurate to the workers

Grasping bulky objects with two anthropomorphic hands

© 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThis paper presents an algorithm to compute precision grasps for bulky objects using two anthropomorphic hands. We use objects modeled as point clouds obtained from a sensor camera or from a CAD model. We then process the point clouds dividing them into two set of slices where we look for sets of triplets of points. Each triplet must accomplish some physical conditions based on the structure of the hands. Then, the triplets of points from each set of slices are evaluated to find a combination that satisfies the force closure condition (FC). Once one valid couple of triplets have been found the inverse kinematics of the system is computed in order to know if the corresponding points are reachable by the hands, if so, motion planning and a collision check are performed to asses if the final grasp configuration of the system is suitable. The paper inclu des some application examples of the proposed approachAccepted versio

View suggestion for interactive segmentation of indoor scenes

Point cloud segmentation is a fundamental problem. Due to the complexity of real-world scenes and the limitations of 3D scanners, interactive segmentation is currently the only way to cope with all kinds of point clouds. However, interactively segmenting complex and large-scale scenes is very time-consuming. In this paper, we present a novel interactive system for segmenting point cloud scenes. Our system automatically suggests a series of camera views, in which users can conveniently specify segmentation guidance. In this way, users may focus on specifying segmentation hints instead of manually searching for desirable views of unsegmented objects, thus significantly reducing user effort. To achieve this, we introduce a novel view preference model, which is based on a set of dedicated view attributes, with weights learned from a user study. We also introduce support relations for both graph-cut-based segmentation and finding similar objects. Our experiments show that our segmentation technique helps users quickly segment various types of scenes, outperforming alternative methods