18 research outputs found
Blocks2World: Controlling Realistic Scenes with Editable Primitives
We present Blocks2World, a novel method for 3D scene rendering and editing
that leverages a two-step process: convex decomposition of images and
conditioned synthesis. Our technique begins by extracting 3D parallelepipeds
from various objects in a given scene using convex decomposition, thus
obtaining a primitive representation of the scene. These primitives are then
utilized to generate paired data through simple ray-traced depth maps. The next
stage involves training a conditioned model that learns to generate images from
the 2D-rendered convex primitives. This step establishes a direct mapping
between the 3D model and its 2D representation, effectively learning the
transition from a 3D model to an image. Once the model is fully trained, it
offers remarkable control over the synthesis of novel and edited scenes. This
is achieved by manipulating the primitives at test time, including translating
or adding them, thereby enabling a highly customizable scene rendering process.
Our method provides a fresh perspective on 3D scene rendering and editing,
offering control and flexibility. It opens up new avenues for research and
applications in the field, including authoring and data augmentation.Comment: 16 pages, 15 figure
Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learning
Rearranging objects on a tabletop surface by means of nonprehensile
manipulation is a task which requires skillful interaction with the physical
world. Usually, this is achieved by precisely modeling physical properties of
the objects, robot, and the environment for explicit planning. In contrast, as
explicitly modeling the physical environment is not always feasible and
involves various uncertainties, we learn a nonprehensile rearrangement strategy
with deep reinforcement learning based on only visual feedback. For this, we
model the task with rewards and train a deep Q-network. Our potential
field-based heuristic exploration strategy reduces the amount of collisions
which lead to suboptimal outcomes and we actively balance the training set to
avoid bias towards poor examples. Our training process leads to quicker
learning and better performance on the task as compared to uniform exploration
and standard experience replay. We demonstrate empirical evidence from
simulation that our method leads to a success rate of 85%, show that our system
can cope with sudden changes of the environment, and compare our performance
with human level performance.Comment: 2018 International Conference on Robotics and Automatio
Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation
In this paper, we present a novel method for achieving dexterous manipulation
of complex objects, while simultaneously securing the object without the use of
passive support surfaces. We posit that a key difficulty for training such
policies in a Reinforcement Learning framework is the difficulty of exploring
the problem state space, as the accessible regions of this space form a complex
structure along manifolds of a high-dimensional space. To address this
challenge, we use two versions of the non-holonomic Rapidly-Exploring Random
Trees algorithm; one version is more general, but requires explicit use of the
environment's transition function, while the second version uses
manipulation-specific kinematic constraints to attain better sample efficiency.
In both cases, we use states found via sampling-based exploration to generate
reset distributions that enable training control policies under full dynamic
constraints via model-free Reinforcement Learning. We show that these policies
are effective at manipulation problems of higher difficulty than previously
shown, and also transfer effectively to real robots. Videos of the real-hand
demonstrations can be found on the project website:
https://sbrl.cs.columbia.edu/Comment: 10 pages, 6 figures, submitted to Robotics Science & Systems 202
Manipulation Planning Among Movable Obstacles Using Physics-Based Adaptive Motion Primitives
Robot manipulation in cluttered scenes often requires contact-rich
interactions with objects. It can be more economical to interact via
non-prehensile actions, for example, push through other objects to get to the
desired grasp pose, instead of deliberate prehensile rearrangement of the
scene. For each object in a scene, depending on its properties, the robot may
or may not be allowed to make contact with, tilt, or topple it. To ensure that
these constraints are satisfied during non-prehensile interactions, a planner
can query a physics-based simulator to evaluate the complex multi-body
interactions caused by robot actions. Unfortunately, it is infeasible to query
the simulator for thousands of actions that need to be evaluated in a typical
planning problem as each simulation is time-consuming. In this work, we show
that (i) manipulation tasks (specifically pick-and-place style tasks from a
tabletop or a refrigerator) can often be solved by restricting robot-object
interactions to adaptive motion primitives in a plan, (ii) these actions can be
incorporated as subgoals within a multi-heuristic search framework, and (iii)
limiting interactions to these actions can help reduce the time spent querying
the simulator during planning by up to 40x in comparison to baseline
algorithms. Our algorithm is evaluated in simulation and in the real-world on a
PR2 robot using PyBullet as our physics-based simulator. Supplementary video:
\url{https://youtu.be/ABQc7JbeJPM}.Comment: Under review for the IEEE Robotics and Automation Letters (RA-L)
journal with conference presentation option at the 2021 International
Conference on Robotics and Automation (ICRA). This work has been submitted to
the IEEE for possible publication. Copyright may be transferred without
notice, after which this version may no longer be accessibl
Randomized physics-based motion planning for grasping in cluttered and uncertain environments
© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksPlanning motions to grasp an object in cluttered and uncertain environments is a challenging task, particularly when a collision-free trajectory does not exist and objects obstructing the way are required to be carefully grasped and moved out. This letter takes a different approach and proposes to address this problem by using a randomized physics-based motion planner that permits robot–object and object–object interactions. The main idea is to avoid an explicit high-level reasoning of the task by providing the
motion planner with a physics engine to evaluate possible complex multibody dynamical interactions. The approach is able to solve the problem in complex scenarios, also considering uncertainty in the objects’ pose and in the contact dynamics. The work enhances the state validity checker, the control sampler, and the tree exploration strategy of a kinodynamic motion planner called KPIECE. The enhanced algorithm, called p-KPIECE, has been validated in simulation and with real experiments. The results have been compared with an ontological physics-based motion planner and with task and motion planning approaches, resulting in a significant improvement in terms of planning time, success rate, and quality of the solution path.Peer ReviewedPostprint (author's final draft