1,695 research outputs found
Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes
We present the Semantic Robot Programming (SRP) paradigm as a convergence of
robot programming by demonstration and semantic mapping. In SRP, a user can
directly program a robot manipulator by demonstrating a snapshot of their
intended goal scene in workspace. The robot then parses this goal as a scene
graph comprised of object poses and inter-object relations, assuming known
object geometries. Task and motion planning is then used to realize the user's
goal from an arbitrary initial scene configuration. Even when faced with
different initial scene configurations, SRP enables the robot to seamlessly
adapt to reach the user's demonstrated goal. For scene perception, we propose
the Discriminatively-Informed Generative Estimation of Scenes and Transforms
(DIGEST) method to infer the initial and goal states of the world from RGBD
images. The efficacy of SRP with DIGEST perception is demonstrated for the task
of tray-setting with a Michigan Progress Fetch robot. Scene perception and task
execution are evaluated with a public household occlusion dataset and our
cluttered scene dataset.Comment: published in ICRA 201
Human-Assisted Continual Robot Learning with Foundation Models
Large Language Models (LLMs) have been shown to act like planners that can
decompose high-level instructions into a sequence of executable instructions.
However, current LLM-based planners are only able to operate with a fixed set
of skills. We overcome this critical limitation and present a method for using
LLM-based planners to query new skills and teach robots these skills in a data
and time-efficient manner for rigid object manipulation. Our system can re-use
newly acquired skills for future tasks, demonstrating the potential of open
world and lifelong learning. We evaluate the proposed framework on multiple
tasks in simulation and the real world. Videos are available at:
https://sites.google.com/mit.edu/halp-robot-learning
- …