Task-oriented grasping (TOG) refers to the problem of predicting grasps on an
object that enable subsequent manipulation tasks. To model the complex
relationships between objects, tasks, and grasps, existing methods incorporate
semantic knowledge as priors into TOG pipelines. However, the existing semantic
knowledge is typically constructed based on closed-world concept sets,
restraining the generalization to novel concepts out of the pre-defined sets.
To address this issue, we propose GraspGPT, a large language model (LLM) based
TOG framework that leverages the open-end semantic knowledge from an LLM to
achieve zero-shot generalization to novel concepts. We conduct experiments on
Language Augmented TaskGrasp (LA-TaskGrasp) dataset and demonstrate that
GraspGPT outperforms existing TOG methods on different held-out settings when
generalizing to novel concepts out of the training set. The effectiveness of
GraspGPT is further validated in real-robot experiments. Our code, data,
appendix, and video are publicly available at
https://sites.google.com/view/graspgpt/.Comment: 15 pages, 8 figure