12 research outputs found
Learning grasp affordance reasoning through semantic relations
Reasoning about object affordances allows an autonomous agent to perform
generalised manipulation tasks among object instances. While current approaches
to grasp affordance estimation are effective, they are limited to a single
hypothesis. We present an approach for detection and extraction of multiple
grasp affordances on an object via visual input. We define semantics as a
combination of multiple attributes, which yields benefits in terms of
generalisation for grasp affordance prediction. We use Markov Logic Networks to
build a knowledge base graph representation to obtain a probability
distribution of grasp affordances for an object. To harvest the knowledge base,
we collect and make available a novel dataset that relates different semantic
attributes. We achieve reliable mappings of the predicted grasp affordances on
the object by learning prototypical grasping patches from several examples. We
show our method's generalisation capabilities on grasp affordance prediction
for novel instances and compare with similar methods in the literature.
Moreover, using a robotic platform, on simulated and real scenarios, we
evaluate the success of the grasping task when conditioned on the grasp
affordance prediction.Comment: Accepted in IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS) 201
Affordance-Aware Handovers With Human Arm Mobility Constraints
Reasoning about object handover configurations allows an assistive agent to
estimate the appropriateness of handover for a receiver with different arm
mobility capacities. While there are existing approaches for estimating the
effectiveness of handovers, their findings are limited to users without arm
mobility impairments and to specific objects. Therefore, current
state-of-the-art approaches are unable to hand over novel objects to receivers
with different arm mobility capacities. We propose a method that generalises
handover behaviours to previously unseen objects, subject to the constraint of
a user's arm mobility levels and the task context. We propose a
heuristic-guided hierarchically optimised cost whose optimisation adapts object
configurations for receivers with low arm mobility. This also ensures that the
robot grasps consider the context of the user's upcoming task, i.e., the usage
of the object. To understand preferences over handover configurations, we
report on the findings of an online study, wherein we presented different
handover methods, including ours, to users with different levels of arm
mobility. We find that people's preferences over handover methods are
correlated to their arm mobility capacities. We encapsulate these preferences
in a statistical relational model (SRL) that is able to reason about the most
suitable handover configuration given a receiver's arm mobility and upcoming
task. Using our SRL model, we obtained an average handover accuracy of
when generalising handovers to novel objects.Comment: Accepted for RA-L 202
GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping
Task-oriented grasping (TOG) refers to the problem of predicting grasps on an
object that enable subsequent manipulation tasks. To model the complex
relationships between objects, tasks, and grasps, existing methods incorporate
semantic knowledge as priors into TOG pipelines. However, the existing semantic
knowledge is typically constructed based on closed-world concept sets,
restraining the generalization to novel concepts out of the pre-defined sets.
To address this issue, we propose GraspGPT, a large language model (LLM) based
TOG framework that leverages the open-end semantic knowledge from an LLM to
achieve zero-shot generalization to novel concepts. We conduct experiments on
Language Augmented TaskGrasp (LA-TaskGrasp) dataset and demonstrate that
GraspGPT outperforms existing TOG methods on different held-out settings when
generalizing to novel concepts out of the training set. The effectiveness of
GraspGPT is further validated in real-robot experiments. Our code, data,
appendix, and video are publicly available at
https://sites.google.com/view/graspgpt/.Comment: 15 pages, 8 figure
Affordance-Based Human-Robot Interaction With Reinforcement Learning
Planning precise manipulation in robotics to perform grasp and release-related operations, while interacting with humans is a challenging problem. Reinforcement learning (RL) has the potential to make robots attain this capability. In this paper, we propose an affordance-based human-robot interaction (HRI) framework, aiming to reduce the action space size that would considerably impede the exploration efficiency of the agent. The framework is based on a new algorithm called Contextual Q-learning (CQL). We first show that the proposed algorithm trains in a reduced amount of time (2.7 seconds) and reaches an 84% of success rate. This suits the robot’s learning efficiency to observe the current scenario configuration and learn to solve it. Then, we empirically validate the framework for implementation in HRI real-world scenarios. During the HRI, the robot uses semantic information from the state and the optimal policy of the last training step to search for relevant changes in the environment that may trigger the generation of a new policy
Reasoning and understanding grasp affordances for robot manipulation
This doctoral research focuses on developing new methods that enable an artificial agent
to grasp and manipulate objects autonomously. More specifically, we are using the concept
of affordances to learn and generalise robot grasping and manipulation techniques. [75] defined affordances as the ability of an agent to perform a certain action with an object in a
given environment. In robotics, affordances defines the possibility of an agent to perform
actions with an object. Therefore, by understanding the relation between actions, objects
and the effect of these actions, the agent understands the task at hand, providing the robot
with the potential to bridge perception to action. The significance of affordances in robotics
has been studied from varied perspectives, such as psychology and cognitive sciences.
Many efforts have been made to pragmatically employ the concept of affordances as it
provides the potential for an artificial agent to perform tasks autonomously. We start by reviewing and finding common ground amongst different strategies that use affordances for
robotic tasks. We build on the identified grounds to provide guidance on including the concept of affordances as a medium to boost autonomy for an artificial agent. To this end, we
outline common design choices to build an affordance relation; and their implications on
the generalisation capabilities of the agent when facing previously unseen scenarios. Based
on our exhaustive review, we conclude that prior research on object affordance detection
is effective, however, among others, it has the following technical gaps: (i) the methods are
limited to a single object ↔ affordance hypothesis, and (ii) they cannot guarantee task completion or any level of performance for the manipulation task alone nor (iii) in collaboration
with other agents. In this research thesis, we propose solutions to these technical challenges.
In an incremental fashion, we start by addressing the limited generalisation capabilities
of, at the time state-of-the-art methods, by strengthening the perception to action connection through the construction of an Knowledge Base (KB). We then leverage the information
encapsulated in the KB to design and implement a reasoning and understanding method
based on statistical relational leaner (SRL) that allows us to cope with uncertainty in testing
environments, and thus, improve generalisation capabilities in affordance-aware manipulation tasks. The KB in conjunctions with our SRL are the base for our designed solutions
that guarantee task completion when the robot is performing a task alone as well as when in
collaboration with other agents. We finally expose and discuss a range of interesting avenues
that have the potential to thrive the capabilities of a robotic agent through the use of the
concept of affordances for manipulation tasks. A summary of the contributions of this thesis
can be found at: https://bit.ly/grasp_affordance_reasonin
Graph learning in robotics: a survey
Deep neural networks for graphs have emerged as a powerful tool for learning
on complex non-euclidean data, which is becoming increasingly common for a
variety of different applications. Yet, although their potential has been
widely recognised in the machine learning community, graph learning is largely
unexplored for downstream tasks such as robotics applications. To fully unlock
their potential, hence, we propose a review of graph neural architectures from
a robotics perspective. The paper covers the fundamentals of graph-based
models, including their architecture, training procedures, and applications. It
also discusses recent advancements and challenges that arise in applied
settings, related for example to the integration of perception,
decision-making, and control. Finally, the paper provides an extensive review
of various robotic applications that benefit from learning on graph structures,
such as bodies and contacts modelling, robotic manipulation, action
recognition, fleet motion planning, and many more. This survey aims to provide
readers with a thorough understanding of the capabilities and limitations of
graph neural architectures in robotics, and to highlight potential avenues for
future research
Behaviour-driven motion synthesis
Heightened demand for alternatives to human exposure to strenuous and repetitive labour, as
well as to hazardous environments, has led to an increased interest in real-world deployment of
robotic agents. Targeted applications require robots to be adept at synthesising complex
motions rapidly across a wide range of tasks and environments. To this end, this thesis
proposes leveraging abstractions of the problem at hand to ease and speed up the solving. We
formalise abstractions to hint relevant robotic behaviour to a family of planning problems, and
integrate them tightly into the motion synthesis process to make real-world deployment in
complex environments practical. We investigate three principal challenges of this proposition.
Firstly, we argue that behavioural samples in form of trajectories are of particular interest to
guide robotic motion synthesis. We formalise a framework with behavioural semantic annotation
that enables the storage and bootstrap of sets of problem-relevant trajectories.
Secondly, in the core of this thesis, we study strategies to exploit behavioural samples in task
instantiations that differ significantly from those stored in the framework. We present two
novel strategies to efficiently leverage offline-computed problem behavioural samples: (i) online
modulation based on geometry-tuned potential fields, and (ii) experience-guided exploration
based on trajectory segmentation and malleability.
Thirdly, we demonstrate that behavioural hints can be extracted on-the-fly to tackle highlyconstrained, ever-changing complex problems, from which there is no prior knowledge. We
propose a multi-layer planner that first solves a simplified version of the problem at hand, to
then inform the search for a solution in the constrained space.
Our contributions on efficient motion synthesis via behaviour guidance augment the robots’
capabilities to deal with more complex planning problems, and do so more effectively than
related approaches in the literature by computing better quality paths in lower response time.
We demonstrate our contributions, in both laboratory experiments and field trials, on a
spectrum of planning problems and robotic platforms ranging from high-dimensional
humanoids and robotic arms with a focus on autonomous manipulation in resembling
environments, to high-dimensional kinematic motion planning with a focus on autonomous safe navigation in unknown environments. While this thesis was motivated by challenges on motion
synthesis, we have explored the applicability of our findings on disparate robotic fields, such as
grasp and task planning. We have made some of our contributions open-source hoping they
will be of use to the robotics community at large.The CDT in Robotics and Autonomous Systems at Heriot-Watt University and The University of EdinburghThe ORCA Hub EPSRC project (EP/R026173/1)The Scottish Informatics and Computer Science
Alliance (SICSA