272 research outputs found
Learning at the Ends: From Hand to Tool Affordances in Humanoid Robots
One of the open challenges in designing robots that operate successfully in
the unpredictable human environment is how to make them able to predict what
actions they can perform on objects, and what their effects will be, i.e., the
ability to perceive object affordances. Since modeling all the possible world
interactions is unfeasible, learning from experience is required, posing the
challenge of collecting a large amount of experiences (i.e., training data).
Typically, a manipulative robot operates on external objects by using its own
hands (or similar end-effectors), but in some cases the use of tools may be
desirable, nevertheless, it is reasonable to assume that while a robot can
collect many sensorimotor experiences using its own hands, this cannot happen
for all possible human-made tools.
Therefore, in this paper we investigate the developmental transition from
hand to tool affordances: what sensorimotor skills that a robot has acquired
with its bare hands can be employed for tool use? By employing a visual and
motor imagination mechanism to represent different hand postures compactly, we
propose a probabilistic model to learn hand affordances, and we show how this
model can generalize to estimate the affordances of previously unseen tools,
ultimately supporting planning, decision-making and tool selection tasks in
humanoid robots. We present experimental results with the iCub humanoid robot,
and we publicly release the collected sensorimotor data in the form of a hand
posture affordances dataset.Comment: dataset available at htts://vislab.isr.tecnico.ulisboa.pt/, IEEE
International Conference on Development and Learning and on Epigenetic
Robotics (ICDL-EpiRob 2017
Affordances in Psychology, Neuroscience, and Robotics: A Survey
The concept of affordances appeared in psychology during the late 60s as an alternative perspective on the visual perception of the environment. It was revolutionary in the intuition that the way living beings perceive the world is deeply influenced by the actions they are able to perform. Then, across the last 40 years, it has influenced many applied fields, e.g., design, human-computer interaction, computer vision, and robotics. In this paper, we offer a multidisciplinary perspective on the notion of affordances. We first discuss the main definitions and formalizations of the affordance theory, then we report the most significant evidence in psychology and neuroscience that support it, and finally we review the most relevant applications of this concept in robotics
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Large language models (LLMs) are shown to possess a wealth of actionable
knowledge that can be extracted for robot manipulation in the form of reasoning
and planning. Despite the progress, most still rely on pre-defined motion
primitives to carry out the physical interactions with the environment, which
remains a major bottleneck. In this work, we aim to synthesize robot
trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
large variety of manipulation tasks given an open-set of instructions and an
open-set of objects. We achieve this by first observing that LLMs excel at
inferring affordances and constraints given a free-form language instruction.
More importantly, by leveraging their code-writing capabilities, they can
interact with a vision-language model (VLM) to compose 3D value maps to ground
the knowledge into the observation space of the agent. The composed value maps
are then used in a model-based planning framework to zero-shot synthesize
closed-loop robot trajectories with robustness to dynamic perturbations. We
further demonstrate how the proposed framework can benefit from online
experiences by efficiently learning a dynamics model for scenes that involve
contact-rich interactions. We present a large-scale study of the proposed
method in both simulated and real-robot environments, showcasing the ability to
perform a large variety of everyday manipulation tasks specified in free-form
natural language. Videos and code at https://voxposer.github.i
Invariant Feature Mappings for Generalizing Affordance Understanding Using Regularized Metric Learning
This paper presents an approach for learning invariant features for object
affordance understanding. One of the major problems for a robotic agent
acquiring a deeper understanding of affordances is finding sensory-grounded
semantics. Being able to understand what in the representation of an object
makes the object afford an action opens up for more efficient manipulation,
interchange of objects that visually might not be similar, transfer learning,
and robot to human communication. Our approach uses a metric learning algorithm
that learns a feature transform that encourages objects that affords the same
action to be close in the feature space. We regularize the learning, such that
we penalize irrelevant features, allowing the agent to link what in the sensory
input caused the object to afford the action. From this, we show how the agent
can abstract the affordance and reason about the similarity between different
affordances
A Survey of Knowledge Representation in Service Robotics
Within the realm of service robotics, researchers have placed a great amount
of effort into learning, understanding, and representing motions as
manipulations for task execution by robots. The task of robot learning and
problem-solving is very broad, as it integrates a variety of tasks such as
object detection, activity recognition, task/motion planning, localization,
knowledge representation and retrieval, and the intertwining of
perception/vision and machine learning techniques. In this paper, we solely
focus on knowledge representations and notably how knowledge is typically
gathered, represented, and reproduced to solve problems as done by researchers
in the past decades. In accordance with the definition of knowledge
representations, we discuss the key distinction between such representations
and useful learning models that have extensively been introduced and studied in
recent years, such as machine learning, deep learning, probabilistic modelling,
and semantic graphical structures. Along with an overview of such tools, we
discuss the problems which have existed in robot learning and how they have
been built and used as solutions, technologies or developments (if any) which
have contributed to solving them. Finally, we discuss key principles that
should be considered when designing an effective knowledge representation.Comment: Accepted for RAS Special Issue on Semantic Policy and Action
Representations for Autonomous Robots - 22 Page
- …