1,163 research outputs found
A Visuo-Tactile Control Framework for Manipulation and Exploration of Unknown Objects
Li Q, Haschke R, Ritter H. A Visuo-Tactile Control Framework for Manipulation and Exploration of Unknown Objects. Presented at the Humanoids2015, Seoul,Korea.We present a novel hierarchical control framework
that unifies our previous work on tactile-servoing with
visual-servoing approaches to allow for robust manipulation
and exploration of unknown objects, including – but not
limited to – robust grasping, online grasp optimization, in-hand
manipulation, and exploration of object surfaces. The control
framework is divided into three layers: a joint-level positioncontrol
layer, a tactile servoing control layer, and a high-level
visual servoing control layer. While the middle layer provides
“blind” surface exploration skills, maintaining desired contact
patterns, the visual layer monitors and controls the actual object
pose providing high-level finger-tip motion commands that are
merged with the tactile-servoing control commands.
Because the high spatial resolution tactile array and tactile
servoing method is used, the robot end-effector can actively
perform slide, roll and twist motion in order to improve the
contact quality with the unknown object only depending on
the tactile feedback. Our control method can be consider
as another alternative option for vision-force shared control
method and vision-force-tactile control method which heavily
depend on the 3D force/torque sensor to perform end-effector
fine manipulation after the contact happening.
We illustrate the efficiency of the proposed framework using
a series of manipulation actions performed with two KUKA
LWR arms equipped with a tactile sensor array as a “sensitive
fingertip”. The two considered objects are unknown to the
robot, i.e. neither shape nor friction properties are available
Push to know! -- Visuo-Tactile based Active Object Parameter Inference with Dual Differentiable Filtering
For robotic systems to interact with objects in dynamic environments, it is
essential to perceive the physical properties of the objects such as shape,
friction coefficient, mass, center of mass, and inertia. This not only eases
selecting manipulation action but also ensures the task is performed as
desired. However, estimating the physical properties of especially novel
objects is a challenging problem, using either vision or tactile sensing. In
this work, we propose a novel framework to estimate key object parameters using
non-prehensile manipulation using vision and tactile sensing. Our proposed
active dual differentiable filtering (ADDF) approach as part of our framework
learns the object-robot interaction during non-prehensile object push to infer
the object's parameters. Our proposed method enables the robotic system to
employ vision and tactile information to interactively explore a novel object
via non-prehensile object push. The novel proposed N-step active formulation
within the differentiable filtering facilitates efficient learning of the
object-robot interaction model and during inference by selecting the next best
exploratory push actions (where to push? and how to push?). We extensively
evaluated our framework in simulation and real-robotic scenarios, yielding
superior performance to the state-of-the-art baseline.Comment: 8 pages. Accepted at IROS 202
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
For humans, the process of grasping an object relies heavily on rich tactile
feedback. Most recent robotic grasping work, however, has been based only on
visual input, and thus cannot easily benefit from feedback after initiating
contact. In this paper, we investigate how a robot can learn to use tactile
information to iteratively and efficiently adjust its grasp. To this end, we
propose an end-to-end action-conditional model that learns regrasping policies
from raw visuo-tactile data. This model -- a deep, multimodal convolutional
network -- predicts the outcome of a candidate grasp adjustment, and then
executes a grasp by iteratively selecting the most promising actions. Our
approach requires neither calibration of the tactile sensors, nor any
analytical modeling of contact forces, thus reducing the engineering effort
required to obtain efficient grasping policies. We train our model with data
from about 6,450 grasping trials on a two-finger gripper equipped with GelSight
high-resolution tactile sensors on each finger. Across extensive experiments,
our approach outperforms a variety of baselines at (i) estimating grasp
adjustment outcomes, (ii) selecting efficient grasp adjustments for quick
grasping, and (iii) reducing the amount of force applied at the fingers, while
maintaining competitive performance. Finally, we study the choices made by our
model and show that it has successfully acquired useful and interpretable
grasping behaviors.Comment: 8 pages. Published on IEEE Robotics and Automation Letters (RAL).
Website: https://sites.google.com/view/more-than-a-feelin
Tactile Mapping and Localization from High-Resolution Tactile Imprints
This work studies the problem of shape reconstruction and object localization
using a vision-based tactile sensor, GelSlim. The main contributions are the
recovery of local shapes from contact, an approach to reconstruct the tactile
shape of objects from tactile imprints, and an accurate method for object
localization of previously reconstructed objects. The algorithms can be applied
to a large variety of 3D objects and provide accurate tactile feedback for
in-hand manipulation. Results show that by exploiting the dense tactile
information we can reconstruct the shape of objects with high accuracy and do
on-line object identification and localization, opening the door to reactive
manipulation guided by tactile sensing. We provide videos and supplemental
information in the project's website
http://web.mit.edu/mcube/research/tactile_localization.html.Comment: ICRA 2019, 7 pages, 7 figures. Website:
http://web.mit.edu/mcube/research/tactile_localization.html Video:
https://youtu.be/uMkspjmDbq
Visuo-Haptic Grasping of Unknown Objects through Exploration and Learning on Humanoid Robots
Die vorliegende Arbeit befasst sich mit dem Greifen unbekannter Objekte durch humanoide Roboter. Dazu werden visuelle Informationen mit haptischer Exploration kombiniert, um Greifhypothesen zu erzeugen. Basierend auf simulierten Trainingsdaten wird außerdem eine Greifmetrik gelernt, welche die Erfolgswahrscheinlichkeit der Greifhypothesen bewertet und die mit der größten geschätzten Erfolgswahrscheinlichkeit auswählt. Diese wird verwendet, um Objekte mit Hilfe einer reaktiven Kontrollstrategie zu greifen. Die zwei Kernbeiträge der Arbeit sind zum einen die haptische Exploration von unbekannten Objekten und zum anderen das Greifen von unbekannten Objekten mit Hilfe einer neuartigen datengetriebenen Greifmetrik
Dexterous manipulation of unknown objects using virtual contact points
The manipulation of unknown objects is a problem of special interest in robotics since it is not always possible to have exact models of the objects with which the robot interacts. This paper presents a simple strategy to manipulate unknown objects using a robotic hand equipped with tactile sensors. The hand configurations that allow the rotation of an unknown object are computed using only tactile and kinematic information, obtained during the manipulation process and reasoning about the desired and real positions of the fingertips during the manipulation. This is done taking into account that the desired positions of the fingertips are not physically reachable since they are located in the interior of the manipulated object and therefore they are virtual positions with associated virtual contact points. The proposed approach was satisfactorily validated using three fingers of an anthropomorphic robotic hand (Allegro Hand), with the original fingertips replaced by tactile sensors (WTS-FT). In the experimental validation, several everyday objects with different shapes were successfully manipulated, rotating them without the need of knowing their shape or any other physical property.Peer ReviewedPostprint (author's final draft
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Humans make extensive use of vision and touch as complementary senses, with
vision providing global information about the scene and touch measuring local
information during manipulation without suffering from occlusions. While prior
work demonstrates the efficacy of tactile sensing for precise manipulation of
deformables, they typically rely on supervised, human-labeled datasets. We
propose Self-Supervised Visuo-Tactile Pretraining (SSVTP), a framework for
learning multi-task visuo-tactile representations in a self-supervised manner
through cross-modal supervision. We design a mechanism that enables a robot to
autonomously collect precisely spatially-aligned visual and tactile image
pairs, then train visual and tactile encoders to embed these pairs into a
shared latent space using cross-modal contrastive loss. We apply this latent
space to downstream perception and control of deformable garments on flat
surfaces, and evaluate the flexibility of the learned representations without
fine-tuning on 5 tasks: feature classification, contact localization, anomaly
detection, feature search from a visual query (e.g., garment feature
localization under occlusion), and edge following along cloth edges. The
pretrained representations achieve a 73-100% success rate on these 5 tasks.Comment: RSS 2023, site: https://sites.google.com/berkeley.edu/ssvt
VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects
Deformable objects manipulation can benefit from representations that
seamlessly integrate vision and touch while handling occlusions. In this work,
we present a novel approach for, and real-world demonstration of, multimodal
visuo-tactile state-estimation and dynamics prediction for deformable objects.
Our approach, VIRDO++, builds on recent progress in multimodal neural implicit
representations for deformable object state-estimation [1] via a new
formulation for deformation dynamics and a complementary state-estimation
algorithm that (i) maintains a belief over deformations, and (ii) enables
practical real-world application by removing the need for privileged contact
information. In the context of two real-world robotic tasks, we show:(i)
high-fidelity cross-modal state-estimation and prediction of deformable objects
from partial visuo-tactile feedback, and (ii) generalization to unseen objects
and contact formations
- …