Lump detection with a gelsight sensor

A GelSight sensor is a tactile sensing device comprising a clear elastomeric pad covered with a reflective membrane, coupled with optics to measure the membrane's deformations. When the pad is pressed against an object's surface, the membrane changes shape in accord with mechanical and geometrical properties of the object. Since soft tissue is more compliant than hard tissue, one can detect an embedded lump by pressing the GelSight pad against the tissue surface and observing the hump that forms over the lump. We tested this system's sensitivity by constructing phantoms of soft rubber with hard embedded lumps. The system is quite sensitive; for example it could detect a 2mm lump at a depth of 5mm. The sensor was more sensitive than previous tactile lump detectors. It was also better than human observers using their fingertips. Such a capability could help in tumor screening, and could augment the sensory information available in telemedicine or minimally invasive surgery.National Science Foundation (U.S.) (Grant 6922551

Improved GelSight Tactile Sensor for Measuring Geometry and Slip

A GelSight sensor uses an elastomeric slab covered with a reflective membrane to measure tactile signals. It measures the 3D geometry and contact force information with high spacial resolution, and successfully helped many challenging robot tasks. A previous sensor, based on a semi-specular membrane, produces high resolution but with limited geometry accuracy. In this paper, we describe a new design of GelSight for robot gripper, using a Lambertian membrane and new illumination system, which gives greatly improved geometric accuracy while retaining the compact size. We demonstrate its use in measuring surface normals and reconstructing height maps using photometric stereo. We also use it for the task of slip detection, using a combination of information about relative motions on the membrane surface and the shear distortions. Using a robotic arm and a set of 37 everyday objects with varied properties, we find that the sensor can detect translational and rotational slip in general cases, and can be used to improve the stability of the grasp.Comment: IEEE/RSJ International Conference on Intelligent Robots and System

Tactile sensing using elastomeric sensors

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 99-111).GelSight, namely, elastomeric sensor, is a novel tactile sensor to get the 3D information of contacting surfaces. Using GelSight, some tactile properties, such as softness and roughness, could be gained through image processing techniques. In this thesis, I implemented GelSight principle to reconstruct surface geometry of tested surfaces, based on which, the roughness comparison and lump detection experiment are conducted. Roughness of five different types of sandpapers are successfully compared using GelSight Ra value. In the lump detection experiment, a visual display for tactile information is presented. To get binary feedback of lump presence or not, a simple threshold method is introduced in this thesis. To evaluate the performance of GelSight sensor, human psychological experiments are conducted. In similar tasks, GelSight sensor outperforms humans in lump detection.by Xiaodan (Stella) Jia.S.M

Shape-independent hardness estimation using deep learning and a GelSight tactile sensor

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale

Endoscopic Tactile Capsule for Non-Polypoid Colorectal Tumour Detection

An endoscopic tactile robotic capsule, embedding miniaturized MEMS force sensors, is presented. The capsule is conceived to provide automatic palpation of non-polypoid colorectal tumours during colonoscopy, since it is characterized by high degree of dysplasia, higher invasiveness and lower detection rates with respect to polyps. A first test was performed employing a silicone phantom that embedded inclusions with variable hardness and curvature. A hardness-based classification was implemented, demonstrating detection robustness to curvature variation. By comparing a set of supervised classification algorithms, a weighted 3-nearest neighbor classifier was selected. A bias force normalization model was introduced in order to make different acquisition sets consistent. Parameters of this model were chosen through a particle swarm optimization method. Additionally, an ex-vivo test was performed to assess the capsule detection performance when magnetically-driven along a colonic tissue. Lumps were identified as voltage peaks with a prominence depending on the total magnetic force applied to the capsule. Accuracy of 94 % in hardness classification was achieved, while a 100 % accuracy is obtained for the lump detection within a tolerance of 5 mm from the central path described by the capsule. In real application scenario, we foresee our device aiding physicians to detect tumorous tissues

Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features

Humans make extensive use of vision and touch as complementary senses, with vision providing global information about the scene and touch measuring local information during manipulation without suffering from occlusions. While prior work demonstrates the efficacy of tactile sensing for precise manipulation of deformables, they typically rely on supervised, human-labeled datasets. We propose Self-Supervised Visuo-Tactile Pretraining (SSVTP), a framework for learning multi-task visuo-tactile representations in a self-supervised manner through cross-modal supervision. We design a mechanism that enables a robot to autonomously collect precisely spatially-aligned visual and tactile image pairs, then train visual and tactile encoders to embed these pairs into a shared latent space using cross-modal contrastive loss. We apply this latent space to downstream perception and control of deformable garments on flat surfaces, and evaluate the flexibility of the learned representations without fine-tuning on 5 tasks: feature classification, contact localization, anomaly detection, feature search from a visual query (e.g., garment feature localization under occlusion), and edge following along cloth edges. The pretrained representations achieve a 73-100% success rate on these 5 tasks.Comment: RSS 2023, site: https://sites.google.com/berkeley.edu/ssvt

Pilot Study: Low Cost GelSight Sensor

GelSight sensor and related technology have been studied a decade to the date. It was proven that it is worth to explore in many haptics and tactile sensing applications. Elastomer, reflective coating, lighting, and camera were the main challenges of making a GelSight sensor within a short period. In this workshop paper, we present our preliminary studies on how to make a GelSight sensor using low cost material. In this study, we used a clear silicone cosmetic sponge as the elastomeric slab and that skipped the degassing process and hours of curing time in making it. Moreover, we used Psycho Paint® for the reflective coating, Light Emitting Diodes (LEDs) for the lighting, and Logitech C270 webcam for our experimental setup. Furthermore, in this study Ultraviolet (UV) ink and UV LEDs have been tested as a marker for the reflective coating and lighting respectively. UV ink markers are invisible using ordinary LED but can be made visible using UV lighting. Comparable results have been found to show the effectiveness of our setup

Low-cost GelSight with UV Markings: Feature Extraction of Objects Using AlexNet and Optical Flow without 3D Image Reconstruction

GelSight sensor has been used to study microgeometry of objects since 2009 in tactile sensing applications. Elastomer, reflective coating, lighting, and camera were the main challenges of making a GelSight sensor within a short period. The recent addition of permanent markers to the GelSight was a new era in shear/slip studies. In our previous studies, we introduced Ultraviolet (UV) ink and UV LEDs as a new form of marker and lighting respectively. UV ink markers are invisible using ordinary LED but can be made visible using UV LED. Currently, recognition of objects or surface textures using GelSight sensor is done using fusion of camera-only images and GelSight captured images with permanent markings. Those images are fed to Convolutional Neural Networks (CNN) to classify objects. However, our novel approach in using low-cost GelSight sensor with UV markings, the 3D height map to 2D image conversion, and the additional non-Gelsight captured images for training the CNN can be eliminated. AlexNet and optical flow algorithm have been used for feature recognition of five coins without UV markings and shear/slip of the coin in GelSight with UV markings respectively. Our results on confusion matrix show that, on average coin recognition can reach 93.4% without UV markings using AlexNet. Therefore, our novel method of using GelSight with UV markings would be useful to recognize full/partial object, shear/slip, and force applied to the objects without any 3D image reconstruction