Realtime State Estimation with Tactile and Visual sensing. Application to Planar Manipulation

Accurate and robust object state estimation enables successful object manipulation. Visual sensing is widely used to estimate object poses. However, in a cluttered scene or in a tight workspace, the robot's end-effector often occludes the object from the visual sensor. The robot then loses visual feedback and must fall back on open-loop execution. In this paper, we integrate both tactile and visual input using a framework for solving the SLAM problem, incremental smoothing and mapping (iSAM), to provide a fast and flexible solution. Visual sensing provides global pose information but is noisy in general, whereas contact sensing is local, but its measurements are more accurate relative to the end-effector. By combining them, we aim to exploit their advantages and overcome their limitations. We explore the technique in the context of a pusher-slider system. We adapt iSAM's measurement cost and motion cost to the pushing scenario, and use an instrumented setup to evaluate the estimation quality with different object shapes, on different surface materials, and under different contact modes

Sensors for Robotic Hands: A Survey of State of the Art

Recent decades have seen significant progress in the field of artificial hands. Most of the surveys, which try to capture the latest developments in this field, focused on actuation and control systems of these devices. In this paper, our goal is to provide a comprehensive survey of the sensors for artificial hands. In order to present the evolution of the field, we cover five year periods starting at the turn of the millennium. At each period, we present the robot hands with a focus on their sensor systems dividing them into categories, such as prosthetics, research devices, and industrial end-effectors.We also cover the sensors developed for robot hand usage in each era. Finally, the period between 2010 and 2015 introduces the reader to the state of the art and also hints to the future directions in the sensor development for artificial hands

A Biologically Inspired Controllable Stiffness Multimodal Whisker Follicle

This thesis takes a soft robotics approach to understand the computational role of a soft whisker follicle with mechanisms to control the stiffness of the whisker. In particular, the thesis explores the role of the controllable stiffness whisker follicle to selectively favour low frequency geometric features of an object or the high frequency texture features of the object.Tactile sensing is one of the most essential and complex sensory systems for most living beings. To acquire tactile information and explore the environment, animals use various biological mechanisms and transducing techniques. Whiskers, or vibrissae are a form of mammalian hair, found on almost all mammals other than homo sapiens. For many mammals, and especially rodents, these whiskers are essential as a means of tactile sensing.The mammalian whisker follicle contains multiple sensory receptors strategically organised to capture tactile sensory stimuli of different frequencies via the vibrissal system. Nocturnal mammals such as rats heavily depend on whisker based tactile perception to find their way through burrows and identify objects. There is diversity in the whiskers in terms of the physical structure and nervous innervation. The robotics community has developed many different whisker sensors inspired by this biological basis. They take diverse mechanical, electronic, and computational approaches to use whiskers to identify the geometry, mechanical properties, and objects' texture. Some work addresses specific object identification features and others address multiple features such as texture and shape etc. Therefore, it is vital to have a comprehensive discussion of the literature and to understand the merits of bio-inspired and pure-engineered approaches to whisker-based tactile perception.The most important contribution is the design and use of a novel soft whisker follicle comprising two different frequency-dependent data capturing modules to derive more profound insights into the biological basis of tactile perception in the mammalian whisker follicle. The new insights into the biological basis of tactile perception using whiskers provide new design guidelines to develop efficient robotic whiskers

Fusion of stereo vision, force-torque, and joint sensors for estimation of in-hand object location

This paper develops a method to fuse stereo vision, force-torque sensor, and joint angle encoder measurements to estimate and track the location of a grasped object within the hand. We pose the problem as a hybrid systems estimation problem, where the continuous states are the object 6D pose, finger contact location, wrist-to-camera transform and the discrete states are the finger contact modes with the object. This paper develops the key measurement equations that govern the fusion process. Experiments with a Barrett Hand, Bumblebee 2 stereo camera, and an ATI omega force-torque sensor validate and demonstrate the method

Using tactile sensors and IMUs with an unsupervised feature-learning approach for robotic grasp stability assessment

In a world where industry has a need for ever more complex automated machines, robot grasping is still far from human capabilities. Despite recent innovations in computer vision and grasping planning, it is still a challenge for a robot to pick never-before-seen objects. Researchers are trying to combine vision with tactile sensing to augment the performance of modern intelligent machines. In this thesis, we will present a novel way to improve robotic grasping using tactile sensors and an unsupervised feature-learning algorithm. Using a test bench and sensors at the Control and Robotics (CoRo) laboratory of the ÉTS, we have developed and tested a series of classifiers to predict the outcome of a robotic grasp. Our method improves upon the results of hand-crafted feature learning. We have collected data from 100 different everyday objects, executing 10 grasping attemps per object, for a total of 1000 grasping attemps. The optimal system we developed recognized grasp failures 84.23% of the time

Soft Biomimetic Finger with Tactile Sensing and Sensory Feedback Capabilities

The compliant nature of soft fingers allows for safe and dexterous manipulation of objects by humans in an unstructured environment. A soft prosthetic finger design with tactile sensing capabilities for texture discrimination and subsequent sensory stimulation has the potential to create a more natural experience for an amputee. In this work, a pneumatically actuated soft biomimetic finger is integrated with a textile neuromorphic tactile sensor array for a texture discrimination task. The tactile sensor outputs were converted into neuromorphic spike trains, which emulate the firing pattern of biological mechanoreceptors. Spike-based features from each taxel compressed the information and were then used as inputs for the support vector machine (SVM) classifier to differentiate the textures. Our soft biomimetic finger with neuromorphic encoding was able to achieve an average overall classification accuracy of 99.57% over sixteen independent parameters when tested on thirteen standardized textured surfaces. The sixteen parameters were the combination of four angles of flexion of the soft finger and four speeds of palpation. To aid in the perception of more natural objects and their manipulation, subjects were provided with transcutaneous electrical nerve stimulation (TENS) to convey a subset of four textures with varied textural information. Three able-bodied subjects successfully distinguished two or three textures with the applied stimuli. This work paves the way for a more human-like prosthesis through a soft biomimetic finger with texture discrimination capabilities using neuromorphic techniques that provides sensory feedback; furthermore, texture feedback has the potential to enhance the user experience when interacting with their surroundings. Additionally, this work showed that an inexpensive, soft biomimetic finger combined with a flexible tactile sensor array can potentially help users perceive their environment better