A dynamic tactile sensor on photoelastic effect

Certain photoelastic materials exhibit birefringent characteristics at a very low level of strain. This property of material may be suitable for dynamic or wave propagation studies, which can be exploited for designing tactile sensors. This paper presents the design, construction and testing of a novel dynamic sensor based on photoelastic effect, which is capable of detecting object slip as well as providing normal force information. The paper investigates the mechanics of object slip, and develops an approximate model of the sensor. This allows visualization of various parameters involved in the sensor design. The model also explains design improvements necessary to obtain continuous signal during object slip. The developed sensor has been compared with other existing sensors and experimental results from the sensor have been discussed. The sensor is calibrated for normal force which is in addition to the dynamic signal that it provides from the same contact location. The sensor has a simple design and is of a small size allowing it to be incorporated into robotic fingers, and it provides output signals which are largely unaffected by external disturbances

Tracking objects with point clouds from vision and touch

We present an object-tracking framework that fuses point cloud information from an RGB-D camera with tactile information from a GelSight contact sensor. GelSight can be treated as a source of dense local geometric information, which we incorporate directly into a conventional point-cloud-based articulated object tracker based on signed-distance functions. Our implementation runs at 12 Hz using an online depth reconstruction algorithm for GelSight and a modified second-order update for the tracking algorithm. We present data from hardware experiments demonstrating that the addition of contact-based geometric information significantly improves the pose accuracy during contact, and provides robustness to occlusions of small objects by the robot's end effector

A Developmental Organization for Robot Behavior

This paper focuses on exploring how learning and development can be structured in synthetic (robot) systems. We present a developmental assembler for constructing reusable and temporally extended actions in a sequence. The discussion adopts the traditions of dynamic pattern theory in which behavior is an artifact of coupled dynamical systems with a number of controllable degrees of freedom. In our model, the events that delineate control decisions are derived from the pattern of (dis)equilibria on a working subset of sensorimotor policies. We show how this architecture can be used to accomplish sequential knowledge gathering and representation tasks and provide examples of the kind of developmental milestones that this approach has already produced in our lab

Tactile Sensors for Friction Estimation and Incipient Slip Detection - Toward Dexterous Robotic Manipulation:A Review

Humans can handle and manipulate objects with ease; however, human dexterity has yet to be matched by artificial systems. Receptors in our fingers and hands provide essential tactile information to the motor control system during dexterous manipulation such that the grip force is scaled to the tangential forces according to the coefficient of friction. Likewise, tactile sensing will become essential for robotic and prosthetic gripping performance as applications move toward unstructured environments. However, most existing research ignores the need to sense the frictional properties of the sensor-object interface, which (along with contact forces and torques) is essential for finding the minimum grip force required to securely grasp an object. Here, we review this problem by surveying the field of tactile sensing from the perspective that sensors should: 1) detect gross slip (to adjust the grip force); 2) detect incipient slip (dependent on the frictional properties of the sensor-object interface and the geometries and mechanics of the sensor and the object) as an indication of grip security; or 3) measure friction on contact with an object and/or following a gross or incipient slip event while manipulating an object. Recommendations are made to help focus future sensor design efforts toward a generalizable and practical solution to sense, and hence control grip security. Specifically, we propose that the sensor mechanics should encourage incipient slip, by allowing parts of the sensor to slip while other parts remain stuck, and that instrumentation should measure displacement and deformation to complement conventional force, pressure, and vibration tactile sensing

Objekt-Manipulation und Steuerung der Greifkraft durch Verwendung von Taktilen Sensoren

This dissertation describes a new type of tactile sensor and an improved version of the dynamic tactile sensing approach that can provide a regularly updated and accurate estimate of minimum applied forces for use in the control of gripper manipulation. The pre-slip sensing algorithm is proposed and implemented into two-finger robot gripper. An algorithm that can discriminate between types of contact surface and recognize objects at the contact stage is also proposed. A technique for recognizing objects using tactile sensor arrays, and a method based on the quadric surface parameter for classifying grasped objects is described. Tactile arrays can recognize surface types on contact, making it possible for a tactile system to recognize translation, rotation, and scaling of an object independently.Diese Dissertation beschreibt eine neue Art von taktilen Sensoren und einen verbesserten Ansatz zur dynamischen Erfassung von taktilen daten, der in regelmäßigen Zeitabständen eine genaue Bewertung der minimalen Greifkraft liefert, die zur Steuerung des Greifers nötig ist. Ein Berechnungsverfahren zur Voraussage des Schlupfs, das in einen Zwei-Finger-Greifarm eines Roboters eingebaut wurde, wird vorgestellt. Auch ein Algorithmus zur Unterscheidung von verschiedenen Oberflächenarten und zur Erkennung von Objektformen bei der Berührung wird vorgestellt. Ein Verfahren zur Objekterkennung mit Hilfe einer Matrix aus taktilen Sensoren und eine Methode zur Klassifikation ergriffener Objekte, basierend auf den Daten einer rechteckigen Oberfläche, werden beschrieben. Mit Hilfe dieser Matrix können unter schiedliche Arten von Oberflächen bei Berührung erkannt werden, was es für das Tastsystem möglich macht, Verschiebung, Drehung und Größe eines Objektes unabhängig voneinander zu erkennen

Sensing and Control for Robust Grasping with Simple Hardware

Robots can move, see, and navigate in the real world outside carefully structured factories, but they cannot yet grasp and manipulate objects without human intervention. Two key barriers are the complexity of current approaches, which require complicated hardware or precise perception to function effectively, and the challenge of understanding system performance in a tractable manner given the wide range of factors that impact successful grasping. This thesis presents sensors and simple control algorithms that relax the requirements on robot hardware, and a framework to understand the capabilities and limitations of grasping systems.Engineering and Applied Science

Sensory Communication

Contains table of contents for Section 2, an introduction and reports on twelve research projects.National Institutes of Health Grant 5 R01 DC00117National Institutes of Health Contract 2 P01 DC00361National Institutes of Health Grant 5 R01 DC00126National Institutes of Health Grant R01-DC00270U.S. Air Force - Office of Scientific Research Contract AFOSR-90-0200National Institutes of Health Grant R29-DC00625U.S. Navy - Office of Naval Research Grant N00014-88-K-0604U.S. Navy - Office of Naval Research Grant N00014-91-J-1454U.S. Navy - Office of Naval Research Grant N00014-92-J-1814U.S. Navy - Naval Training Systems Center Contract N61339-93-M-1213U.S. Navy - Naval Training Systems Center Contract N61339-93-C-0055U.S. Navy - Naval Training Systems Center Contract N61339-93-C-0083U.S. Navy - Office of Naval Research Grant N00014-92-J-4005U.S. Navy - Office of Naval Research Grant N00014-93-1-119

In-Material Processing of High Bandwidth Sensor Measurements Using Modular Neural Networks

Robotic materials are a novel class of materials that tightly integrate sensing, computing, and actuation into an engineered material or composite to allow the behavior of the material to be defined algorithmically. Robotic materials are constructed using an embedded network of computing nodes based on small, inexpensive microcontrollers. Examples of such materials include morphable airfoils which change shape in response to flight conditions or mission parameters, robotic skins with rich tactile sensing capabilities that recognize texture or touch gestures, clothing with tightly integrated sensing to assist with or augment the wearer's perception of the environment, or materials with dynamic camouflage capabilities.In this thesis, I develop a framework for in-material processing which tightly couples modularized deep neural networks and high-bandwidth sensors using a network of embedded, material-scale components. This framework enables materials to learn multiple desired responses to stimuli, avoiding the need for accurate modeling of the dynamics of the material and stimuli. I utilize a modular neural network design consisting of convolutional (CNN) and long short-term memory (LSTM) layers implemented in each node in the material as a computational approach for robotic materials. This network architecture allows for nodes in the material to process local sensor values, maintain local state information, and communicate with nodes in a local neighborhood in the materials. A multiobjective optimization approach is employed to automatically design the neural network architectures which maximizes the performance of the network while ensuring hardware budgets, such as memory requirements, are maintained. A communication network design is also developed to allow network modules to learn a communication protocol that limits communication to a desired rate, ensuring in-network bandwidth constraints are maintained.I demonstrate the suitability of this computational model for robotic materials using examples in several domains. An RF-based e-textile gesture input device capable of distinguishing between user control gestures is used to control arbitrary external devices. A tire with embedded piezoelectric sensing capabilites for use in high-performance autonomous vehicles performs state-of-the-art identification of terrains driven on. Two robotic skins are presented---one which is capable of detecting and localizing contact, and identifying the texture of the contacting objects; and a second which assists with avoiding collisions with obstacles and identifies affective touch gestures performed by a human collaborator. Finally, a distributed approach to human activity recognition is presented whose activity identification performance is comparable to a centralized approach, but can be implemented on hardware designed for wearable applications, as opposed to a GPU-enabled device. The examples shown demonstrate that robotic materials can perform significant in-material processing; are loosely coupled from a host system, communicating a minimal number of low-bandwidth events to the host; and can exhibit multifunctional behavior that is analyzed for safety or performance considerations

Development of highly sensitive multimodal tactile sensor

The sense of touch is crucial for interpreting exteroceptive stimuli, and for moderating physical interactions with one’s environment during object grasping and manipulation tasks. For years, tactile researchers have sought a method that will allow robots to achieve the same tactile sensing capabilities as humans, but the solution has remained elusive. This is a problem for people in the medical and robotics communities, as prosthetic and robotic limbs provide little or no force feedback during contact with objects. During object manipulation tasks, the inability to control the force (applied by the prosthetic or robotic hand to the object) frequently results in damage to the object. Moreover, amputees must compensate for the lack of tactility by paying continuous visual attention to the task at hand, making even the simplest task a frustrating and time-consuming endeavor. We believe that these challenges of object manipulation might best be addressed by a closed feedback loop with a tactile sensory system that is capable of detecting multiple stimuli. To this end, the goal of our research is the development of a tactile sensor that mimics the human sensory apparatus as closely as possible. Thus far, tactile sensors have been unable to match the human sensory apparatus in terms of simultaneous multimodality, high resolution, and broad sensitivity. In particular, previous sensors have typically been able to sense either a wide range of forces, or very low forces, but never both at the same time; and they are designed for either static or dynamic sensing, rather than multimodality. These restrictions have left them unsuited to the needs of robotic applications. Capacitance-based sensors represent the most promising approach, but they too must overcome many limitations. Although recent innovations in the touch screen industry have resolved the issue of processing complexity, through the replacement of clunky processing circuits with new integrated circuits (ICs), most capacitive sensors still remain limited by hysteresis and narrow ranges of sensitivity, due to the properties of their dielectrics. In this thesis, we present the design of a new capacitive tactile sensor that is capable of making highly accurate measurements at low force levels, while also being sensitive to a wide range of forces. Our sensor is not limited to the detection of either low forces or broad sensitivity, because the improved soft dielectric that we have constructed allows it to do both at the same time. To construct the base of the dielectric, we used a geometrically modified silicone material. To create this material, we used a soft-lithography process to construct microfeatures that enhance the silicone’s compressibility under pressure. Moreover, the silicone was doped with high-permittivity ceramic nanoparticles, thereby enhancing the capacitive response of the sensor. Our dielectric features a two-stage microstructure, which makes it very sensitive to low forces, while still able to measure a wide range of forces. Despite these steps, and the complexity of the dielectric’s structure, we were still able to fabricate the dielectric using a relatively simple process. In addition, our sensor is not limited to either static or dynamic sensing; unlike previous sensors, it is capable of doing both simultaneously. This multimodality allows our sensor to detect fluctuating forces, even at very low force levels. Whereas past researchers have used separate technologies for static and dynamic sensing, our dynamic sensing unit is formed with same capacitive technology as the static one. This was possible because of the high sensitivity of our dielectric. We used the entire surface area effectively, by integrating the single dynamic sensing taxel on the same layer as the static sensing taxels. Essentially, the dynamic taxel takes the shape of the lines of a grid, filling in the spaces between the individual static taxels. For further optimization, the geometry of the dynamic taxel has been redesigned by fringing miniature traces of the dynamic taxel within the static taxels. In this way, the entire surface of the sensor is sensitive to both dynamic and static events. While this design slightly reduces the area that is covered by the static taxels, the trade-off is justified, as the capacitive behavior is boosted by the edge effect of the capacitor. The fusion of an innovative dielectric with a capacitive sensing IC has produced a highly sensitive tactile sensor that meets our goals regarding resolution, noise immunity, and overall performance. It is sensitive to forces ranging from 1 mN to 15 N. We verified the functionality of our sensor by mounting it on several of the most popular mechanical hands. Our grasp assessment experiments delivered promising results, and showed how our sensor might be further refined so that it can be used to accurately estimate the outcome of an attempted grasp. In future, we believe that combining an advanced robotic hand with the sensor we have developed will allow us to meet the demand for human-like tactile sensing abilities

Oscillatory signatures of unimodal, bimodal, and cross-modal sensory working memory

Neural oscillatory activity is an essential brain mechanism that enables and subserves a vast range of cognitive functions. Studying them non-invasively through electroencephalography (EEG) has proven to be an effective method of discovering associations between oscillations in different frequency bands and various cognitive functions. Studying the oscillatory dynamics of human working memory (WM) \u2013 a core component of human higher cognitive functioning \u2013 has been particularly fruitful, leading to insights about the mental processes, frequency bands, and brain areas involved. In addition to frequency band specificity, the application of source reconstruction methods has led to further insights by revealing specific brain areas associated with WM related processing. In the present study, we focused on the oscillatory power dynamics during sensory working memory (SWM) in auditory and tactile modalities in the alpha band. In a delayed comparison two alternative forced choice task participants received two seque ntial stimuli and had to respond whether the intensity of the second stimulus was stronger than that of the first stimulus. In three related EEG experiments we examined SWM processing under unimodal (stimulation in one modality), bimodal (stimulation in both modalities simultaneously), and cross -modal (sequential stimulation of the modalities) conditions. An additional non -WM control condition allowed us to explore not only the differences between auditory and tactile WM, but also the effects of the WM task itself on the delay period oscillatory activity within each sensory modality. Our results showed that, while the bimodal stimulation condition led to behavioral enhancement, an increased stimulus difference was necessary to maintain the same level of performance also in the cross-modal conditions. Localizing the oscillatory activity in the alpha band (8 \u2013 12Hz) revealed a clear disinhibitory effect over the somatosensory cortex during the early and the late delay period, while the mid-delay did not show any differences in SWM between the two modalities. A similar, albeit weaker , effect was observed over the auditory cortices. A right parietal reduction of alpha power emerged during the late delay when a tactile stimulus had to be compared cross-modally. This suggests the involvement of parietal somatosensory association cortex in the cross-modal transformation of the tactile stimulus. Lastly, the differences between cortical source distributions when contrasting unimodal and cross-modal conditions demonstrated that late delay effects do not reflect only anticipatory effects due to the upcoming modality, but also reflect the influence of the stimulus modality kept in WM. Contrasting the bimodal condition with the unimodal ones revealed a parametric beta band effect in a right parietal area during the early delay only in the bimodal condition, which suggests that beta oscillations might play a role in multimodal integration under SWM conditions. A second effect during the early delay period was observed in the theta (4 \u2013 7Hz) band. An early effect appeared when contrasting conditions in which the first stimuli were identical while the second stimuli differed across the conditions. This result suggests that the early delay period is already shaped by the anticipated comparison context. The clearest differences in the contrast between WM and non-WM task were observed in theta and gamma bands. Source localizing the condition differences suggested the involvement of hippocampal and fronto-central areas in carrying out the WM task. Furthermore, sensory cortices of the respective modality conditions showed the highest levels of connectivity with the rest of the brain during the late delay, further highlighting the involvement of gamma band oscillations in SWM related processing. Overall, this study demonstrates that the results obtained when studying SWM related processing strongly depend on the sensory modality examined and the type of WM task employed. Any observations with regard to SWM related oscillatory power dynamics should be explored in multiple contexts before drawing any generalized conclusions