Inferring Object Properties from Incidental Contact with a Tactile-Sensing Forearm

arXiv:1409.4972v1 [cs.RO]Whole-arm tactile sensing enables a robot to sense properties of contact across its entire arm. By using this large sensing area, a robot has the potential to acquire useful information from incidental contact that occurs while performing a task. Within this paper, we demonstrate that data-driven methods can be used to infer mechanical properties of objects from incidental contact with a robot’s forearm. We collected data from a tactile-sensing forearm as it made contact with various objects during a simple reaching motion. We then used hidden Markov models (HMMs) to infer two object properties (rigid vs. soft and fixed vs. movable) based on low-dimensional features of time-varying tactile sensor data (maximum force, contact area, and contact motion). A key issue is the extent to which data-driven methods can generalize to robot actions that differ from those used during training. To investigate this issue, we developed an idealized mechanical model of a robot with a compliant joint making contact with an object. This model provides intuition for the classification problem. We also conducted tests in which we varied the robot arm’s velocity and joint stiffness. We found that, in contrast to our previous methods [1], multivariate HMMs achieved high cross-validation accuracy and successfully generalized what they had learned to new robot motions with distinct velocities and joint stiffnesses

Model-free vision-based shaping of deformable plastic materials

We address the problem of shaping deformable plastic materials using non-prehensile actions. Shaping plastic objects is challenging, since they are difficult to model and to track visually. We study this problem, by using kinetic sand, a plastic toy material which mimics the physical properties of wet sand. Inspired by a pilot study where humans shape kinetic sand, we define two types of actions: \textit{pushing} the material from the sides and \textit{tapping} from above. The chosen actions are executed with a robotic arm using image-based visual servoing. From the current and desired view of the material, we define states based on visual features such as the outer contour shape and the pixel luminosity values. These are mapped to actions, which are repeated iteratively to reduce the image error until convergence is reached. For pushing, we propose three methods for mapping the visual state to an action. These include heuristic methods and a neural network, trained from human actions. We show that it is possible to obtain simple shapes with the kinetic sand, without explicitly modeling the material. Our approach is limited in the types of shapes it can achieve. A richer set of action types and multi-step reasoning is needed to achieve more sophisticated shapes.Comment: Accepted to The International Journal of Robotics Research (IJRR

Realistic tool-tissue interaction models for surgical simulation and planning

Surgical simulators present a safe and potentially effective method for surgical training, and can also be used in pre- and intra-operative surgical planning. Realistic modeling of medical interventions involving tool-tissue interactions has been considered to be a key requirement in the development of high-fidelity simulators and planners. The soft-tissue constitutive laws, organ geometry and boundary conditions imposed by the connective tissues surrounding the organ, and the shape of the surgical tool interacting with the organ are some of the factors that govern the accuracy of medical intervention planning.\ud \ud This thesis is divided into three parts. First, we compare the accuracy of linear and nonlinear constitutive laws for tissue. An important consequence of nonlinear models is the Poynting effect, in which shearing of tissue results in normal force; this effect is not seen in a linear elastic model. The magnitude of the normal force for myocardial tissue is shown to be larger than the human contact force discrimination threshold. Further, in order to investigate and quantify the role of the Poynting effect on material discrimination, we perform a multidimensional scaling study. Second, we consider the effects of organ geometry and boundary constraints in needle path planning. Using medical images and tissue mechanical properties, we develop a model of the prostate and surrounding organs. We show that, for needle procedures such as biopsy or brachytherapy, organ geometry and boundary constraints have more impact on target motion than tissue material parameters. Finally, we investigate the effects surgical tool shape on the accuracy of medical intervention planning. We consider the specific case of robotic needle steering, in which asymmetry of a bevel-tip needle results in the needle naturally bending when it is inserted into soft tissue. We present an analytical and finite element (FE) model for the loads developed at the bevel tip during needle-tissue interaction. The analytical model explains trends observed in the experiments. We incorporated physical parameters (rupture toughness and nonlinear material elasticity) into the FE model that included both contact and cohesive zone models to simulate tissue cleavage. The model shows that the tip forces are sensitive to the rupture toughness. In order to model the mechanics of deflection of the needle, we use an energy-based formulation that incorporates tissue-specific parameters such as rupture toughness, nonlinear material elasticity, and interaction stiffness, and needle geometric and material properties. Simulation results follow similar trends (deflection and radius of curvature) to those observed in macroscopic experimental studies of a robot-driven needle interacting with gels

Design and development of new tactile softness displays for minimally invasive surgery

Despite an influential shortcoming of minimally invasive sugary (MIS), which is the lack of tactile feedback to the surgeon, MIS has increasingly been used in various types of surgeries. Restoring the missing tactile feedback, especially information which can be obtained by the palpation of tissue, such as detection of embedded lump and softness characterization is important in MIS. The present study aims to develop tactile feedback systems both graphically and physically. In graphical rendering approach, the proposed system receives signals from the previously fabricated piezoelectric softness sensors which are integrated with an MIS grasper. After processing the signals, the tactile information is displayed by means of a color coding method. Using the graphical images, the softness of the grasped objects can visually be differentiated. A physical tactile display system is also designed and fabricated. This system simulates non-linear material properties of different soft objects. The system consists of a linear actuator, force and position sensors and processing software. A PID controller is used to control the motion of a linear actuator according to the properties of the simulated material and applied force. Graphical method was also examined to render the tactile information of embedded lumps within a soft tissue/object. The necessary information on the size and location of the hidden features are collected using sensorized MIS graspers. The information is then processed and graphically rendered to the surgeon. Using the proposed system surgeons can identify presence, location and approximate size of hidden lumps by grasping the target object with a reasonable accuracy. Finally, in order to determine the softness of the grasped object, another novel approach is taken by the design and fabrication of a smart endoscopic tool equipped with sensors for measuring the applied force and the angle of the grasper jaws. Using this method, the softness/compliance of the grasped object can be estimated and presented to the surgeo

Identification of Mechanical Properties of Nonlinear Materials and Development of Tactile Displays for Robotic Assisted Surgery Applications

This PhD work presents novel methods of mechanical property identification for soft nonlinear materials and methods of recreating and modeling the deformation behavior of these nonlinear materials for tactile feedback systems. For the material property identification, inverse modeling method is employed for the identification of hyperelastic and hyper-viscoelastic (HV) materials by use of the spherical indentation test. Identification experiments are performed on soft foam materials and fresh harvested bovine liver tissue. It is shown that reliability and accuracy of the identified material parameters are directly related to size of the indenter and depth of the indentation. Results show that inverse FE modeling based on MultiStart optimization algorithm and the spherical indentation, is a reliable and scalable method of identification for biological tissues based on HV constitutive models. The inverse modeling method based on the spherical indentation is adopted for realtime applications using variation and Kalman filter methods. Both the methods are evaluated on hyperelastic foams and biological tissues on experiments which are analogous to the robot assisted surgery. Results of the experiments are compared and discussed for the proposed methods. It is shown that increasing the indentation rate eliminates time dependency in material behavior, thus increases the successful recognition rate. The deviation of an identified parameter at indentation rates of V=1, 2 and 4 mm/s was found as 28%, 21.3% and 7.3%. It is found that although the Kalman filter method yields less dispersion in identified parameters compared to the variance method, it requires almost 900 times more computation power compared to the variance method, which is a limiting factor for increasing the indentation rate. Three bounding methods are proposed and implemented for the Kalman filter estimation. It was found that the Projection and Penalty bounding methods yield relatively accurate results without failure. However, the Nearest Neighbor method found with a high chance of non-convergence. The second part of the thesis is focused on the development of tactile displays for modeling the mechanical behavior of the nonlinear materials for human tactile perception. An accurate finite element (FE) model of human finger pad is constructed and validated in experiments of finger pad contact with soft and relatively rigid materials. Hyperfoam material parameters of the identified elastomers from the previous section are used for validation of the finger pad model. A magneto-rheological fluid (MRF) based tactile display is proposed and its magnetic FE model is constructed and validated in Gauss meter measurements. FE models of the human finger pad and the proposed tactile display are used in a model based control algorithm for the proposed display. FE models of the identified elastomers are used for calculation of control curves for these elastomers. An experiment is set up for evaluation of the proposed display. Experiments are performed on biological tissue and soft nonlinear foams. Comparison between curves of desired and recreated reaction force from subject's finger pad contact with the display showed above 84% accuracy. As a complementary work, new modeling and controlling approaches are proposed and tested for tactile displays based on linear actuators. Hertzian model of contact between the human finger pad and actuator cap is derived and curves of material deformation are obtained and improved based on this model. A PID controller is designed for controlling the linear actuators. Optimization based controller tuning approach is explained in detail and robust stability of the system is also investigated. Results showed maximum tracking error of 16.6% for the actuator controlled by the PID controller. Human subject tests of recreated softness perception show 100% successful recognition rate for group of materials with high difference in their softness

Analysis of Product Architectures of Pin Array Technologies for Tactile Displays

Refreshable tactile displays based on pin array technologies have a significant impact on the education of children with visual impairments, but they are prohibitively expensive. To better understand their design and the reason for the high cost, we created a database and analyzed the product architectures of 67 unique pin array technologies from literature and patents. We qualitatively coded their functional elements and analyzed the physical parts that execute the functions. Our findings highlight that pin array surfaces aim to achieve three key functions, i.e., raise and lower pins, lock pins, and create a large array. We also contribute a concise morphological chart that organises the various mechanisms for these three functions. Based on this, we discuss the reasons for the high cost and complexity of these surface haptic technologies and infer why larger displays and more affordable devices are not available. Our findings can be used to design new mechanisms for more affordable and scalable pin array display systems

Advanced Mobile Robotics: Volume 3

Mobile robotics is a challenging field with great potential. It covers disciplines including electrical engineering, mechanical engineering, computer science, cognitive science, and social science. It is essential to the design of automated robots, in combination with artificial intelligence, vision, and sensor technologies. Mobile robots are widely used for surveillance, guidance, transportation and entertainment tasks, as well as medical applications. This Special Issue intends to concentrate on recent developments concerning mobile robots and the research surrounding them to enhance studies on the fundamental problems observed in the robots. Various multidisciplinary approaches and integrative contributions including navigation, learning and adaptation, networked system, biologically inspired robots and cognitive methods are welcome contributions to this Special Issue, both from a research and an application perspective

Human-Machine Interfaces using Distributed Sensing and Stimulation Systems

As the technology moves towards more natural human-machine interfaces (e.g. bionic limbs, teleoperation, virtual reality), it is necessary to develop a sensory feedback system in order to foster embodiment and achieve better immersion in the control system. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing a wide bandwidth of information. To provide this type of feedback, it is necessary to develop a distributed sensing system that could extract a wide range of information during the interaction between the robot and the environment. In addition, a distributed feedback interface is needed to deliver such information to the user. This thesis proposes the development of a distributed sensing system (e-skin) to acquire tactile sensation, a first integration of distributed sensing system on a robotic hand, the development of a sensory feedback system that compromises the distributed sensing system and a distributed stimulation system, and finally the implementation of deep learning methods for the classification of tactile data. It\u2019s core focus addresses the development and testing of a sensory feedback system, based on the latest distributed sensing and stimulation techniques. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives, and the used methodology and contributions; as well as six studies that tackled the development of human-machine interfaces