Novel Configurations of Ionic Polymer-Metal Composites (IPMCs) As Sensors, Actuators, and Energy Harvesters

This dissertation starts with describing the IPMC and defining its chemical structure and fundamental characteristics in Chapter 1. The application of these materials in the form of actuator, sensor, and energy harvester are reported through a literature review in Chapter 2. The literature review involves some electromechanical modeling approaches toward physics of the IPMC as well as some of the experimental results and test reports. This chapter also includes a short description of the manufacturing process of the IPMC. Chapter 3 presents the mechanical modeling of IPMC in actuation. For modeling, shear deformation expected not to be significant. Hence, the Euler-Bernoulli beam theory considered to be the approach defining the shape and critical points of the proposed IPMC elements. Description of modeling of IPMC in sensing mode is in Chapter 4. Since the material undergoes large deformation, large beam deformation is considered for both actuation and sensing model. Basic configurations of IPMC as sensor and actuator are introduced in Chapter 5. These basic configurations, based on a systematic approach, generate a large number of possible configurations. Based on the presented mechanisms, some parameters can be defined, but the selection of a proper arrangement remained as an unknown parameter. This mater is addressed by introducing a decision-making algorithm. A series of design for slit cylindrical/tubular/helical IPMC actuators and sensors are introduced in chapter 5. A consideration related to twisting of IPMCs in helical formations is reported through some experiments. Combinations of these IPMC actuators and sensors can be made to make biomimetic robotic devices as some of them are discussed in this chapter and the following Chapters 6 and 7. Another set of IPMC actuator/sensor configurations are introduced as a loop sensor and actuator that are presented subsequently in Chapter 6. These configurations may serve as haptic and tactile feedback sensors, particularly for robotic surgery. Both of these configurations (loop and slit cylindrical) of IPMCs are discussed in details, and some experimental measurements and results are also carried out and reported. The model for different inputs is studied, and report of the feedback is presented. Various designs of these configurations of IPMC are also presented in chapter 7, including their extension to mechanical metamaterials and soft robots

An Untethered Multimodal Haptic Hand Wearable

Haptic primary colors correspond to temperature, vibration, and force. Previous studies combined these three haptic primary colors to produce different types of cutaneous sensations without the need to touch a real object. This study presents a low-cost untethered hand wearable with temperature, vibration, and force feedback. It is made from low-cost and commercial off-the-shelf components. A 26 mm annular Peltier element with a 10 mm hole is coupled to an 8 mm mini disc vibration motor, forming vibro-thermal tactile feedback for the user. All the other fingertips have an 8 mm disc vibration motor strapped on them using Velcro. Moreover, kinesthetic feedback extracted from a retractable ID badge holder with a small solenoid stopper is used as force feedback that restricts the fingers’ movement. Hand and finger tracking is done using Leap Motion Controller interfaced to a virtual setup with different geometric figures developed using Unity software. Therefore, we argue this prototype as a whole actuates cutaneous and kinesthetic feedback that would be useful in many virtual applications such as Virtual Reality (VR), teleoperated surgeries, and teleoperated farming and agriculture

Haptics Rendering and Applications

There has been significant progress in haptic technologies but the incorporation of haptics into virtual environments is still in its infancy. A wide range of the new society's human activities including communication, education, art, entertainment, commerce and science would forever change if we learned how to capture, manipulate and reproduce haptic sensory stimuli that are nearly indistinguishable from reality. For the field to move forward, many commercial and technological barriers need to be overcome. By rendering how objects feel through haptic technology, we communicate information that might reflect a desire to speak a physically- based language that has never been explored before. Due to constant improvement in haptics technology and increasing levels of research into and development of haptics-related algorithms, protocols and devices, there is a belief that haptics technology has a promising future

Fine-grained Haptics: Sensing and Actuating Haptic Primary Colours (force, vibration, and temperature)

This thesis discusses the development of a multimodal, fine-grained visual-haptic system for teleoperation and robotic applications. This system is primarily composed of two complementary components: an input device known as the HaptiTemp sensor (combines “Haptics” and “Temperature”), which is a novel thermosensitive GelSight-like sensor, and an output device, an untethered multimodal finegrained haptic glove. The HaptiTemp sensor is a visuotactile sensor that can sense haptic primary colours known as force, vibration, and temperature. It has novel switchable UV markers that can be made visible using UV LEDs. The switchable markers feature is a real novelty of the HaptiTemp because it can be used in the analysis of tactile information from gel deformation without impairing the ability to classify or recognise images. The use of switchable markers in the HaptiTemp sensor is the solution to the trade-off between marker density and capturing high-resolution images using one sensor. The HaptiTemp sensor can measure vibrations by counting the number of blobs or pulses detected per unit time using a blob detection algorithm. For the first time, temperature detection was incorporated into a GelSight-like sensor, making the HaptiTemp sensor a haptic primary colours sensor. The HaptiTemp sensor can also do rapid temperature sensing with a 643 ms response time for the 31°C to 50°C temperature range. This fast temperature response of the HaptiTemp sensor is comparable to the withdrawal reflex response in humans. This is the first time a sensor can trigger a sensory impulse that can mimic a human reflex in the robotic community. The HaptiTemp sensor can also do simultaneous temperature sensing and image classification using a machine vision camera—the OpenMV Cam H7 Plus. This capability of simultaneous sensing and image classification has not been reported or demonstrated by any tactile sensor. The HaptiTemp sensor can be used in teleoperation because it can communicate or transmit tactile analysis and image classification results using wireless communication. The HaptiTemp sensor is the closest thing to the human skin in tactile sensing, tactile pattern recognition, and rapid temperature response. In order to feel what the HaptiTemp sensor is touching from a distance, a corresponding output device, an untethered multimodal haptic hand wearable, is developed to actuate the haptic primary colours sensed by the HaptiTemp sensor. This wearable can communicate wirelessly and has fine-grained cutaneous feedback to feel the edges or surfaces of the tactile images captured by the HaptiTemp sensor. This untethered multimodal haptic hand wearable has gradient kinesthetic force feedback that can restrict finger movements based on the force estimated by the HaptiTemp sensor. A retractable string from an ID badge holder equipped with miniservos that control the stiffness of the wire is attached to each fingertip to restrict finger movements. Vibrations detected by the HaptiTemp sensor can be actuated by the tapping motion of the tactile pins or by a buzzing minivibration motor. There is also a tiny annular Peltier device, or ThermoElectric Generator (TEG), with a mini-vibration motor, forming thermo-vibro feedback in the palm area that can be activated by a ‘hot’ or ‘cold’ signal from the HaptiTemp sensor. The haptic primary colours can also be embedded in a VR environment that can be actuated by the multimodal hand wearable. A VR application was developed to demonstrate rapid tactile actuation of edges, allowing the user to feel the contours of virtual objects. Collision detection scripts were embedded to activate the corresponding actuator in the multimodal haptic hand wearable whenever the tactile matrix simulator or hand avatar in VR collides with a virtual object. The TEG also gets warm or cold depending on the virtual object the participant has touched. Tests were conducted to explore virtual objects in 2D and 3D environments using Leap Motion control and a VR headset (Oculus Quest 2). Moreover, a fine-grained cutaneous feedback was developed to feel the edges or surfaces of a tactile image, such as the tactile images captured by the HaptiTemp sensor, or actuate tactile patterns in 2D or 3D virtual objects. The prototype is like an exoskeleton glove with 16 tactile actuators (tactors) on each fingertip, 80 tactile pins in total, made from commercially available P20 Braille cells. Each tactor can be controlled individually to enable the user to feel the edges or surfaces of images, such as the high-resolution tactile images captured by the HaptiTemp sensor. This hand wearable can be used to enhance the immersive experience in a virtual reality environment. The tactors can be actuated in a tapping manner, creating a distinct form of vibration feedback as compared to the buzzing vibration produced by a mini-vibration motor. The tactile pin height can also be varied, creating a gradient of pressure on the fingertip. Finally, the integration of the high-resolution HaptiTemp sensor, and the untethered multimodal, fine-grained haptic hand wearable is presented, forming a visuotactile system for sensing and actuating haptic primary colours. Force, vibration, and temperature sensing tests with corresponding force, vibration, and temperature actuating tests have demonstrated a unified visual-haptic system. Aside from sensing and actuating haptic primary colours, touching the edges or surfaces of the tactile images captured by the HaptiTemp sensor was carried out using the fine-grained cutaneous feedback of the haptic hand wearable

A Framework for Tumor Localization in Robot-Assisted Minimally Invasive Surgery

Manual palpation of tissue is frequently used in open surgery, e.g., for localization of tumors and buried vessels and for tissue characterization. The overall objective of this work is to explore how tissue palpation can be performed in Robot-Assisted Minimally Invasive Surgery (RAMIS) using laparoscopic instruments conventionally used in RAMIS. This thesis presents a framework where a surgical tool is moved teleoperatively in a manner analogous to the repetitive pressing motion of a finger during manual palpation. We interpret the changes in parameters due to this motion such as the applied force and the resulting indentation depth to accurately determine the variation in tissue stiffness. This approach requires the sensorization of the laparoscopic tool for force sensing. In our work, we have used a da Vinci needle driver which has been sensorized in our lab at CSTAR for force sensing using Fiber Bragg Grating (FBG). A computer vision algorithm has been developed for 3D surgical tool-tip tracking using the da Vinci \u27s stereo endoscope. This enables us to measure changes in surface indentation resulting from pressing the needle driver on the tissue. The proposed palpation framework is based on the hypothesis that the indentation depth is inversely proportional to the tissue stiffness when a constant pressing force is applied. This was validated in a telemanipulated setup using the da Vinci surgical system with a phantom in which artificial tumors were embedded to represent areas of different stiffnesses. The region with high stiffness representing tumor and region with low stiffness representing healthy tissue showed an average indentation depth change of 5.19 mm and 10.09 mm respectively while maintaining a maximum force of 8N during robot-assisted palpation. These indentation depth variations were then distinguished using the k-means clustering algorithm to classify groups of low and high stiffnesses. The results were presented in a colour-coded map. The unique feature of this framework is its use of a conventional laparoscopic tool and minimal re-design of the existing da Vinci surgical setup. Additional work includes a vision-based algorithm for tracking the motion of the tissue surface such as that of the lung resulting from respiratory and cardiac motion. The extracted motion information was analyzed to characterize the lung tissue stiffness based on the lateral strain variations as the surface inflates and deflates

Visuotactile Sensors with Emphasis on GelSight Sensor: A Review

This review paper focuses on vision and touch-based sensors known as visuotactile. The study of visuotactile sensation and perception became a multidisciplinary field of study by philosophers, psychologists, biologists, engineers, technologists, and roboticists in the fields of haptics, machine vision, and artificial intelligence and it dates back centuries. To the best of our knowledge, the earliest records of visuotactile sensor was not applied to robotics and was not even for hand or finger imprint analysis yet for recording the foot pressure distribution of a walking or standing human known as pedobarograph. Our review paper presents the different literature related to visuotactile sensors that lead to a high-resolution miniature pedobarographlike sensor known as the GelSight sensor. Moreover, this review paper focuses on architecture, different techniques, hardware, and software development of GelSight sensor since 2009 with its applications in haptics, robotics, and computer vision

Integrating passive ubiquitous surfaces into human-computer interaction

Mobile technologies enable people to interact with computers ubiquitously. This dissertation investigates how ordinary, ubiquitous surfaces can be integrated into human-computer interaction to extend the interaction space beyond the edge of the display. It turns out that acoustic and tactile features generated during an interaction can be combined to identify input events, the user, and the surface. In addition, it is shown that a heterogeneous distribution of different surfaces is particularly suitable for realizing versatile interaction modalities. However, privacy concerns must be considered when selecting sensors, and context can be crucial in determining whether and what interaction to perform.Mobile Technologien ermöglichen den Menschen eine allgegenwärtige Interaktion mit Computern. Diese Dissertation untersucht, wie gewöhnliche, allgegenwärtige Oberflächen in die Mensch-Computer-Interaktion integriert werden können, um den Interaktionsraum über den Rand des Displays hinaus zu erweitern. Es stellt sich heraus, dass akustische und taktile Merkmale, die während einer Interaktion erzeugt werden, kombiniert werden können, um Eingabeereignisse, den Benutzer und die Oberfläche zu identifizieren. Darüber hinaus wird gezeigt, dass eine heterogene Verteilung verschiedener Oberflächen besonders geeignet ist, um vielfältige Interaktionsmodalitäten zu realisieren. Bei der Auswahl der Sensoren müssen jedoch Datenschutzaspekte berücksichtigt werden, und der Kontext kann entscheidend dafür sein, ob und welche Interaktion durchgeführt werden soll

Development and evaluation of hand-held robotic technology for safe and successful peripheral intravenous catheterization on pediatric patients

Peripheral IntraVenous Catheterization (PIVC) is often required in hospitals to fulfil urgent needs of blood sampling or fluid/medication administration. Despite of the importance of a high success rate, the conventional PIVC operation suffers from low insertion accuracy especially on young pediatric patients. On average, each pediatric patient is submitted to 2.1 attempts before venous access is obtained, with around 50% failure for the first attempt. The risks of such multiple attempts can be severe and life-threatening as they can cause serious extravasation injuries. Given the levels of precision and controllability needed for PIVC, robotic systems show a good potential to effectively assist the operation and improve its success rate. Therefore, this study aims to provide such robotic assistance by focusing on the most challenging and error-prone parts of the operation. In order to understand the difficulties of a pediatric PIVC, a survey investigation is conducted with specialists at the beginning of this research. The feedbacks from this survey indicates an urgent need of a hand-held robot to assist in the catheter insertion control to precisely access the target vein. To achieve the above goal, a novel venipuncture detection system based on sensing the electrical impedance of the contacting tissue at the needle tip has been proposed and developed. Then several ex-vivo and in-vivo experiments were conducted to assess this detection system. Experimental results show that this system can be highly effective to detect venipuncture. Subsequently, based on this venipuncture detection system, four different handheld robots have been developed to provide different levels of autonomy and assistance while executing a PIVC insertion: 1. SVEI, short for \u2018Smart Venous Enter Indicator\u2019, is the simplest device without actuation. The user needs to do the whole PIVC operation, and this device only provides an indication of venipuncture by lighting up an LED. 5 2. SAID, short for \u2018Semi-Autonomous Intravenous access Device\u2019, integrates a motor to control the catheter insertion. The user is required to hold the device still and target it to a vein site. He/She then activates the device. The device inserts the catheter automatically and stops it when venipuncture is detected. 3. SDOP, short for \u2018Smart hand-held Device for Over-puncture Prevention\u2019, integrates a latch-based disengage mechanism to prevent over-puncture during PIVC. The user can keep the conventional way of operation and do the insertion manually. At the moment of venipuncture, the device automatically activates the disengage mechanism to stop further advancement of the catheter. 4. CathBot represents \u2018hand-held roBot for peripheral intravenous Catheterization\u2019. The device uses a crank-slider mechanism and a solenoid actuator to convert the complicated intravenous catheterization motion to a simple linear forward motion. The user just needs to push the device\u2019s handle forwards and the device completes the whole PIVC insertion procedure automatically. All the devices were characterized to ensure they can satisfy the design specifications. Then a series of comparative experiments were conducted to assess each of them. In the first experiment, 25 na\uefve subjects were invited to perform 10 trials of PIVC on a realistic baby arm phantom. The subjects were divided into 5 groups, and each group was asked to do the PIVC with one device only (SVEI, SAID, SDOP, CathBot and regular iv catheter). The experimental results show that all devices can provide the needed assistance to significantly facilitate and improve the success rates compared to the conventional method. People who have no experience of PIVC operation before can achieve considerably high success rates in robot-assisted PIVC (86% with SVEI, 80% with SAID, 78% with SDOP and 84% with CathBot) compared to the control group (12%) who used a regular iv catheter. Also, all 5 subjects using SVEI, 3 out of 5 subjects using SAID, 2 out of 5 subjects using SDOP and 4 out of 5 subjects using CathBot were able to successfully catheterize the baby arm phantom on their first attempt, while no subjects in the control group succeeded in their first attempts. Since SVEI showed the best results, it was selected for the second round of evaluation. In the second experiment, clinicians including both PIVC experts and general clinicians were invited to perform PIVC on a realistic baby arm phantom with 3 trials using SVEI and 3 trials in the conventional way. The results demonstrate that SVEI can bring great benefits to both specialists and general clinicians. The average success rates were found to be significantly improved from 48.3% to 71.7% when SVEI was used. The experimental results reveal that all experts achieved better or equal results with SVEI compared to the conventional method, and 9 out of 12 non-experts also had better or equal performance when SVEI was used. Finally, subjective feedback acquired through post-trial questionnaires showed that all devices were highly rated in terms of usability. Overall, the results of this doctoral research support continued investment in the technology to bring the handheld robotic devices closer to clinical us