A Haptic Surface Robot Interface for Large-Format Touchscreen Displays

This thesis presents the design for a novel haptic interface for large-format touchscreens. Techniques such as electrovibration, ultrasonic vibration, and external braked devices have been developed by other researchers to deliver haptic feedback to touchscreen users. However, these methods do not address the need for spatial constraints that only restrict user motion in the direction of the constraint. This technology gap contributes to the lack of haptic technology available for touchscreen-based upper-limb rehabilitation, despite the prevalent use of haptics in other forms of robotic rehabilitation. The goal of this thesis is to display kinesthetic haptic constraints to the touchscreen user in the form of boundaries and paths, which assist or challenge the user in interacting with the touchscreen. The presented prototype accomplishes this by steering a single wheel in contact with the display while remaining driven by the user. It employs a novel embedded force sensor, which it uses to measure the interaction force between the user and the touchscreen. The haptic response of the device is controlled using this force data to characterize user intent. The prototype can operate in a simulated free mode as well as simulate rigid and compliant obstacles and path constraints. A data architecture has been created to allow the prototype to be used as a peripheral add-on device which reacts to haptic environments created and modified on the touchscreen. The long-term goal of this work is to create a haptic system that enables a touchscreen-based rehabilitation platform for people with upper limb impairments

遠隔操作での視覚的力触覚の脳波評価とロボット実装に関する研究

関西大

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Distributed Sensing and Stimulation Systems Towards Sense of Touch Restoration in Prosthetics

Modern prostheses aim at restoring the functional and aesthetic characteristics of the lost limb. To foster prosthesis embodiment and functionality, it is necessary to restitute both volitional control and sensory feedback. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing high-fidelity spatial information. To provide this type of feedback in prosthetics, it is necessary to sense tactile information from artificial skin placed on the prosthesis and transmit tactile feedback above the amputation in order to map the interaction between the prosthesis and the environment. This thesis proposes the integration of distributed sensing systems (e-skin) to acquire tactile sensation, and non-invasive multichannel electrotactile feedback and virtual reality to deliver high-bandwidth information to the user. Its core focus addresses the development and testing of close-loop sensory feedback human-machine interface, based on the latest distributed sensing and stimulation techniques for restoring the sense of touch in prosthetics. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives and the used methodology and contributions; as well as three studies distributed over stimulation system level and sensing system level. The first study presents the development of close-loop compensatory tracking system to evaluate the usability and effectiveness of electrotactile sensory feedback in enabling real-time close-loop control in prosthetics. It examines and compares the subject\u2019s adaptive performance and tolerance to random latencies while performing the dynamic control task (i.e. position control) and simultaneously receiving either visual feedback or electrotactile feedback for communicating the momentary tracking error. Moreover, it reported the minimum time delay needed for an abrupt impairment of users\u2019 performance. The experimental results have shown that electrotactile feedback performance is less prone to changes with longer delays. However, visual feedback drops faster than electrotactile with increased time delays. This is a good indication for the effectiveness of electrotactile feedback in enabling close- loop control in prosthetics, since some delays are inevitable. The second study describes the development of a novel non-invasive compact multichannel interface for electrotactile feedback, containing 24 pads electrode matrix, with fully programmable stimulation unit, that investigates the ability of able-bodied human subjects to localize the electrotactile stimulus delivered through the electrode matrix. Furthermore, it designed a novel dual parameter -modulation (interleaved frequency and intensity) and compared it to conventional stimulation (same frequency for all pads). In addition and for the first time, it compared the electrotactile stimulation to mechanical stimulation. More, it exposes the integration of virtual prosthesis with the developed system in order to achieve better user experience and object manipulation through mapping the acquired real-time collected tactile data and feedback it simultaneously to the user. The experimental results demonstrated that the proposed interleaved coding substantially improved the spatial localization compared to same-frequency stimulation. Furthermore, it showed that same-frequency stimulation was equivalent to mechanical stimulation, whereas the performance with dual-parameter modulation was significantly better. The third study presents the realization of a novel, flexible, screen- printed e-skin based on P(VDF-TrFE) piezoelectric polymers, that would cover the fingertips and the palm of the prosthetic hand (particularly the Michelangelo hand by Ottobock) and an assistive sensorized glove for stroke patients. Moreover, it developed a new validation methodology to examine the sensors behavior while being solicited. The characterization results showed compatibility between the expected (modeled) behavior of the electrical response of each sensor to measured mechanical (normal) force at the skin surface, which in turn proved the combination of both fabrication and assembly processes was successful. This paves the way to define a practical, simplified and reproducible characterization protocol for e-skin patches In conclusion, by adopting innovative methodologies in sensing and stimulation systems, this thesis advances the overall development of close-loop sensory feedback human-machine interface used for restoration of sense of touch in prosthetics. Moreover, this research could lead to high-bandwidth high-fidelity transmission of tactile information for modern dexterous prostheses that could ameliorate the end user experience and facilitate it acceptance in the daily life

The Role of Fluorescence and Human Factors in Quantitative Transdermal Blood and Tissue Analysis Using NIR Raman Spectroscopy

This research is part of an ongoing project aimed at the application of combined near infrared (NIR) Raman and fluorescence spectroscopy to noninvasive in vivo blood analysis including but not limited to glucose monitoring. Coping with practicalities of human factors and exploring ways to obtain and use knowledge gained about autofluorescence to improve algorithms for blood and tissue analysis are the general goals of this research. Firstly, the study investigated the various sources of human factors pertinent to our concerns, such as fingerprints, turgor, skin hydration and pigmentation. We then introduced specialized in vivo apparatus including means for precise and reproducible placement of the tissues relative to the optical aperture, i.e., the position detector pressure monitor (PDPM). Based on solid instrumental performances, appropriate methodology is now provided for applying and maintaining pressure to keep surface tissues immobile during experiments while obtaining the desired blood content and flow. Secondly, in vivo human fingertip skin autofluorescence photobleaching under 200 mW 830 nm NIR irradiation is observed and it is characterized that: i) the majority of the photobleached fluorescence originates from static tissue not blood, ii) the bleaching (1/e point) occurs in 101-102 sec timescale, and also iii) a photobleached region remains bleached for at least 45 min but recovers completely within several hours. A corresponding extensive but not exhaustive in vitro systematic study narrowed down the major contributors of such fluorescence and bleaching to collagen, melanin, plasma and hemoglobin: two major static tissue constituents and two major blood proteins. Thirdly, we established that measuring the inelastic and elastic emissions simultaneously leads to a sensitive probe for volume changes of both red blood cells and plasma. An algorithm based on measurements obtained while performing research needed for this thesis, as well as some empirical calibration approaches, was presented. The calibrated algorithm showed real potential to track hematocrit variations in cardiac pulses, centrifugal loading, blood vessel blockage using tourniquet, and even during as subtle an occurrence as in a Valsalva maneuver. Finally, NIR fluorescence and photochemistry of pentosidine, a representative of the advanced glycation endproducts (AGEs) which accumulate with age and hyperglycemia, was studied. The results indicate that oxygen plays a pivotal role in its photobleaching process. We hypothesized and offered proofs showing that pentosidine is a 1O2 sensitizer that is also subject to attack by the 1O2 resulting in the photobleaching that is observed when probing tissue using NIR. The photobleaching reaction is kinetically first order in pentosidine and ground state oxygen, and in vivo effectively first order with NIR irradiation also

Haptics Rendering and Applications

There has been significant progress in haptic technologies but the incorporation of haptics into virtual environments is still in its infancy. A wide range of the new society's human activities including communication, education, art, entertainment, commerce and science would forever change if we learned how to capture, manipulate and reproduce haptic sensory stimuli that are nearly indistinguishable from reality. For the field to move forward, many commercial and technological barriers need to be overcome. By rendering how objects feel through haptic technology, we communicate information that might reflect a desire to speak a physically- based language that has never been explored before. Due to constant improvement in haptics technology and increasing levels of research into and development of haptics-related algorithms, protocols and devices, there is a belief that haptics technology has a promising future

Biomechanics

Biomechanics is a vast discipline within the field of Biomedical Engineering. It explores the underlying mechanics of how biological and physiological systems move. It encompasses important clinical applications to address questions related to medicine using engineering mechanics principles. Biomechanics includes interdisciplinary concepts from engineers, physicians, therapists, biologists, physicists, and mathematicians. Through their collaborative efforts, biomechanics research is ever changing and expanding, explaining new mechanisms and principles for dynamic human systems. Biomechanics is used to describe how the human body moves, walks, and breathes, in addition to how it responds to injury and rehabilitation. Advanced biomechanical modeling methods, such as inverse dynamics, finite element analysis, and musculoskeletal modeling are used to simulate and investigate human situations in regard to movement and injury. Biomechanical technologies are progressing to answer contemporary medical questions. The future of biomechanics is dependent on interdisciplinary research efforts and the education of tomorrow’s scientists

Enabling wearable soft tactile displays with dielectric elastomer actuators

PhDTouch is one of the less exploited sensory channels in human machine interactions. While the introduction of the tactile feedback would improve the user experience in several fields, such as training for medical operators, teleoperation, computer aided design and 3D model exploration, no interfaces able to mimic accurately and realistically the tactile feeling produced by the contact with a real soft object are currently available. Devices able to simulate the contact with soft bodies, such as the human organs, might improve the experience. The existing commercially available tactile displays consist of complex mechanisms that limit their portability. Moreover, no devices are able to provide tactile stimuli via a soft interface that can also modulate the contact area with the finger pad, which is required to realistically mimic the contact with soft bodies, as needed for example in systems aimed at simulating interactions with virtual biological tissues or in robot-assisted minimally invasive surgery. The aim of this thesis is to develop such a wearable tactile display based on the dielectric elastomer actuators (DEAs). DEAs are a class of materials that respond to an electric field producing a deformation. In particular, in this thesis, the tactile element consists of a so-called hydrostatically coupled dielectric elastomer actuator (HC-DEAs). HC-DEAs rely on an incompressible fluid that hydrostatically couples a DEA-based active part to a passive part interfaced to the user. The display was also tested within a closed-loop configuration consisting of a hand tracking system and a custom made virtual environment. This proof of concept system allowed for a validation of the abilities of the display. Mechanical and psychophysical tests were performed in order to assess the ability of the system to provide tactile stimuli that can be distinguished by the users. Also, the miniaturisation of the HC-DEA was investigated for applications in refreshable Braille displays or arrays of tactile elements for tactile maps

Human-Robot Collaborations in Industrial Automation

Technology is changing the manufacturing world. For example, sensors are being used to track inventories from the manufacturing floor up to a retail shelf or a customer’s door. These types of interconnected systems have been called the fourth industrial revolution, also known as Industry 4.0, and are projected to lower manufacturing costs. As industry moves toward these integrated technologies and lower costs, engineers will need to connect these systems via the Internet of Things (IoT). These engineers will also need to design how these connected systems interact with humans. The focus of this Special Issue is the smart sensors used in these human–robot collaborations

Towards observable haptics: Novel sensors for capturing tactile interaction patterns

Kõiva R. Towards observable haptics: Novel sensors for capturing tactile interaction patterns. Bielefeld: Bielefeld University; 2014.Touch is one of the primary senses humans use when performing coordinated interaction, but the lack of a sense of touch in the majority of contemporary interactive technical systems, such as robots, which operate in non-deterministic environments, results in interactions that can at best be described as clumsy. Observing human haptics and extracting the salient information from the gathered data is not only relevant if we are to try to understand the involved underlying cognitive processes, but should also provide us with significant clues to design future intelligent interactive systems. Such systems could one day help to take the burden of tedious tasks off our hands in a similar fashion to how industrial robots revolutionized manufacturing. The aim of the work in this thesis was to provide significant advancements in tactile sensing technology, and thus move us a step closer to realizing this goal. The contributions contained herein can be broken into two major parts. The first part investigates capturing interaction patterns in humans with the goals of better understanding manual intelligence and improving the lives of hand amputees, while the second part is focused on augmenting technical systems with a sense of touch. tacTiles, a wireless tactile sensitive surface element attached to a deformable textile, was developed to capture human full-body interactions with large surfaces we come into contact with in our daily lives, such as floors, chairs, sofas or other furniture. The Tactile Dataglove, iObject and the Tactile Pen were developed especially to observe human manual intelligence. Whereas iObject allows motion sensing and a higher definition tactile signal to be captured than the Tactile Dataglove (220 tactile cells in the first iObject prototype versus 54 cells in the glove), the wearable glove makes haptic interactions with arbitrary objects observable. The Tactile Pen was designed to measure grip force during handwriting in order to better facilitate therapeutic treatment assessments. These sensors have already been extensively used by various research groups, including our own, to gain a better understanding of human manual intelligence. The Finger-Force-Linear-Sensor and the Tactile Bracelet are two novel sensors that were developed to facilitate more natural control of dexterous multi Degree-of-Freedom (DOF) hand prostheses. The Finger-Force-Linear-Sensor is a very accurate bidirectional single finger force ground-truth measurement device that was designed to enable testing and development of single finger forces and muscle activations mapping algorithms. The Tactile Bracelet was designed with the goal to provide a more robust and intuitive means of control for multi-DOF hand prostheses by measuring the muscle bulgings of the remnant muscles of lower arm amputees. It is currently in development and will eventually cover the complete forearm circumference with high spatial resolution tactile sensitive surfaces. An experiment involving a large number of lower arm amputees has already been planned. The Modular flat tactile sensor system, the Fabric-based touch sensitive artificial skin and the 3D shaped tactile sensor were developed to cover and to add touch sensing capabilities to the surfaces of technical systems. The rapid augmentation of systems with a sense of touch was the main goal of the modular flat tactile sensor system. The developed sensor modules can be used alone or in an array to form larger tactile sensitive surfaces such as tactile sensitive tabletops. As many robots have curved surfaces, using flat rigid modules severely limits the areas that can be covered with tactile sensors. The Fabric-based tactile sensor, originally developed to form a tactile dataglove for human hands, can with minor modifications also function as an artificial skin for technical systems. Finally, the 3D shaped tactile sensor based on Laser-Direct-Structuring technology is a novel tactile sensor that has a true 3D shape and provides high sensitivity and a high spatial resolution. These sensors take us further along the path towards creating general purpose technical systems that in time can be of great help to us in our daily lives. The desired tactile sensor characteristics differ significantly according to which haptic interaction patterns we wish to measure. Large tactile sensor arrays that are used to capture full body haptic interactions with floors and upholstered furniture, or that are designed to cover large areas of technical system surfaces, need to be scalable, have low power consumption and should ideally have a low material cost. Two examples of such sensors are tacTiles and the Fabric-based sensor for curved surfaces. At the other end of the tactile sensor development spectrum, if we want to observe manual interactions, high spatial and temporal resolution are crucial to enable the measurement of fine grasping and manipulation actions. Our fingertips contain the highest density area of mechanoreceptors, the organs that sense mechanical pressure and distortions. Thus, to construct biologically inspired anthropomorphic robotic hands, the artificial tactile sensors for the fingertips require similar high-fidelity sensors with surfaces that are curved under small bending radii in 2 dimensions, have high spatial densities, while simultaneously providing high sensitivity. With the fingertip tactile sensor, designed to fit the Shadow Robot Hands' fingers, I show that such sensors can indeed be constructed in the 3D-shaped high spatial resolution tactile sensor section of my thesis. With my work I have made a significant contribution towards making haptics more observable. I achieved this by developing a high number of novel tactile sensors that are usable, give a deeper insight into human haptic interactions, have great potential to help amputees and that make technical systems, such as robots, more capable