Distributed Sensing and Stimulation Systems Towards Sense of Touch Restoration in Prosthetics

Modern prostheses aim at restoring the functional and aesthetic characteristics of the lost limb. To foster prosthesis embodiment and functionality, it is necessary to restitute both volitional control and sensory feedback. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing high-fidelity spatial information. To provide this type of feedback in prosthetics, it is necessary to sense tactile information from artificial skin placed on the prosthesis and transmit tactile feedback above the amputation in order to map the interaction between the prosthesis and the environment. This thesis proposes the integration of distributed sensing systems (e-skin) to acquire tactile sensation, and non-invasive multichannel electrotactile feedback and virtual reality to deliver high-bandwidth information to the user. Its core focus addresses the development and testing of close-loop sensory feedback human-machine interface, based on the latest distributed sensing and stimulation techniques for restoring the sense of touch in prosthetics. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives and the used methodology and contributions; as well as three studies distributed over stimulation system level and sensing system level. The first study presents the development of close-loop compensatory tracking system to evaluate the usability and effectiveness of electrotactile sensory feedback in enabling real-time close-loop control in prosthetics. It examines and compares the subject\u2019s adaptive performance and tolerance to random latencies while performing the dynamic control task (i.e. position control) and simultaneously receiving either visual feedback or electrotactile feedback for communicating the momentary tracking error. Moreover, it reported the minimum time delay needed for an abrupt impairment of users\u2019 performance. The experimental results have shown that electrotactile feedback performance is less prone to changes with longer delays. However, visual feedback drops faster than electrotactile with increased time delays. This is a good indication for the effectiveness of electrotactile feedback in enabling close- loop control in prosthetics, since some delays are inevitable. The second study describes the development of a novel non-invasive compact multichannel interface for electrotactile feedback, containing 24 pads electrode matrix, with fully programmable stimulation unit, that investigates the ability of able-bodied human subjects to localize the electrotactile stimulus delivered through the electrode matrix. Furthermore, it designed a novel dual parameter -modulation (interleaved frequency and intensity) and compared it to conventional stimulation (same frequency for all pads). In addition and for the first time, it compared the electrotactile stimulation to mechanical stimulation. More, it exposes the integration of virtual prosthesis with the developed system in order to achieve better user experience and object manipulation through mapping the acquired real-time collected tactile data and feedback it simultaneously to the user. The experimental results demonstrated that the proposed interleaved coding substantially improved the spatial localization compared to same-frequency stimulation. Furthermore, it showed that same-frequency stimulation was equivalent to mechanical stimulation, whereas the performance with dual-parameter modulation was significantly better. The third study presents the realization of a novel, flexible, screen- printed e-skin based on P(VDF-TrFE) piezoelectric polymers, that would cover the fingertips and the palm of the prosthetic hand (particularly the Michelangelo hand by Ottobock) and an assistive sensorized glove for stroke patients. Moreover, it developed a new validation methodology to examine the sensors behavior while being solicited. The characterization results showed compatibility between the expected (modeled) behavior of the electrical response of each sensor to measured mechanical (normal) force at the skin surface, which in turn proved the combination of both fabrication and assembly processes was successful. This paves the way to define a practical, simplified and reproducible characterization protocol for e-skin patches In conclusion, by adopting innovative methodologies in sensing and stimulation systems, this thesis advances the overall development of close-loop sensory feedback human-machine interface used for restoration of sense of touch in prosthetics. Moreover, this research could lead to high-bandwidth high-fidelity transmission of tactile information for modern dexterous prostheses that could ameliorate the end user experience and facilitate it acceptance in the daily life

Development of a Touch Stimulator for Functional Magnetic-Resonance Imaging

A tactile display system has been built with 25 contactors in a 5 × 5 array with 2mm spacing, designed to stimulate the fingertip. The drive mechanism for each contactor is a piezoelectric bimorph, allowing the display to use in functional magnetic resonance imaging experiments (fMRI). The amplitude and frequency of stimulation can be pre-set, and each contactor can be activated separately using a personal computer. The tactile produce a wide variety of time-varying spatial patterns of touch stimulation. The sensation is “natural” and the participants do not find the experience unpleasant. The psychophysics experiment and the first fMRI experiment involved identification of various patterns on the display: the tactile stimulus was stationary or moved in a circle or in a “random” trajectory with no obvious shape. Response was by push buttons. The second fMRI experiment focused on the relationship between the speed of tactile motion and the corresponding activation in the brain, using stimuli moving in a circular trajectory on the tactile display at various speeds in the range 2.9 to 77.9 mm s –1. In the psychophysics experiment, the mean identification score was 80% after only a few minutes’ practice. The results of the first fMRI experiment showed highly significant activations in primary and secondary somatosensory cortices for contrasts of circle or random stimuli with the rest condition; low significant activations in SI and SII were observed for the contrast of stationary stimuli with rest. Broca's area was found to be activated for circle and random stimulation but not for stationary stimulation. Results from the second fMRI experiment showed small speed-sensitive activations in the left side of the brain, mostly in the primary somatosensory cortex. The conclusion in present study was our tactile system can produce different types of tactile patterns and it works inside MRI scanner

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Haptics: Science, Technology, Applications

This open access book constitutes the proceedings of the 12th International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, EuroHaptics 2020, held in Leiden, The Netherlands, in September 2020. The 60 papers presented in this volume were carefully reviewed and selected from 111 submissions. The were organized in topical sections on haptic science, haptic technology, and haptic applications. This year's focus is on accessibility

Recent progress in piezotronic sensors based on one-dimensional zinc oxide nanostructures and its regularly ordered arrays: from design to application

Piezotronic sensors and self-powered gadgets are highly sought-after for flexible, wearable, and intelligent electronics for their applications in cutting-edge healthcare and human-machine interfaces. With the advantages of a well-known piezoelectric effect, excellent mechanical properties, and emerging nanotechnology applications, one-dimensional (1D) ZnO nanostructures organized in the form of a regular array have been regarded as one of the most promising inorganic active materials for piezotronics. This report intends to review the recent developments of 1D ZnO nanostructure arrays for multifunctional piezotronic sensors. Prior to discussing rational design and fabrication approaches for piezotronic devices in precisely controlled dimensions, well-established synthesis methods for high-quality and well-controlled 1D ZnO nanostructures are addressed. The challenges associated with the well-aligned, site-specific synthesis of 1D ZnO nanostructures, development trends of piezotronic devices, advantages of an ordered array of 1D ZnO in device performances, exploring new sensing mechanisms, incorporating new functionalities by constructing heterostructures, the development of novel flexible device integration technology, the deployment of novel synergistic strategies in piezotronic device performances, and potential multifunctional applications are covered. A brief evaluation of the end products, such as small-scale miniaturized unconventional power sources in sensors, high-resolution image sensors, and personalized healthcare medical devices, is also included. The paper is summarized towards the conclusion by outlining the present difficulties and promising future directions. This study will provide guidance for future research directions in 1D ZnO nanostructure-based piezotronics, which will hasten the development of multifunctional devices, sensors, chips for human-machine interfaces, displays, and self-powered systems

Leading the Charge in Bone Healing: Design of Compliant Layer Adaptive Composite Stacks for Electrical Stimulation in Orthopedic Implants

The overall aim of this research is to develop a robust, adaptable piezoelectric composite load-bearing biomaterial that when integrated with current implants, can harvest human motion and subsequently deliver electrical stimulation to trigger the natural bone healing and remodeling process. Building on the preclinical success of a stacked piezocomposite spinal fusion implant, compliant layer adaptive composite stacks (CLACS) were designed as a scalable biomaterial to increase efficiency of power generation while maintaining mechanical integrity under fatigue loading seen in orthopedic implants. Energy harvesting with piezoelectric material is challenging at low frequencies due to material properties that limit total power generation at these frequencies and brittle mechanical properties. Stacked generators increase power generation at lower voltage levels and resistances, but are not efficient at low frequencies seen in human motion. CLACS integrates compliant layers between the stiff piezoelectric elements within a stack, capitalizing on the benefits of stacked piezoelectric generators, while decreasing stiffness and increasing strain to amplify power generation. The first study evaluated CLACS under compressive loads, demonstrating the power amplification effect as the thickness of the compliant layer increases. The second study characterized the effect of poling direction of piezoelectric discs within a CLACS structure under multiaxial loads, demonstrating an additional increase in power generation when mixed poling directions are used to create mixed-mode CLACS. The final study compared the fatigue performance and power generation capability of three commercially fabricated piezoelectric stack generators with and without CLACS technology in modified implant assemblies. All configurations produced sufficient power to stimulate bone growth, and maintained mechanical strength throughout a high load, low cycle fatigue analysis, thus validating feasibility for use in orthopedic implants. The presented work in this dissertation provides a robust experimental understanding of CLACS and a characterization of how piezoelectric properties and composite structures can be tailored within the CLACS structure to efficiently generate power in low frequency, low impedance applications. The main motivation of this work was to develop a thorough understanding of CLACS behavior for implementation into medical implants to deliver therapeutic electrical stimulation and accelerate rate of bone growth, helping patients completely heal faster. However, the ability to tune composite stiffness by changing compliant material properties, type of piezoelectric material and poling direction, or volume fractions could benefit the energy harvesting potential in fields ranging from civil infrastructure to wind energy, to wearables and athletic equipment

Design and fabrication of flexible tactile sensing and feedback interface for communication by deafblind people

Humans generally interact and communicate using five basic sensory modalities and mainly through vision, touch and audio. However, this does not work for deafblind people as they have both impaired hearing and vision modalities, and hence rely on touch-sensing. This necessitates the development of alternative means that allows them to independently interact and communicate. To do this requires a solution which has the capability for tactile sensing and feedback. Therefore, tactile interface becomes a critical component of any assistive device usable by deafblind people for interaction and communication. Given that existing solutions mainly use rigid and commercial components, there is a need to tap into the advancements in flexible electronics in order develop more effective and conformable solutions. This research involves the development of flexible tactile communication interface usable in assistive communication devices for deafblind people. First, commercial sensors and actuators were utilised as a proof-of-concept and then four novel tactile interfaces were explored which include two similar touch-sensitive electromagnetic actuators, one capacitive tactile sensing array, and a facile flexible inductance-based pressure sensor. The two fabricated touch-sensitive electromagnetic actuators (Type 1 and 2) are both based on electromagnetic principle and capable of simultaneous tactile sensing and feedback. Each comprises of a tandem combination of two main modules - the touch-sensing and the actuation module, with both modules integrated as a single device in each case. The actuation module employs a flexible planar spiral coil and a Neodymium magnet assembled in a soft Polydimethylsiloxane (PDMS) structure, while the touch-sensing module is a planar capacitive metal- insulator-metal structure of copper. The flexible coil (~17µm thick and with 45 turns) was fabricated on a Polyimide sheet using Lithographie Galvanoformung Abformung (LIGA) process. The results of characterisation of these actuators at frequencies ranging from 10Hz to 200Hz, shows a maximum displacement (~ 190µm) around 40Hz. Evaluation of this by 40 (20 deafblind and 20 sighted and hearing) participants show that they can feel vibration at this range. Another tactile interface fabricated is an 8 x 8 capacitive tactile sensing array. The sensor was developed on a flexible Polyvinyl Chloride (PVC) sheet with column electrodes deposited on one side and row electrodes on the reverse side. It is intended for use as an assistive tactile communication interface for deafblind people who communicate using deafblind manual alphabets as well as the English block letters. An inductance-based pressure sensor was also designed, fabricated and characterised for use as an input interface for finger Braille as well as other tactile communication methods for deafblind people. It was realised with a soft ferromagnetic elastomer and a 17µm-thick coil fabricated on a flexible 50 µm-thick polyimide sheet. The ferromagnetic elastomer acts as the core of the coil, which when pressed, sees the metal particles moving closer to each other, leading to changes in the inductance. The coil, with 75µm conductor and 25µm pitch, was also realised using LIGA micromolding technique. Seven different sensors were fabricated using different ratios (1:1, 1:2, 1:3, 1:5, 2:1, 3:1, and 5:1) of Ecoflex to Iron particles. The performance of each sensor was investigated and generally, sensors with higher Iron particles gave better sensitivity, linear as well as dynamic range. In comparison with all other fabricated sensors, the sensor made with 1:5DD was recommended for application as a tactile interface

Thermal and Mechanical Energy Harvesting Using Lead Sulfide Colloidal Quantum Dots

The human body is an abundant source of energy in the form of heat and mechanical movement. The ability to harvest this energy can be useful for supplying low-consumption wearable and implantable devices. Thermoelectric materials are usually used to harvest human body heat for wearable devices; however, thermoelectric generators require temperature gradient across the device to perform appropriately. Since they need to attach to the heat source to absorb the heat, temperature equalization decreases their efficiencies. Moreover, the electrostatic energy harvester, working based on the variable capacitor structure, is the most compatible candidate for harvesting low-frequency-movement of the human body. Although it can provide a high output voltage and high-power density at a small scale, they require an initial start-up voltage source to charge the capacitor for initiating the conversion process. The current methods for initially charging the variable capacitor suffer from the complexity of the design and fabrication process. In this research, a solution-processed photovoltaic structure was proposed to address the temperature equalization problem of the thermoelectric generators by harvesting infrared radiations emitted from the human body. However, normal photovoltaic devices have the bandgap limitation to absorb low energy photons radiated from the human body. In this structure, mid-gap states were intentionally introduced to the absorbing layer to activate the multi-step photon absorption process enabling electron promotion from the valence band to the conduction band. The fabricated device showed promising performance in harvesting low energy thermal radiations emitted from the human body. Finally, in order to increase the generated power, a hybrid structure was proposed to harvest both mechanical and heat energy sources available in the human body. The device is designed to harvest both the thermal radiation of the human body based on the proposed solution-processed photovoltaic structure and the mechanical movement of the human body based on an electrostatic generator. The photovoltaic structure was used to charge the capacitor at the initial step of each conversion cycle. The simple fabrication process of the photovoltaic device can potentially address the problem associated with the charging method of the electrostatic generators. The simulation results showed that the combination of two methods can significantly increase the harvested energy

On the critical role of the sensorimotor loop on the design of interaction techniques and interactive devices

People interact with their environment thanks to their perceptual and motor skills. This is the way they both use objects around them and perceive the world around them. Interactive systems are examples of such objects. Therefore to design such objects, we must understand how people perceive them and manipulate them. For example, haptics is both related to the human sense of touch and what I call the motor ability. I address a number of research questions related to the design and implementation of haptic, gestural, and touch interfaces and present examples of contributions on these topics. More interestingly, perception, cognition, and action are not separated processes, but an integrated combination of them called the sensorimotor loop. Interactive systems follow the same overall scheme, with differences that make the complementarity of humans and machines. The interaction phenomenon is a set of connections between human sensorimotor loops, and interactive systems execution loops. It connects inputs with outputs, users and systems, and the physical world with cognition and computing in what I call the Human-System loop. This model provides a complete overview of the interaction phenomenon. It helps to identify the limiting factors of interaction that we can address to improve the design of interaction techniques and interactive devices.Les humains interagissent avec leur environnement grâce à leurs capacités perceptives et motrices. C'est ainsi qu'ils utilisent les objets qui les entourent et perçoivent le monde autour d'eux. Les systèmes interactifs sont des exemples de tels objets. Par conséquent, pour concevoir de tels objets, nous devons comprendre comment les gens les perçoivent et les manipulent. Par exemple, l'haptique est à la fois liée au sens du toucher et à ce que j'appelle la capacité motrice. J'aborde un certain nombre de questions de recherche liées à la conception et à la mise en œuvre d'interfaces haptiques, gestuelles et tactiles et je présente des exemples de contributions sur ces sujets. Plus intéressant encore, la perception, la cognition et l'action ne sont pas des processus séparés, mais une combinaison intégrée d'entre eux appelée la boucle sensorimotrice. Les systèmes interactifs suivent le même schéma global, avec des différences qui forme la complémentarité des humains et des machines. Le phénomène d'interaction est un ensemble de connexions entre les boucles sensorimotrices humaines et les boucles d'exécution des systèmes interactifs. Il relie les entrées aux sorties, les utilisateurs aux systèmes, et le monde physique à la cognition et au calcul dans ce que j'appelle la boucle Humain-Système. Ce modèle fournit un aperçu complet du phénomène d'interaction. Il permet d'identifier les facteurs limitatifs de l'interaction que nous pouvons aborder pour améliorer la conception des techniques d'interaction et des dispositifs interactifs