Reflex: A Closed-Loop Tactile Feedback System for Use in Upper Limb Prosthesis Grip Control

Tactile sensing provides valuable insight to the environment in which we interact with. Upper limb amputees lack the sensations that generates the necessary information to stably grasp the wide variety of objects we interact with on a daily basis. Utilizing tactile sensing to provide feedback to a prosthetic hand provides a mechanism for replacing the grip control functionality of the mechanoreceptors found in human skin. Novel customizable, low cost tactile sensors for monitoring the dynamics of an object grasped by a prosthetic hand are developed and presented as part of this thesis. The response of sensors placed on a prosthetic hand provides information regarding the state of a grasped object, particularly contact and slip. The sensors are made up of various textile materials, including stretchable interfacing layers and conductive traces. Essentially a force sensitive resistor, each sensor is shaped into stretchable cu ff that can be placed around the finger of a prosthetic hand. An outer rubber layer on the sensor provides compliance, which is found to enhance grasping performance with a prosthesis. Two control algorithms were developed as part of the closed-loop tactile feedback system, called Reflex, to enhance grasping functionality with a prosthesis. A Contact Detection strategy uses force information to effectively reduce the user's electromyography (EMG) signals, which are used to control the prosthesis. Essentially, the goal of this strategy is to help a user grab fragile objects without breaking them. A second strategy, Slip Prevention, uses the derivative of a force signal to detect slip of a grasped object. Instances of slip trigger electrical pulses sent from the prosthesis control unit to close the hand in an effort to prevent additional slip. The Reflex system, comprised of two control strategies along with flexible textile based force sensors on the fingers of a prosthesis, was shown to improve the grasping functionality of a prosthesis under normal use conditions. Able body participants were used to test the system. Results show the sensors' ability to greatly enhance grasping fragile objects while also helping prevent object slip. The compliant nature of the sensors enables users to more confidently pick up and move small,fragile objects, such as foam peanuts and crackers. Without sensors and tactile feedback, users had a higher likelihood of breaking objects while grabbing them. The addition of sensors reduced this failure rate, and the failure rate was reduced even further with the implementation of control algorithms running in real-time. The slip prevention strategy was also shown to help reduce the amount of object movement after a grasp is initiated, although the most benefit comes from the compliant nature of the sensors. Reflex is the first closed-loop tactile feedback system with multiple control strategies that can be used on a prosthetic hand to enhance grasping functionality. The system allows one to switch between Contact Detection or Slip Prevention control strategies, giving the user the ability to use each control as needed. Feedback from the textile sensors directly to the prosthesis control unit provides valuable information regarding grasping forces. This research aims to help improve prosthetic technology so that one day amputees will feel as if their device is a natural extension of their body

Human-centered Electric Prosthetic (HELP) Hand

In developing countries such as India, there is a higher rate of amputations among the population but a lack of viable, low cost solutions. Through a partnership with Indian non-profit Bhagwan Mahaveer Viklang Sahayata Samiti (BMVSS), the team designed a functional, robust, and low cost electrically powered prosthetic hand that communicates with people with unilateral, transradial amputations in urban India through a biointerface. The device uses compliant tendon actuation, small linear servos, and a wearable sleeve outfitted with electromyography (EMG) sensors to produce a device that, once placed inside a prosthetic glove, is anthropomorphic in both look and feel. The hand is capable of forming three grips through the use of a manually adjustable opposable thumb: the key, pinch, and wrap grips. The hand also provides vibrotactile user feedback upon completion of a grip. The design includes a prosthetic gel liner to provide a layer of cushion and comfort for safe use by the user. These results show that it is possible to create a low cost, electrically powered prosthetic hand for users in developing countries without sacrificing functionality. In order for this design to be truly adjustable to each user, the creation of an easily navigable graphical user interface (GUI) will have to be a future goal. The prosthesis prototype was developed such that future groups can design for manufacturing and distribution in India

A Review of Non-Invasive Haptic Feedback stimulation Techniques for Upper Extremity Prostheses

A sense of touch is essential for amputees to reintegrate into their social and work life. The design of the next generation of the prostheses will have the ability to effectively convey the tactile information between the amputee and the artificial limbs. This work reviews non-invasive haptic feedback stimulation techniques to convey the tactile information from the prosthetic hand to the amputee’s brain. Various types of actuators that been used to stimulate the patient’s residual limb for different types of artificial prostheses in previous studies have been reviewed in terms of functionality, effectiveness, wearability and comfort. The non-invasive hybrid feedback stimulation system was found to be better in terms of the stimulus identification rate of the haptic prostheses’ users. It can be conclude that integrating hybrid haptic feedback stimulation system with the upper limb prostheses leads to improving its acceptance among users

A hybrid haptic stimulation prosthetic wearable device to recover the missing sensation of the upper limb amputees

A hybrid haptic feedback stimulation system that is capable in sensing the contact pressure, the surface texture, and the temperature, simultaneously, was designed for a prosthetic hand to provide a tactile sensation to amputation patients. In addition, the haptic system was developed to enable the prosthetic’s users to implement withdrawal reflexes due to the thermal noxious stimulus in a quick manner. The re-sensation is achieved by non-invasively stimulating the skin of the patients’ residual limbs, based on the type and the level of tactile signals provided by the sensory system of the prostheses. Accordingly, three stages of design and development were performed to satisfy the research methodology. A vibrotactile prosthetic device, which is designed for the detection of contact pressure and surface texture in upper extremity, represents. While, the design of a novel wearable hybrid pressure-vibration haptic feedback stimulation device for conveying the tactile information regarding the contact pressure between the prosthetic hand and the grasped objects represents the second methodology stage. Lastly, the third stage was achieved by designing a novel hybrid pressure-vibration-temperature feedback stimulation system to provide a huge information regarding the prostheses environment to the users without brain confusing or requiring long pre-training. The main contribution of this work is the development and evaluation of the first step of a novel approach for a lightweight, 7 Degrees-Of-Freedom (DOF) tactile prosthetic arm to perform an effective as well as fast object manipulation and grasping. Furthermore, this study investigates the ability to convey the tactile information about the contact pressure, surface texture, and object temperature to the amputees with high identification accuracy by mean of using the designed hybrid pressure-vibration-temperature feedback wearable device. An evaluation of sensation and response has been conducted on forty healthy volunteers to evaluate the ability of the haptic system to stimulate the human nervous system. The results in term of Stimulus Identification Rate (SIR) show that all the volunteers were correctly able to discriminate the sensation of touch, start of touch, end of touch, and grasping objects. While 94%, 96%, 97%, and 95.24% of the entire stimuli were successfully identified by the volunteers during the experiments of slippage, pressure level, surface texture, and temperature, respectively. The position tracking controller system was designed to synchronize the movements of the volunteers’ elbow joints and the prosthetic’s elbow joint to record the withdrawal reflexes. The results verified the ability of the haptic system to excite the human brain at the abnormal noxious stimulus and enable the volunteers to perform a quick withdrawal reflex within 0.32 sec. The test results and the volunteers' response established evidence that amputees are able to recover their sense of the contact pressure, the surface texture, and the object temperature as well as to perform thermal withdrawal reflexes using the solution developed in this work

Distributed Sensing and Stimulation Systems Towards Sense of Touch Restoration in Prosthetics

Modern prostheses aim at restoring the functional and aesthetic characteristics of the lost limb. To foster prosthesis embodiment and functionality, it is necessary to restitute both volitional control and sensory feedback. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing high-fidelity spatial information. To provide this type of feedback in prosthetics, it is necessary to sense tactile information from artificial skin placed on the prosthesis and transmit tactile feedback above the amputation in order to map the interaction between the prosthesis and the environment. This thesis proposes the integration of distributed sensing systems (e-skin) to acquire tactile sensation, and non-invasive multichannel electrotactile feedback and virtual reality to deliver high-bandwidth information to the user. Its core focus addresses the development and testing of close-loop sensory feedback human-machine interface, based on the latest distributed sensing and stimulation techniques for restoring the sense of touch in prosthetics. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives and the used methodology and contributions; as well as three studies distributed over stimulation system level and sensing system level. The first study presents the development of close-loop compensatory tracking system to evaluate the usability and effectiveness of electrotactile sensory feedback in enabling real-time close-loop control in prosthetics. It examines and compares the subject\u2019s adaptive performance and tolerance to random latencies while performing the dynamic control task (i.e. position control) and simultaneously receiving either visual feedback or electrotactile feedback for communicating the momentary tracking error. Moreover, it reported the minimum time delay needed for an abrupt impairment of users\u2019 performance. The experimental results have shown that electrotactile feedback performance is less prone to changes with longer delays. However, visual feedback drops faster than electrotactile with increased time delays. This is a good indication for the effectiveness of electrotactile feedback in enabling close- loop control in prosthetics, since some delays are inevitable. The second study describes the development of a novel non-invasive compact multichannel interface for electrotactile feedback, containing 24 pads electrode matrix, with fully programmable stimulation unit, that investigates the ability of able-bodied human subjects to localize the electrotactile stimulus delivered through the electrode matrix. Furthermore, it designed a novel dual parameter -modulation (interleaved frequency and intensity) and compared it to conventional stimulation (same frequency for all pads). In addition and for the first time, it compared the electrotactile stimulation to mechanical stimulation. More, it exposes the integration of virtual prosthesis with the developed system in order to achieve better user experience and object manipulation through mapping the acquired real-time collected tactile data and feedback it simultaneously to the user. The experimental results demonstrated that the proposed interleaved coding substantially improved the spatial localization compared to same-frequency stimulation. Furthermore, it showed that same-frequency stimulation was equivalent to mechanical stimulation, whereas the performance with dual-parameter modulation was significantly better. The third study presents the realization of a novel, flexible, screen- printed e-skin based on P(VDF-TrFE) piezoelectric polymers, that would cover the fingertips and the palm of the prosthetic hand (particularly the Michelangelo hand by Ottobock) and an assistive sensorized glove for stroke patients. Moreover, it developed a new validation methodology to examine the sensors behavior while being solicited. The characterization results showed compatibility between the expected (modeled) behavior of the electrical response of each sensor to measured mechanical (normal) force at the skin surface, which in turn proved the combination of both fabrication and assembly processes was successful. This paves the way to define a practical, simplified and reproducible characterization protocol for e-skin patches In conclusion, by adopting innovative methodologies in sensing and stimulation systems, this thesis advances the overall development of close-loop sensory feedback human-machine interface used for restoration of sense of touch in prosthetics. Moreover, this research could lead to high-bandwidth high-fidelity transmission of tactile information for modern dexterous prostheses that could ameliorate the end user experience and facilitate it acceptance in the daily life

Human-Machine Interfaces using Distributed Sensing and Stimulation Systems

As the technology moves towards more natural human-machine interfaces (e.g. bionic limbs, teleoperation, virtual reality), it is necessary to develop a sensory feedback system in order to foster embodiment and achieve better immersion in the control system. Contemporary feedback interfaces presented in research use few sensors and stimulation units to feedback at most two discrete feedback variables (e.g. grasping force and aperture), whereas the human sense of touch relies on a distributed network of mechanoreceptors providing a wide bandwidth of information. To provide this type of feedback, it is necessary to develop a distributed sensing system that could extract a wide range of information during the interaction between the robot and the environment. In addition, a distributed feedback interface is needed to deliver such information to the user. This thesis proposes the development of a distributed sensing system (e-skin) to acquire tactile sensation, a first integration of distributed sensing system on a robotic hand, the development of a sensory feedback system that compromises the distributed sensing system and a distributed stimulation system, and finally the implementation of deep learning methods for the classification of tactile data. It\u2019s core focus addresses the development and testing of a sensory feedback system, based on the latest distributed sensing and stimulation techniques. To this end, the thesis is comprised of two introductory chapters that describe the state of art in the field, the objectives, and the used methodology and contributions; as well as six studies that tackled the development of human-machine interfaces

Control system of bionic hand

V této práci hlavní důraz je kladen na vývoj systému zpětné vazby pro řízení dostupných protéz horní končetiny HACKberry, který byl navržen japonskou společností Exiii a je otevřeným projektem. Tato protéza byla vytištěna na 3D tiskárně a modernizována na Technické univerzitě v Liberci. V této diplomové práci zkoumána možnost vytvoření zpětnovazebního systému založeného na hmatové zpětné vazbě. K dispozici jsou také další možnosti implementace tohoto úkolu, které byly vyvinuty nebo jsou vyvíjeny jinými společnostmi a institucemi, které řeší stejné úkoly. V našem projektu používá dálkové ovládání ruční protézy, které se provádí pomocí bezdrátového přenosu signálů z ovládacího zařízení, - desky Arduino Esplora - na samotnou protézu, kde ovládání patří do desky Arduino Micro. Kromě toho je k dispozici provedení různých gest pomocí myoelektrických signálů. Pro organizaci zpětnovazebního systému bylo rozhodnuto použít dva typy snímačů - snímač ohybu a snímač síly. Cenová dostupnost těchto snímačů umožňuje vývoj levného zpětnovazebního systému pro rozšíření funkčnosti protézy. Kromě použití komerčních senzorů v diplomové práci se také zvažuje možnost vytváření ručně vyrobených senzorů z inteligentních materiálů. Všechny typy senzorů - komerční a ručně vyrobené - byly testovány a vyhodnoceny z hlediska jejich technických vlastností a možnosti použití v našem projektu. Výsledky testů jsou také k dispozici.This work focuses on the development of feedback control system for inexpensive prosthetic hand HACKberry. This hand was originally designed in Japan and then printed with 3D printing technology and modernized in Technical university of Liberec. Before explaining the components and the feedback control design some information about previous works of this topic was collected and presented here. Arduino Esplora board is used here for establishing remote gestures and grips control due to its functional ability. The circuit of the hand powered by Arduino Micro board is not changed and thus, it is possible to control the hand both with muscles activity which and with Arduino Esplora. Communication between Arduino Esplora and Arduino Micro is performed via the wireless transmission. Since the prosthetic hand should be extended with feedback control it was decided to utilize force sensing resistors and flex sensors. Such types of sensors match to the project aims and at the same time allow the prosthetic hand remains at the same price category. Another advantage is in their affordability. Furthermore some hand-made sensors were created. All of the sensors were tested and estimated. The results of these tests are discussed herein

A strengthened and sensorised custom silicone glove for use with an intelligent prosthetic hand

External gloves for anthropomorphic prosthetic hands protect the mechanisms from damage and ingress of contaminants and can be used to create a pleasing, life-like appearance. The properties of the glove material are the result of a compromise between the resistance to damage and flexibility. Silicone gloves are easier to flex and keep clean, but also more easily damaged. This paper details the use of nanoclay fillers to enhance the properties of silicone, successfully increasing strength whilst maintaining flexibility. The performance of the enhanced silicone is as robust and resistant to tear and puncture as commercial gloves, while being more flexible. This flexibility makes the incorporation of a piezo-electric pressure sensor based on the EEonyx conductive fabric, practical. A sandwich of the cloth and copper fabric creates the sensor, which decreases in resistance with increasing pressure. The sensors are characterised and production variability within the silicone are tested. Three sensors are incorporated into a glove made to fit around a Southampton Intelligent Hand. The hand adapts its grip shape and force depending on the object held. The technology is adaptable and it can be incorporated in a glove produced to fit any prosthetic hand. <br/

Examination of the performance characteristics of velostat as an in-socket pressure sensor

Velostat is a low-cost, low-profile electrical bagging material with piezoresistive properties, making it an attractive option for in-socket pressure sensing. The focus of this research was to explore the suitability of a Velostat-based system for providing real-time socket pressure profiles. The prototype system performance was explored through a series of bench tests to determine properties including accuracy, repeatability and hysteresis responses, and through participant testing with a single subject. The fabricated sensors demonstrated mean accuracy errors of 110 kPa and significant cyclical and thermal drift effects of up to 0.00715 V/cycle and leading to up to a 67% difference in voltage range respectively. Despite these errors the system was able to capture data within a prosthetic socket, aligning to expected contact and loading patterns for the socket and amputation type. Distinct pressure maps were obtained for standing and walking tasks displaying loading patterns indicative of posture and gait phase. The system demonstrated utility for assessing contact and movement patterns within a prosthetic socket, potentially useful for improvement of socket fit, in a low cost, low profile and adaptable format. However, Velostat requires significant improvement in its electrical properties before proving suitable for accurate pressure measurement tools in lower limb prosthetics