Prosthetic Control and Sensory Feedback for Upper Limb Amputees

Hand amputation could dramatically degrade the life quality of amputees. Many amputees use prostheses to restore part of the hand functions. Myoelectric prosthesis provides the most dexterous control. However, they are facing high rejection rate. One of the reasons is the lack of sensory feedback. There is a need for providing sensory feedback for myoelectric prosthesis users. It can improve object manipulation abilities, enhance the perceptual embodiment of myoelectric prostheses and help reduce phantom limb pain. This PhD work focuses on building bi-directional prostheses for upper limb amputees. In the introduction chapter, first, an overview of upper limb amputee demographics and upper limb prosthesis is given. Then the human somatosensory system is briefly introduced. The next part reviews invasive and non-invasive sensory feedback methods reported in the literature. The rest of the chapter describes the motivation of the project and the thesis organization. The first step to build a bi-directional prostheses is to investigate natural and robust multifunctional prosthetic control. Most of the commerical prostheses apply non-pattern recognition based myoelectric control methods, which offers only limited functionalities. In this thesis work, pattern recognition based prosthetic control employing three commonly used and representative machine learning algorithms is investigated. Three datasets involving different levels of upper arm movements are used for testing the algorithm effectiveness. The influence of time-domain features, window and increment sizes, algorithms, and post-processing techniques are analyzed and discussed. The next three chapters address different aspects of providing sensory feedback. The first focus of sensory feedback process is the automatic phantom map detection. Many amputees have referred sensation from their missing hand on their residual limbs (phantom maps). This skin area can serve as a target for providing amputees with non-invasive tactile sensory feedback. One of the challenges of providing sensory feedback on the phantom map is to define the accurate boundary of each phantom digit because the phantom map distribution varies from person to person. Automatic phantom map detection methods based on four decomposition support vector machine algorithms and three sampling methods are proposed. The accuracy and training/ classification time of each algorithm using a dense stimulation array and two coarse stimulation arrays are presented and compared. The next focus of the thesis is to develop non-invasive tactile display. The design and psychophysical testing results of three types of non-invasive tactile feedback arrays are presented: two with vibrotactile modality and one with multi modality. For vibrotactile, two types of miniaturized vibrators: eccentric rotating masses (ERMs) and linear resonant actuators (LRAs) were first tested on healthy subjects and their effectiveness was compared. Then the ERMs are integrated into a vibrotactile glove to assess the feasibility of providing sensory feedback for unilateral upper limb amputees on the contralateral hand. For multimodal stimulation, miniature multimodal actuators integrating servomotors and vibrators were designed. The actuator can be used to deliver both high-frequency vibration and low-frequency pressures simultaneously. By utilizing two modalities at the same time, the actuator stimulates different types of mechanoreceptors and thus h

Novel Bidirectional Body - Machine Interface to Control Upper Limb Prosthesis

Objective. The journey of a bionic prosthetic user is characterized by the opportunities and limitations involved in adopting a device (the prosthesis) that should enable activities of daily living (ADL). Within this context, experiencing a bionic hand as a functional (and, possibly, embodied) limb constitutes the premise for mitigating the risk of its abandonment through the continuous use of the device. To achieve such a result, different aspects must be considered for making the artificial limb an effective support for carrying out ADLs. Among them, intuitive and robust control is fundamental to improving amputees’ quality of life using upper limb prostheses. Still, as artificial proprioception is essential to perceive the prosthesis movement without constant visual attention, a good control framework may not be enough to restore practical functionality to the limb. To overcome this, bidirectional communication between the user and the prosthesis has been recently introduced and is a requirement of utmost importance in developing prosthetic hands. Indeed, closing the control loop between the user and a prosthesis by providing artificial sensory feedback is a fundamental step towards the complete restoration of the lost sensory-motor functions. Within my PhD work, I proposed the development of a more controllable and sensitive human-like hand prosthesis, i.e., the Hannes prosthetic hand, to improve its usability and effectiveness. Approach. To achieve the objectives of this thesis work, I developed a modular and scalable software and firmware architecture to control the Hannes prosthetic multi-Degree of Freedom (DoF) system and to fit all users’ needs (hand aperture, wrist rotation, and wrist flexion in different combinations). On top of this, I developed several Pattern Recognition (PR) algorithms to translate electromyographic (EMG) activity into complex movements. However, stability and repeatability were still unmet requirements in multi-DoF upper limb systems; hence, I started by investigating different strategies to produce a more robust control. To do this, EMG signals were collected from trans-radial amputees using an array of up to six sensors placed over the skin. Secondly, I developed a vibrotactile system to implement haptic feedback to restore proprioception and create a bidirectional connection between the user and the prosthesis. Similarly, I implemented an object stiffness detection to restore tactile sensation able to connect the user with the external word. This closed-loop control between EMG and vibration feedback is essential to implementing a Bidirectional Body - Machine Interface to impact amputees’ daily life strongly. For each of these three activities: (i) implementation of robust pattern recognition control algorithms, (ii) restoration of proprioception, and (iii) restoration of the feeling of the grasped object's stiffness, I performed a study where data from healthy subjects and amputees was collected, in order to demonstrate the efficacy and usability of my implementations. In each study, I evaluated both the algorithms and the subjects’ ability to use the prosthesis by means of the F1Score parameter (offline) and the Target Achievement Control test-TAC (online). With this test, I analyzed the error rate, path efficiency, and time efficiency in completing different tasks. Main results. Among the several tested methods for Pattern Recognition, the Non-Linear Logistic Regression (NLR) resulted to be the best algorithm in terms of F1Score (99%, robustness), whereas the minimum number of electrodes needed for its functioning was determined to be 4 in the conducted offline analyses. Further, I demonstrated that its low computational burden allowed its implementation and integration on a microcontroller running at a sampling frequency of 300Hz (efficiency). Finally, the online implementation allowed the subject to simultaneously control the Hannes prosthesis DoFs, in a bioinspired and human-like way. In addition, I performed further tests with the same NLR-based control by endowing it with closed-loop proprioceptive feedback. In this scenario, the results achieved during the TAC test obtained an error rate of 15% and a path efficiency of 60% in experiments where no sources of information were available (no visual and no audio feedback). Such results demonstrated an improvement in the controllability of the system with an impact on user experience. Significance. The obtained results confirmed the hypothesis of improving robustness and efficiency of a prosthetic control thanks to of the implemented closed-loop approach. The bidirectional communication between the user and the prosthesis is capable to restore the loss of sensory functionality, with promising implications on direct translation in the clinical practice

Closed-loop prosthetic hand : understanding sensorimotor and multisensory integration under uncertainty.

To make sense of our unpredictable world, humans use sensory information streaming through billions of peripheral neurons. Uncertainty and ambiguity plague each sensory stream, yet remarkably our perception of the world is seamless, robust and often optimal in the sense of minimising perceptual variability. Moreover, humans have a remarkable capacity for dexterous manipulation. Initiation of precise motor actions under uncertainty requires awareness of not only the statistics of our environment but also the reliability of our sensory and motor apparatus. What happens when our sensory and motor systems are disrupted? Upper-limb amputees tted with a state-of-the-art prostheses must learn to both control and make sense of their robotic replacement limb. Tactile feedback is not a standard feature of these open-loop limbs, fundamentally limiting the degree of rehabilitation. This thesis introduces a modular closed-loop upper-limb prosthesis, a modified Touch Bionics ilimb hand with a custom-built linear vibrotactile feedback array. To understand the utility of the feedback system in the presence of multisensory and sensorimotor influences, three fundamental open questions were addressed: (i) What are the mechanisms by which subjects compute sensory uncertainty? (ii) Do subjects integrate an artificial modality with visual feedback as a function of sensory uncertainty? (iii) What are the influences of open-loop and closed-loop uncertainty on prosthesis control? To optimally handle uncertainty in the environment people must acquire estimates of the mean and uncertainty of sensory cues over time. A novel visual tracking experiment was developed in order to explore the processes by which people acquire these statistical estimators. Subjects were required to simultaneously report their evolving estimate of the mean and uncertainty of visual stimuli over time. This revealed that subjects could accumulate noisy evidence over the course of a trial to form an optimal continuous estimate of the mean, hindered only by natural kinematic constraints. Although subjects had explicit access to a measure of their continuous objective uncertainty, acquired from sensory information available within a trial, this was limited by a conservative margin for error. In the Bayesian framework, sensory evidence (from multiple sensory cues) and prior beliefs (knowledge of the statistics of sensory cues) are combined to form a posterior estimate of the state of the world. Multiple studies have revealed that humans behave as optimal Bayesian observers when making binary decisions in forced-choice tasks. In this thesis these results were extended to a continuous spatial localisation task. Subjects could rapidly accumulate evidence presented via vibrotactile feedback (an artificial modality ), and integrate it with visual feedback. The weight attributed to each sensory modality was chosen so as to minimise the overall objective uncertainty. Since subjects were able to combine multiple sources of sensory information with respect to their sensory uncertainties, it was hypothesised that vibrotactile feedback would benefit prosthesis wearers in the presence of either sensory or motor uncertainty. The closed-loop prosthesis served as a novel manipulandum to examine the role of feed-forward and feed-back mechanisms for prosthesis control, known to be required for successful object manipulation in healthy humans. Subjects formed economical grasps in idealised (noise-free) conditions and this was maintained even when visual, tactile and both sources of feedback were removed. However, when uncertainty was introduced into the hand controller, performance degraded significantly in the absence of visual or tactile feedback. These results reveal the complementary nature of feed-forward and feed-back processes in simulated prosthesis wearers, and highlight the importance of tactile feedback for control of a prosthesis

A Haptic Feedback System for Lower Limb Amputees Based on Gait Event Detection

Lower limb amputation has significant effects on a person’s quality of life and ability to perform activities of daily living. Prescription of prosthetic device post amputation aims to help restore some degrees of mobility function, however studies have shown evidence of low balance confidence and higher risk of falling among amputee community, especially those suffering from above knee amputation. While advanced prostheses offer better control, they often lack a form of feedback that delivers the awareness of the limb position to the prosthetic user while walking. This research presents the development and evaluation of a wearable skinstretch haptic feedback system intended to deliver cues of two crucial gait events, namely the Initial Contact (IC) and Toe-off (TO) to its wearer. The system comprises a haptic module that applies lateral skin-stretch on the upper leg or the trunk, corresponding to the gait event detection module based on Inertial Measurement Unit (IMU) attached at the shank. The design and development iterations of the haptic module is presented, and characterization of the feedback parameters is discussed. The validation of the gait event detection module is carried out and finally the integration of the haptic feedback system is described. Experimental work with healthy subjects and an amputee indicated good perceptibility of the feedback during static and dynamic (walking) condition, although higher magnitude of stretch was required to perceive the feedback during dynamic condition. User response time during dynamic activity showed that the haptic feedback system is suitable for delivering cues of IC and TO within the duration of the stance phase. In addition, feedback delivered in discernible patterns can be learned and adapted by the subjects. Finally, a case study was carried out with an above-knee amputee to assess the effects of the haptic feedback on spatio-temporal gait parameters and on the vertical ground reaction force during treadmill and overground walking. The research presented in this report introduces a novel design of a haptic feedback device. As such, the outcome includes a well-controlled skin-stretch effect which contributes to the research by investigating skin-stretch feedback for conveying discrete event information rather than conveying direction information as presented in other studies. In addition, it is found that stretch magnitude as small as 3 mm could be perceived in short duration of 150 ms during dynamic condition, making it a suitable alternative to other widely investigated haptic modality such as vibration for ambulatory feedback application. With continuous training, the haptic feedback system could possibly benefit lower limb amputees by creating awareness of the limb placement during ambulation, potentially reducing visual dependency and increasing walking confidence

A Pilot Study of a Thermal Display Using a Miniature Tactor for Upper Extremity Prosthesis

1

Tactile displays, design and evaluation

Fritschi M. Tactile displays, design and evaluation. Bielefeld: Universität Bielefeld; 2016.This thesis presents the design and development of several tactile displays, as well as their eventual integration into a framework of tactile and kinesthetic stimulation. As a basis for the design of novel devices, an extensive survey of existing actuator principles and existing realizations of tactile displays is complemented by neurobiological and psychophysical findings. The work is structured along three main goals: First, novel actuator concepts are explored whose performance can match the challenging capabilities of human tactile perception. Second, novel kinematic concepts for experimental platforms are investigated that target an almost unknown sub-modality of tactile perception: The perception of shear force. Third, a setup for integrated tactile-kinesthetic displays is realized, and a first study on the psychophysical correlation between the tactile and the kinesthetic portion of haptic information is conducted. The developed devices proved to exceed human tactile capabilities and have already been used to learn more about the human tactile sense