Robust Electromyography Based Control of Multifunctional Prostheses of The Upper Extremity

Multifunctional, highly dexterous and complex mechanic hand prostheses are emerging and currently entering the market. However, the bottleneck to fully exploiting all capabilities of these mechatronic devices, and to making all available functions controllable reliably and intuitively by the users, remains a considerable challenge. The robustness of scientific methods proposed to overcome this barrier is a crucial factor for their future commercial success. Therefore, in this thesis the matter of robust, multifunctional and dexterous control of prostheses of the upper limb was addressed and some significant advancements in the scientific field were aspired. To this end, several investigations grouped in four studies were conducted, all with the same focus on understanding mechanisms that influence the robustness of myoelectric control and resolving their deteriorating effects. For the first study, a thorough literature review of the field was conducted and it was revealed that many non-stationarities, which could be expected to affect the reliability of surface EMG pattern recognition myoprosthesis control, had been identified and studied previously. However, one significant factor had not been addressed to a sufficient extent: the effect of long-term usage and day-to-day testing. Therefore, a dedicated study was designed and carried out, in order to address the previously unanswered question of how reliable surface electromyography pattern recognition was across days. Eleven subjects, involving both able-bodied and amputees, participated in this study over the course of 5 days, and a pattern recognition system was tested without daily retraining. As the main result of this study, it was revealed that the time between training and testing a classifier was indeed a very relevant factor influencing the classification accuracy. More estimation errors were observed as more time lay between the classifier training and testing. With the insights obtained from the first study, the need for compensating signal non-stationarities was identified. Hence, in a second study, building up on the data obtained from the first investigation, a self-correction mechanism was elaborated. The goal of this approach was to increase the systems robustness towards non-stationarities such as those identified in the first study. The system was capable of detecting and correcting its own mistakes, yielding a better estimation of movements than the uncorrected classification or other, previously proposed strategies for error removal. In the third part of this thesis, the previously investigated ideas for error suppression for increased robustness of a classification based system were extended to regression based movement estimation. While the same method as tested in the second study was not directly applicable to regression, the same underlying idea was used for developing a novel proportional estimator. It was validated in online tests, with the control of physical prostheses by able-bodied and transradial amputee subjects. The proposed method, based on common spatial patterns, outperformed two state-of-the art control methods, demonstrating the benefit of increased robustness in movement estimation during applied tasks. The results showed the superior performance of robust movement estimation in real life investigations, which would have hardly been observable in offline or abstract cursor control tests, underlining the importance of tests with physical prostheses. In the last part of this work, the limitation of sequential movements of the previously explored system was addressed and a methodology for enhancing the system with simultaneous and proportional control was developed. As a result of these efforts, a system robust, natural and fluent in its movements was conceived. Again, online control tests of physical prostheses were performed by able-bodied and amputee subjects, and the novel system proved to outperform the sequential controller of the third study of this thesis, yielding the best control technique tested. An extensive set of tests was conducted with both able-bodied and amputee subjects, in scenarios close to clinical routine. Custom prosthetic sockets were manufactured for all subjects, allowing for experimental control of multifunction prostheses with advanced machine learning based algorithms in real-life scenarios. The tests involved grasping and manipulating objects, in ways as they are often encountered in everyday living. Similar investigations had not been conducted before. One of the main conclusions of this thesis was that the suppression of wrong prosthetic motions was a key factor for robust prosthesis control and that simultaneous wrist control was a beneficial asset especially for experienced users. As a result of all investigations performed, clinically relevant conclusions were drawn from these tests, maximizing the impact of the developed systems on potential future commercialization of the newly conceived control methods. This was emphasized by the close collaboration with Otto Bock as an industrial partner of the AMYO project and hence this work.2016-02-2

A real-time, practical sensor fault-tolerant module for robust EMG pattern recognition

BACKGROUND: Unreliability of surface EMG recordings over time is a challenge for applying the EMG pattern recognition (PR)-controlled prostheses in clinical practice. Our previous study proposed a sensor fault-tolerant module (SFTM) by utilizing redundant information in multiple EMG signals. The SFTM consists of multiple sensor fault detectors and a self-recovery mechanism that can identify anomaly in EMG signals and remove the recordings of the disturbed signals from the input of the pattern classifier to recover the PR performance. While the proposed SFTM has shown great promise, the previous design is impractical. A practical SFTM has to be fast enough, lightweight, automatic, and robust under different conditions with or without disturbances. METHODS: This paper presented a real-time, practical SFTM towards robust EMG PR. A novel fast LDA retraining algorithm and a fully automatic sensor fault detector based on outlier detection were developed, which allowed the SFTM to promptly detect disturbances and recover the PR performance immediately. These components of SFTM were then integrated with the EMG PR module and tested on five able-bodied subjects and a transradial amputee in real-time for classifying multiple hand and wrist motions under different conditions with different disturbance types and levels. RESULTS: The proposed fast LDA retraining algorithm significantly shortened the retraining time from nearly 1 s to less than 4 ms when tested on the embedded system prototype, which demonstrated the feasibility of a nearly “zero-delay” SFTM that is imperceptible to the users. The results of the real-time tests suggested that the SFTM was able to handle different types of disturbances investigated in this study and significantly improve the classification performance when one or multiple EMG signals were disturbed. In addition, the SFTM could also maintain the system’s classification performance when there was no disturbance. CONCLUSIONS: This paper presented a real-time, lightweight, and automatic SFTM, which paved the way for reliable and robust EMG PR for prosthesis control

A real-time, practical sensor fault-tolerant module for robust EMG pattern recognition

BackgroundUnreliability of surface EMG recordings over time is a challenge for applying the EMG pattern recognition (PR)-controlled prostheses in clinical practice. Our previous study proposed a sensor fault-tolerant module (SFTM) by utilizing redundant information in multiple EMG signals. The SFTM consists of multiple sensor fault detectors and a self-recovery mechanism that can identify anomaly in EMG signals and remove the recordings of the disturbed signals from the input of the pattern classifier to recover the PR performance. While the proposed SFTM has shown great promise, the previous design is impractical. A practical SFTM has to be fast enough, lightweight, automatic, and robust under different conditions with or without disturbances.MethodsThis paper presented a real-time, practical SFTM towards robust EMG PR. A novel fast LDA retraining algorithm and a fully automatic sensor fault detector based on outlier detection were developed, which allowed the SFTM to promptly detect disturbances and recover the PR performance immediately. These components of SFTM were then integrated with the EMG PR module and tested on five able-bodied subjects and a transradial amputee in real-time for classifying multiple hand and wrist motions under different conditions with different disturbance types and levels.ResultsThe proposed fast LDA retraining algorithm significantly shortened the retraining time from nearly 1s to less than 4ms when tested on the embedded system prototype, which demonstrated the feasibility of a nearly “zero-delay” SFTM that is imperceptible to the users. The results of the real-time tests suggested that the SFTM was able to handle different types of disturbances investigated in this study and significantly improve the classification performance when one or multiple EMG signals were disturbed. In addition, the SFTM could also maintain the system’s classification performance when there was no disturbance.ConclusionsThis paper presented a real-time, lightweight, and automatic SFTM, which paved the way for reliable and robust EMG PR for prosthesis control

Towards Natural Control of Artificial Limbs

The use of implantable electrodes has been long thought as the solution for a more natural control of artificial limbs, as these offer access to long-term stable and physiologically appropriate sources of control, as well as the possibility to elicit appropriate sensory feedback via neurostimulation. Although these ideas have been explored since the 1960’s, the lack of a long-term stable human-machine interface has prevented the utilization of even the simplest implanted electrodes in clinically viable limb prostheses.In this thesis, a novel human-machine interface for bidirectional communication between implanted electrodes and the artificial limb was developed and clinically implemented. The long-term stability was achieved via osseointegration, which has been shown to provide stable skeletal attachment. By enhancing this technology as a communication gateway, the longest clinical implementation of prosthetic control sourced by implanted electrodes has been achieved, as well as the first in modern times. The first recipient has used it uninterruptedly in daily and professional activities for over one year. Prosthetic control was found to improve in resolution while requiring less muscular effort, as well as to be resilient to motion artifacts, limb position, and environmental conditions.In order to support this work, the literature was reviewed in search of reliable and safe neuromuscular electrodes that could be immediately used in humans. Additional work was conducted to improve the signal-to-noise ratio and increase the amount of information retrievable from extraneural recordings. Different signal processing and pattern recognition algorithms were investigated and further developed towards real-time and simultaneous prediction of limb movements. These algorithms were used to demonstrate that higher functionality could be restored by intuitive control of distal joints, and that such control remains viable over time when using epimysial electrodes. Lastly, the long-term viability of direct nerve stimulation to produce intuitive sensory feedback was also demonstrated.The possibility to permanently and reliably access implanted electrodes, thus making them viable for prosthetic control, is potentially the main contribution of this work. Furthermore, the opportunity to chronically record and stimulate the neuromuscular system offers new venues for the prediction of complex limb motions and increased understanding of somatosensory perception. Therefore, the technology developed here, combining stable attachment with permanent and reliable human-machine communication, is considered by the author as a critical step towards more functional artificial limbs

Addressing the challenges posed by human machine interfaces based on force sensitive resistors for powered prostheses

Despite the advancements in the mechatronics aspect of prosthetic devices, prostheses control still lacks an interface that satisfies the needs of the majority of users. The research community has put great effort into the advancements of prostheses control techniques to address users’ needs. However, most of these efforts are focused on the development and assessment of technologies in the controlled environments of laboratories. Such findings do not fully transfer to the daily application of prosthetic systems. The objectives of this thesis focus on factors that affect the use of Force Myography (FMG) controlled prostheses in practical scenarios. The first objective of this thesis assessed the use of FMG as an alternative or synergist Human Machine Interface (HMI) to the more traditional HMI, i.e. surface Electromyography (sEMG). The assessment for this study was conducted in conditions that are relatively close to the real use case of prosthetic applications. The HMI was embedded in the custom prosthetic prototype that was developed for the pilot participant of the study using an off-the-shelf prosthetic end effector. Moreover, prostheses control was assessed as the user moved their limb in a dynamic protocol.The results of the aforementioned study motivated the second objective of this thesis: to investigate the possibility of reducing the complexity of high density FMG systems without sacrificing classification accuracies. This was achieved through a design method that uses a high density FMG apparatus and feature selection to determine the number and location of sensors that can be eliminated without significantly sacrificing the system’s performance. The third objective of this thesis investigated two of the factors that contribute to increased errors in force sensitive resistor (FSR) signals used in FMG controlled prostheses: bending of force sensors and variations in the volume of the residual limb. Two studies were conducted that proposed solutions to mitigate the negative impact of these factors. The incorporation of these solutions into prosthetic devices is discussed in these studies.It was demonstrated that FMG is a promising HMI for prostheses control. The facilitation of pattern recognition with FMG showed potential for intuitive prosthetic control. Moreover, a method for the design of a system that can determine the required number of sensors and their locations on each individual to achieve a simpler system with comparable performance to high density FMG systems was proposed and tested. The effects of the two factors considered in the third objective were determined. It was also demonstrated that the proposed solutions in the studies conducted for this objective can be used to increase the accuracy of signals that are commonly used in FMG controlled prostheses

Аналіз методів визначення просторового положення кінцівки пацієнта

У даній статті наведено огляд методів визначення положення верхньої кінцівки людини і їх вплив на класифікацію жестів кисті руки, в результаті якого запропоновано впровадження у вимірювальну систему трьохосьового гіроскопа

Data and Sensor Fusion Using FMG, sEMG and IMU Sensors for Upper Limb Prosthesis Control

Whether someone is born with a missing limb or an amputation occurs later in life, living with this disability can be extremely challenging. The robotic prosthetic devices available today are capable of giving users more functionality, but the methods available to control these prostheses restrict their use to simple actions, and are part of the reason why users often reject prosthetic technologies. Using multiple myography modalities has been a promising approach to address these control limitations; however, only two myography modalities have been rigorously tested so far, and while the results have shown improvements, they have not been robust enough for out-of-lab use. In this work, a novel multi-modal device that allows data to be collected from three myography modalities was created. Force myography (FMG), surface electromyography (sEMG), and inertial measurement unit (IMU) sensors were integrated into a wearable armband and used to collect signal data while subjects performed gestures important for the activities of daily living. An established machine learning algorithm was used to decipher the signals to predict the user\u27s intent/gesture being held, which could be used to control a prosthetic device. Using all three modalities provided statistically-significant improvements over most other modality combinations, as it provided the most accurate and consistent classification results. This work provides justification for using three sensing modalities and future work is suggested to explore this modality combination to decipher more complex actions and tasks with more sophisticated pattern recognition algorithms

Deep Learning Based Upper-limb Motion Estimation Using Surface Electromyography

To advance human-machine interfaces (HMI) that can help disabled people reconstruct lost functions of upper-limbs, machine learning (ML) techniques, particularly classification-based pattern recognition (PR), have been extensively implemented to decode human movement intentions from surface electromyography (sEMG) signals. However, performances of ML can be substantially affected, or even limited, by feature engineering that requires expertise in both domain knowledge and experimental experience. To overcome this limitation, researchers are now focusing on deep learning (DL) techniques to derive informative, representative, and transferable features from raw data automatically. Despite some progress reported in recent literature, it is still very challenging to achieve reliable and robust interpretation of user intentions in practical scenarios. This is mainly because of the high complexity of upper-limb motions and the non-stable characteristics of sEMG signals. Besides, the PR scheme only identifies discrete states of motion. To complete coordinated tasks such as grasping, users have to rely on a sequential on/off control of each individual function, which is inherently different from the simultaneous and proportional control (SPC) strategy adopted by the natural motions of upper-limbs. The aim of this thesis is to develop and advance several DL techniques for the estimation of upper-limb motions from sEMG, and the work is centred on three themes: 1) to improve the reliability of gesture recognition by rejecting uncertain classification outcomes; 2) to build regression frameworks for joint kinematics estimation that enables SPC; and 3) to reduce the degradation of estimation performances when DL model is applied to a new individual. In order to achieve these objectives, the following efforts were made: 1) a confidence model was designed to predict the possibility of correctness with regard to each classification of convolutional neural networks (CNN), such that the uncertain recognition can be identified and rejected; 2) a hybrid framework using CNN for deep feature extraction and long short-term memory neural network (LSTM) was constructed to conduct sequence regression, which could simultaneously exploit the temporal and spatial information in sEMG data; 3) the hybrid framework was further extended by integrating Kalman filter with LSTM units in the recursive learning process, obtaining a deep Kalman filter network (DKFN) to perform kinematics estimation more effectively; and 4) a novel regression scheme was proposed for supervised domain adaptation (SDA), based on which the model generalisation among subjects can be substantially enhanced