Upper Limb Movement Recognition utilising EEG and EMG Signals for Rehabilitative Robotics

Upper limb movement classification, which maps input signals to the target activities, is a key building block in the control of rehabilitative robotics. Classifiers are trained for the rehabilitative system to comprehend the desires of the patient whose upper limbs do not function properly. Electromyography (EMG) signals and Electroencephalography (EEG) signals are used widely for upper limb movement classification. By analysing the classification results of the real-time EEG and EMG signals, the system can understand the intention of the user and predict the events that one would like to carry out. Accordingly, it will provide external help to the user. However, the noise in the real-time EEG and EMG data collection process contaminates the effectiveness of the data, which undermines classification performance. Moreover, not all patients process strong EMG signals due to muscle damage and neuromuscular disorder. To address these issues, this paper explores different feature extraction techniques and machine learning and deep learning models for EEG and EMG signals classification and proposes a novel decision-level multisensor fusion technique to integrate EEG signals with EMG signals. This system retrieves effective information from both sources to understand and predict the desire of the user, and thus aid. By testing out the proposed technique on a publicly available WAY-EEG-GAL dataset, which contains EEG and EMG signals that were recorded simultaneously, we manage to conclude the feasibility and effectiveness of the novel system.Comment: 20 pages, 11 figures, 2 tables; Thesis for Undergraduate Research Project in Computing, NUS; Accepted by Future of Information and Communication Conference 2023, San Francisc

Real‐time and offline evaluation of myoelectric pattern recognition for the decoding of hand movements

Pattern recognition algorithms have been widely used to map surface electromyographic signals to target movements as a source for prosthetic control. However, most investigations have been conducted offline by performing the analysis on pre‐recorded datasets. While real‐time data analysis (i.e., classification when new data becomes available, with limits on latency under 200–300 milliseconds) plays an important role in the control of prosthetics, less knowledge has been gained with respect to real‐time performance. Recent literature has underscored the differences between offline classification accuracy, the most common performance metric, and the usability of upper limb prostheses. Therefore, a comparative offline and real‐time performance analysis between common algorithms had yet to be performed. In this study, we investigated the offline and real‐time performance of nine different classification algorithms, decoding ten individual hand and wrist movements. Surface myoelectric signals were recorded from fifteen able‐bodied subjects while performing the ten movements. The offline decoding demonstrated that linear discriminant analysis (LDA) and maximum likelihood estimation (MLE) significantly (p < 0.05) outperformed other clas-sifiers, with an average classification accuracy of above 97%. On the other hand, the real‐time investigation revealed that, in addition to the LDA and MLE, multilayer perceptron also outperformed the other algorithms and achieved a classification accuracy and completion rate of above 68% and 69%, respectively

Evaluation of surface EMG-based recognition algorithms for decoding hand movements

Myoelectric pattern recognition (MPR) to decode limb movements is an important advancement regarding the control of powered prostheses. However, this technology is not yet in wide clinical use. Improvements in MPR could potentially increase the functionality of powered prostheses. To this purpose, offline accuracy and processing time were measured over 44 features using six classifiers with the aim of determining new configurations of features and classifiers to improve the accuracy and response time of prosthetics control. An efficient feature set (FS: waveform length, correlation coefficient, Hjorth Parameters) was found to improve the motion recognition accuracy. Using the proposed FS significantly increased the performance of linear discriminant analysis, K-nearest neighbor, maximum likelihood estimation (MLE), and support vector machine by 5.5%, 5.7%, 6.3%, and 6.2%, respectively, when compared with the Hudgins\u27 set. Using the FS with MLE provided the largest improvement in offline accuracy over the Hudgins feature set, with minimal effect on the processing time. Among the 44 features tested, logarithmic root mean square and normalized logarithmic energy yielded the highest recognition rates (above 95%). We anticipate that this work will contribute to the development of more accurate surface EMG-based motor decoding systems for the control prosthetic hands

Comparison of sEMG-Based Feature Extraction and Motion Classification Methods for Upper-Limb Movement

The surface electromyography (sEMG) technique is proposed for muscle activation detection and intuitive control of prostheses or robot arms. Motion recognition is widely used to map sEMG signals to the target motions. One of the main factors preventing the implementation of this kind of method for real-time applications is the unsatisfactory motion recognition rate and time consumption. The purpose of this paper is to compare eight combinations of four feature extraction methods (Root Mean Square (RMS), Detrended Fluctuation Analysis (DFA), Weight Peaks (WP), and Muscular Model (MM)) and two classifiers (Neural Networks (NN) and Support Vector Machine (SVM)), for the task of mapping sEMG signals to eight upper-limb motions, to find out the relation between these methods and propose a proper combination to solve this issue. Seven subjects participated in the experiment and six muscles of the upper-limb were selected to record sEMG signals. The experimental results showed that NN classifier obtained the highest recognition accuracy rate (88.7%) during the training process while SVM performed better in real-time experiments (85.9%). For time consumption, SVM took less time than NN during the training process but needed more time for real-time computation. Among the four feature extraction methods, WP had the highest recognition rate for the training process (97.7%) while MM performed the best during real-time tests (94.3%). The combination of MM and NN is recommended for strict real-time applications while a combination of MM and SVM will be more suitable when time consumption is not a key requirement

Classification de mouvements fantômes du membre supérieur chez des amputés huméraux à l'aide de mesures électromyographiques et cinématiques

RÉSUMÉ La perte d’un membre supérieur engendre de nombreux déficits fonctionnels pour l’amputé dans sa vie de tous les jours. En effet, la plupart des activités de la vie quotidienne, telles qu’attacher ses souliers ou ouvrir une bouteille, sont complexes et difficiles à réaliser avec un seul bras fonctionnel. Les impacts de ces déficits augmentent à mesure que le niveau d’amputation est plus haut au niveau du bras. Pour toutes ces personnes, les nombreuses avancées dans le domaine des prothèses myoélectriques, c’est-à-dire commandées par l’activité musculaire des muscles restants après l’amputation, sont encourageantes parce qu’elles permettent d’entretenir l’espoir d’une prothèse à la commande intuitive. Un phénomène particulier, présent chez la majorité des amputés, est celui des sensations au membre fantôme. Ces sensations peuvent se manifester sous plusieurs formes : thermiques, douleurs, mobilités. Les mobilités du membre fantôme sont particulièrement intéressantes pour le développement des prothèses myoélectriques étant donné qu’il a été démontré que les mouvements fantômes produisent une activité électromyographique (EMG) au niveau du membre amputé. Cependant, les études s’intéressant à la détection des mouvements fantômes ont enregistré l’activité EMG provenant de muscles difficilement intégrables dans l’emboiture d’une prothèse myoélectriques, tels que ceux du dos, du torse et de l’épaule. La présente étude se concentre sur la classification des mouvements fantômes chez les amputés huméraux à l’aide de l’EMG dans l’optique de développer une prothèse myoélectrique commandée par reconnaissance de formes. Cinq adultes ayant subi une amputation unilatérale humérale suite à un trauma ont participé à cette étude. L’activité EMG des participants a été enregistrée exclusivement autour de leur moignon. Durant les enregistrements, il était demandé aux participants de réaliser l’un des principaux mouvements fantômes du membre supérieur : la flexion ou l’extension du coude, la pronation ou la supination de l’avant-bras, la flexion ou l’extension du poignet, l’ouverture ou la fermeture de la main et le repos. Chaque mouvement fantôme devait être réalisé symétriquement à l’aide du bras sain et la cinématique de ce dernier a été enregistrée à l’aide d’un système d’analyse du mouvement. Dix caractéristiques (ou « features » en anglais) temporels ont été extraites des signaux EMG et utilisées pour entraîner un réseau de neurones permettant de classifier les mouvements fantômes du membre supérieur.----------ABSTRACT Upper limb amputation creates substantial functional deficits for the amputee. Indeed, most activities of daily living, such as tying shoelaces or opening a bottle, are complex and hard to achieve with only one functional arm. These functional impairments increase as the level of amputation is higher up the arm. For these people, recent advances in the field of myoelectric prostheses, i.e. controlled by the activity of the remaining muscles after amputation, are encouraging because they help maintain the hope of an intuitive prosthesis. A particular phenomenon, occurring in the majority of amputees, is the presence of phantom limb sensations. Phantom limb sensations are of many types: thermal, pain, and mobility. Phantom limb mobilities are particularly interesting for the development of myoelectric prostheses since it has been shown that they produce an electromyographic (EMG) activity in the amputated limb. However, the studies focusing on the detection of phantom movements recorded EMG from muscles that are hard to integrate into the socket element of a myoelectric prosthesis, such as the back, chest and shoulder muscles. This study focuses on the classification of phantom movements in transhumeral amputees using EMG in the context of developing a myoelectric prosthesis controlled by pattern recognition. Five adults who underwent unilateral humeral amputation following a trauma participated in this study. The EMG activity of the participants was recorded exclusively around their stump. During the recordings, participants were asked to perform one of the main upper limb phantom movements: flexion or extension of the elbow, pronation or supination of the forearm, flexion or extension of the wrist, opening or closing the hand and rest. Each phantom movement was to be made symmetrical with the unaffected arm and the kinematics of the latter was recorded using a motion analysis system. Ten time-domain features were extracted from the EMG signals and used to train a neural network to classify the phantom limb movements. The performance of the classifier was evaluated based on the number of movements studied and an optimal set of four EMG features was determined. The impact of kinematic information on the classification performance was also evaluated. The accuracy of the classification varies from one amputee to another, but some trends are common: performance decreases if the number of degrees of freedom considered in the classification increases and/or if the phantom movements become more distal. Moreover, the optimal set of four EMG features provided a performance equivalent to that obtained with all ten EMG features. The addition of the kinematic information improved classification accuracy for all amputees

Novel Bidirectional Body - Machine Interface to Control Upper Limb Prosthesis

Objective. The journey of a bionic prosthetic user is characterized by the opportunities and limitations involved in adopting a device (the prosthesis) that should enable activities of daily living (ADL). Within this context, experiencing a bionic hand as a functional (and, possibly, embodied) limb constitutes the premise for mitigating the risk of its abandonment through the continuous use of the device. To achieve such a result, different aspects must be considered for making the artificial limb an effective support for carrying out ADLs. Among them, intuitive and robust control is fundamental to improving amputees’ quality of life using upper limb prostheses. Still, as artificial proprioception is essential to perceive the prosthesis movement without constant visual attention, a good control framework may not be enough to restore practical functionality to the limb. To overcome this, bidirectional communication between the user and the prosthesis has been recently introduced and is a requirement of utmost importance in developing prosthetic hands. Indeed, closing the control loop between the user and a prosthesis by providing artificial sensory feedback is a fundamental step towards the complete restoration of the lost sensory-motor functions. Within my PhD work, I proposed the development of a more controllable and sensitive human-like hand prosthesis, i.e., the Hannes prosthetic hand, to improve its usability and effectiveness. Approach. To achieve the objectives of this thesis work, I developed a modular and scalable software and firmware architecture to control the Hannes prosthetic multi-Degree of Freedom (DoF) system and to fit all users’ needs (hand aperture, wrist rotation, and wrist flexion in different combinations). On top of this, I developed several Pattern Recognition (PR) algorithms to translate electromyographic (EMG) activity into complex movements. However, stability and repeatability were still unmet requirements in multi-DoF upper limb systems; hence, I started by investigating different strategies to produce a more robust control. To do this, EMG signals were collected from trans-radial amputees using an array of up to six sensors placed over the skin. Secondly, I developed a vibrotactile system to implement haptic feedback to restore proprioception and create a bidirectional connection between the user and the prosthesis. Similarly, I implemented an object stiffness detection to restore tactile sensation able to connect the user with the external word. This closed-loop control between EMG and vibration feedback is essential to implementing a Bidirectional Body - Machine Interface to impact amputees’ daily life strongly. For each of these three activities: (i) implementation of robust pattern recognition control algorithms, (ii) restoration of proprioception, and (iii) restoration of the feeling of the grasped object's stiffness, I performed a study where data from healthy subjects and amputees was collected, in order to demonstrate the efficacy and usability of my implementations. In each study, I evaluated both the algorithms and the subjects’ ability to use the prosthesis by means of the F1Score parameter (offline) and the Target Achievement Control test-TAC (online). With this test, I analyzed the error rate, path efficiency, and time efficiency in completing different tasks. Main results. Among the several tested methods for Pattern Recognition, the Non-Linear Logistic Regression (NLR) resulted to be the best algorithm in terms of F1Score (99%, robustness), whereas the minimum number of electrodes needed for its functioning was determined to be 4 in the conducted offline analyses. Further, I demonstrated that its low computational burden allowed its implementation and integration on a microcontroller running at a sampling frequency of 300Hz (efficiency). Finally, the online implementation allowed the subject to simultaneously control the Hannes prosthesis DoFs, in a bioinspired and human-like way. In addition, I performed further tests with the same NLR-based control by endowing it with closed-loop proprioceptive feedback. In this scenario, the results achieved during the TAC test obtained an error rate of 15% and a path efficiency of 60% in experiments where no sources of information were available (no visual and no audio feedback). Such results demonstrated an improvement in the controllability of the system with an impact on user experience. Significance. The obtained results confirmed the hypothesis of improving robustness and efficiency of a prosthetic control thanks to of the implemented closed-loop approach. The bidirectional communication between the user and the prosthesis is capable to restore the loss of sensory functionality, with promising implications on direct translation in the clinical practice

Modelo adaptativo baseado em sensor virtual para eletromiografia de superfície com sistema de classificação tolerante a falhas

Apenas alguns sistemas de controle protético na literatura científica são baseados em algoritmos de reconhecimento de padrões, os quais são adaptados às mudanças que ocorrem no sinal mioelétrico ao longo do tempo, e, frequentemente, tais sistemas não são naturais e intuitivos. As mudanças no sinal mioelétrico são alguns dos vários desafios para as próteses mioelétricas serem amplamente utilizadas. O conceito do sensor virtual, que tem como objetivo fundamental estimar medidas indisponíveis por trás de outras medidas disponíveis, vem sendo utilizado em outras áreas de pesquisa. O sensor virtual aplicado à eletromiografia de superfície (sEMG) pode ajudar a minimizar esses problemas, tipicamente relacionados à degradação do sinal mioelétrico, os quais geralmente provocam uma diminuição na taxa de acerto da classificação dos movimentos por sistemas de inteligência computacional. A principal contribuição deste trabalho é o desenvolvimento de um sistema de classificação de movimentos tolerante a falhas, o qual utiliza o conceito de sensores virtuais para reduzir o impacto de degradação de sinais de sEMG. A segunda contribuição é um modelo do sinal de sEMG dinâmico e adaptativo para o sensor virtual, o qual produz um modelo de saída de sinal independente da aquisição física do sinal de interesse. A modelagem do sinal de sEMG é projetada de forma a combinar os conceitos de multicanais e sua correlação cruzada, além de utilizar um sistema de ajuste dos coeficientes de correlação, a fim de substituir os canais de sinais degradados Dois modelos são avaliados e detalhados: Time-Varying Autoregressive Moving Average (TVARMA) e o Time- Varying Kalman Filter (TVK). A terceira contribuição é a combinação de uma análise e detecção da contaminação do sinal realizada por um sensor de detecção tolerante a falhas (Sensor Fault-Tolerant Detector – SFTD). Os resultados da classificação dos movimentos foram apresentados comparando as técnicas usuais de classificação com o método da substituição do sinal degradado e um processo de retreinamento do classificador simplificado. Os resultados foram avaliados para cinco tipos de ruído em 16 estudos de caso da degradação dos canais de sEMG. O sistema adaptativo proposto sem o uso de técnicas de retreinamento do classificador recuperou a taxa de acerto média de classificação em até 46% para os ruídos de deslocamento de eletrodos e de saturação. Devido às limitações do sistema proposto quanto aos ruídos de artefato de movimento, de interferência de linha de energia e ECG, o sistema apresentado pode ser utilizado como uma técnica complementar com outras técnicas de classificação para aumentar o impacto clínico da prótese mioelétrica. Entretanto, o sistema ainda requer uma análise quanto a diferentes níveis de SNR antes de uma otimização do algoritmo. Além disso, o modelo TVARMA do sensor virtual obteve uma taxa de acerto média superior em comparação ao modelo TVK na maioria das situações avaliadas neste trabalho.Nowadays, only a few prosthetic control systems in the scientific literature are founded on pattern recognition algorithms adapted to changes that occur in the myoelectric signal over time and, frequently, such systems are not natural and intuitive. These are some of the several challenges for myoelectric prostheses for everyday use. The concept of the virtual sensor, which has as its fundamental objective to estimate unavailable measures based on other available measures, is already being used in other fields of research. The virtual sensor technique applied to surface electromyography (sEMG) can help to mitigate these problems, typically related to the degradation of the myoelectric signal that usually leads to a decrease in the classification accuracy of the movements characterized by intelligent computational systems. Therefore, the main contribution of this work is the Fault-Tolerant Classification System, that was developed using the concept of virtual sensors to reduce the degradation impact of sEMG signals. The second contribution is a dynamic and adaptive virtual sensor model, which produces a signal output model independent of the physical acquisition of the interest signal. The sEMG signal modeling was designed to combine multichannel concepts and their cross-correlation, in addition to the use of the correlation coefficient adjustment system to replace degraded signal channels. Two models were evaluated and detailed: Time-Varying Autoregressive Moving Average (TVARMA) and Time-Varying Kalman Filter (TVK) The third contribution is the analysis and detection of signal contamination by a Sensor Fault-Tolerant Detector (SFTD). The classification results of the movements were compared to the traditional classification techniques, the classification with the degraded signal replacement method and a simplified retraining process of the classifier. The results were evaluated for five noise types in 16 case studies of the sEMG channels degradation. The adaptive system proposed, without the classifier re-training techniques, was able to recover 46% of the mean classification accuracy for the electrodes displacement and saturation noise. Moreover, the proposed system can be used as a complementary technique with other classification techniques to increase the clinical impact of the myoelectric prosthesis since there are still limitations in the proposed method regarding the movement artifact noise, power line, and ECG interference. However, the system still requires an analysis of different SNR levels before the algorithm optimization. Also, the TVARMA model of the virtual sensor obtained a higher classification accuracy compared to the TVK model in most of the evaluated situations