On the Utility of Representation Learning Algorithms for Myoelectric Interfacing

Electrical activity produced by muscles during voluntary movement is a reflection of the firing patterns of relevant motor neurons and, by extension, the latent motor intent driving the movement. Once transduced via electromyography (EMG) and converted into digital form, this activity can be processed to provide an estimate of the original motor intent and is as such a feasible basis for non-invasive efferent neural interfacing. EMG-based motor intent decoding has so far received the most attention in the field of upper-limb prosthetics, where alternative means of interfacing are scarce and the utility of better control apparent. Whereas myoelectric prostheses have been available since the 1960s, available EMG control interfaces still lag behind the mechanical capabilities of the artificial limbs they are intended to steer—a gap at least partially due to limitations in current methods for translating EMG into appropriate motion commands. As the relationship between EMG signals and concurrent effector kinematics is highly non-linear and apparently stochastic, finding ways to accurately extract and combine relevant information from across electrode sites is still an active area of inquiry.This dissertation comprises an introduction and eight papers that explore issues afflicting the status quo of myoelectric decoding and possible solutions, all related through their use of learning algorithms and deep Artificial Neural Network (ANN) models. Paper I presents a Convolutional Neural Network (CNN) for multi-label movement decoding of high-density surface EMG (HD-sEMG) signals. Inspired by the successful use of CNNs in Paper I and the work of others, Paper II presents a method for automatic design of CNN architectures for use in myocontrol. Paper III introduces an ANN architecture with an appertaining training framework from which simultaneous and proportional control emerges. Paper Iv introduce a dataset of HD-sEMG signals for use with learning algorithms. Paper v applies a Recurrent Neural Network (RNN) model to decode finger forces from intramuscular EMG. Paper vI introduces a Transformer model for myoelectric interfacing that do not need additional training data to function with previously unseen users. Paper vII compares the performance of a Long Short-Term Memory (LSTM) network to that of classical pattern recognition algorithms. Lastly, paper vIII describes a framework for synthesizing EMG from multi-articulate gestures intended to reduce training burden

AN INVESTIGATION OF ELECTROMYOGRAPHIC (EMG) CONTROL OF DEXTROUS HAND PROSTHESES FOR TRANSRADIAL AMPUTEES

In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Plymouth University's products or services.There are many amputees around the world who have lost a limb through conflict, disease or an accident. Upper-limb prostheses controlled using surface Electromyography (sEMG) offer a solution to help the amputees; however, their functionality is limited by the small number of movements they can perform and their slow reaction times. Pattern recognition (PR)-based EMG control has been proposed to improve the functional performance of prostheses. It is a very promising approach, offering intuitive control, fast reaction times and the ability to control a large number of degrees of freedom (DOF). However, prostheses controlled with PR systems are not available for everyday use by amputees, because there are many major challenges and practical problems that need to be addressed before clinical implementation is possible. These include lack of individual finger control, an impractically large number of EMG electrodes, and the lack of deployment protocols for EMG electrodes site selection and movement optimisation. Moreover, the inability of PR systems to handle multiple forces is a further practical problem that needs to be addressed. The main aim of this project is to investigate the research challenges mentioned above via non-invasive EMG signal acquisition, and to propose practical solutions to help amputees. In a series of experiments, the PR systems presented here were tested with EMG signals acquired from seven transradial amputees, which is unique to this project. Previous studies have been conducted using non-amputees. In this work, the challenges described are addressed and a new protocol is proposed that delivers a fast clinical deployment of multi-functional upper limb prostheses controlled by PR systems. Controlling finger movement is a step towards the restoration of lost human capabilities, and is psychologically important, as well as physically. A central thread running through this work is the assertion that no two amputees are the same, each suffering different injuries and retaining differing nerve and muscle structures. This work is very much about individualised healthcare, and aims to provide the best possible solution for each affected individual on a case-by-case basis. Therefore, the approach has been to optimise the solution (in terms of function and reliability) for each individual, as opposed to developing a generic solution, where performance is optimised against a test population. This work is unique, in that it contributes to improving the quality of life for each individual amputee by optimising function and reliability. The main four contributions of the thesis are as follows: 1- Individual finger control was achieved with high accuracy for a large number of finger movements, using six optimally placed sEMG channels. This was validated on EMG signals for ten non-amputee and six amputee subjects. Thumb movements were classified successfully with high accuracy for the first time. The outcome of this investigation will help to add more movements to the prosthesis, and reduce hardware and computational complexity. 2- A new subject-specific protocol for sEMG site selection and reliable movement subset optimisation, based on the amputee’s needs, has been proposed and validated on seven amputees. This protocol will help clinicians to perform an efficient and fast deployment of prostheses, by finding the optimal number and locations of EMG channels. It will also find a reliable subset of movements that can be achieved with high performance. 3- The relationship between the force of contraction and the statistics of EMG signals has been investigated, utilising an experimental design where visual feedback from a Myoelectric Control Interface (MCI) helped the participants to produce the correct level of force. Kurtosis values were found to decrease monotonically when the contraction level increased, thus indicating that kurtosis can be used to distinguish different forces of contractions. 4- The real practical problem of the degradation of classification performance as a result of the variation of force levels during daily use of the prosthesis has been investigated, and solved by proposing a training approach and the use of a robust feature extraction method, based on the spectrum. The recommendations of this investigation improve the practical robustness of prostheses controlled with PR systems and progress a step further towards clinical implementation and improving the quality of life of amputees. The project showed that PR systems achieved a reliable performance for a large number of amputees, taking into account real life issues such as individual finger control for high dexterity, the effect of force level variation, and optimisation of the movements and EMG channels for each individual amputee. The findings of this thesis showed that the PR systems need to be appropriately tuned before usage, such as training with multiple forces to help to reduce the effect of force variation, aiming to improve practical robustness, and also finding the optimal EMG channel for each amputee, to improve the PR system’s performance. The outcome of this research enables the implementation of PR systems in real prostheses that can be used by amputees.Ministry of Higher Education and Scientific Research and Baghdad University- Baghdad/Ira

Development and optimization of a low-cost myoelectric upper limb prosthesis

Tese de Mestrado Integrado, Engenharia Biomédica e Biofísica (Engenharia Clínica e Instrumentação Médica), 2022, Universidade de Lisboa, Faculdade de CiênciasIn recent years, the increase in the number of accidents, chronic diseases, such as diabetes, and the impoverishment of certain developing countries have contributed to a significant increase in prostheses users. The loss of a particular limb entails numerous changes in the daily life of each user, which are amplified when the user loses their hand. Therefore, replacing the hand is an urgent necessity. Developing upper limb prostheses will allow the re-establishment of the physical and motor functions of the upper limb as well as reduction of the rates of depression. Therefore, the prosthetic industry has been reinventing itself and evolving. It is already possible to control a prosthesis through the user's myoelectric signals, control known as pattern recognition control. In addition, additive manufacturing technologies such as 3D printing have gained strength in prosthetics. The use of this type of technology allows the product to reach the user much faster and reduces the weight of the devices, making them lighter. Despite these advances, the rejection rate of this type of device is still high since most prostheses available on the market are slow, expensive and heavy. Because of that, academia and institutions have been investigating ways to overcome these limitations. Nevertheless, the dependence on the number of acquisition channels is still limiting since most users do not have a large available forearm surface area to acquire the user’s myoelectric signals. This work intends to solve some of these problems and answer the questions imposed by the industry and researchers. The main objective is to test if developing a subject independent, fast and simple microcontroller is possible. Subsequently, we recorded data from forty volunteers through the BIOPAC acquisition system. After that, the signals were filtered through two different processes. The first was digital filtering and the application of wavelet threshold noise reduction. Later, the signal was divided into smaller windows (100 and 250 milliseconds) and thirteen features were extracted in the temporal domain. During all these steps, the MatLab® software was used. After extraction, three feature selection methods were used to optimize the classification process, where machine learning algorithms are implemented. The classification was divided into different parts. First, the classifier had to distinguish whether the volunteer was making some movement or was at rest. In the case of detected movement, the classifier would have to, on a second level, try to understand if they were moving only one finger or performing a movement that involved the flexion of more than one finger (grip). If the volunteer was performing a grip on the third level, the classifier would have to identify whether the volunteer was performing a spherical or triad grip. Finally, to understand the influence of the database on the classification, two methods were used: cross-validation and split validation. After analysing the results, the e-NABLE Unlimbited arm was printed on The Original Prusa i3 MK3, where polylactic acid (PLA) was used. This dissertation showed that the results obtained in the 250-millisecond window were better than the obtained ones in a 100-millisecond window. In general, the best classifier was the K-Nearest Neighbours (KNN) with k=2, except for the first level that was LDA. The best results were obtained for the first classification level, with an accuracy greater than 90%. Although the results obtained for the second and third levels were close to 80%, it was concluded that it was impossible to develop a microcontroller dependent only on one acquisition channel. These results agree with the anatomical characteristics since they are originated from the same muscle group. The cross-validation results were lower than those obtained in the training-test methodology, which allowed us to conclude that the inter variability that exists between the subjects significantly affects the classification performance. Furthermore, both the dominant and non-dominant arms were used in this work, which also increased the discrepancy between signals. Indeed, the results showed that it is impossible to develop a microcontroller adaptable to all users. Therefore, in the future, the best path will be to opt for the customization of the prototype. In order to test the implementation of a microcontroller in the printed model, it was necessary to design a support structure in Solidworks that would support the motors used to flex the fingers and Arduino to control the motors. Consequently, the e-NABLE model was re adapted, making it possible to develop a clinical training prototype. Even though it is a training prototype, it is lighter than those on the market and cheaper. The objectives of this work have been fulfilled and many answers have been given. However, there is always space for improvement. Although, this dissertation has some limitations, it certainly contributed to clarify many of the doubts that still exist in the scientific community. Hopefully, it will help to further develop the prosthetic industry.Nos últimos anos, o aumento do número de acidentes por doenças crónicas, como, por exemplo, a diabetes, e o empobrecimento de determinados países em desenvolvimento têm contribuído para um aumento significativo no número de utilizadores de próteses. A perda de um determinado membro acarreta inúmeras mudanças no dia-a-dia de cada utilizador. Estas são amplificadas quando a perda é referente à mão ou parte do antebraço. A mão é uma ferramenta essencial no dia-a-dia de cada ser humano, uma vez que é através dela que são realizadas as atividades básicas, como, por exemplo, tomar banho, lavar os dentes, comer, preparar refeições, etc. A substituição desta ferramenta é, portanto, uma necessidade, não só porque permitirá restabelecer as funções físicas e motoras do membro superior, como, também, reduzirá o nível de dependência destes utilizadores de outrem e, consequentemente, das taxas de depressão. Para colmatar as necessidades dos utilizadores, a indústria prostética tem-se reinventado e evoluído, desenvolvendo próteses para o membro superior cada vez mais sofisticadas. Com efeito, já é possível controlar uma prótese através da leitura e análise dos sinais mioelétricos do próprio utilizador, o que é denominado por muitos investigadores de controlo por reconhecimento de padrões. Este tipo de controlo é personalizável e permite adaptar a prótese a cada utilizador. Para além do uso de sinais elétricos provenientes do musculo do utilizador, a impressão 3D, uma técnica de manufatura aditiva, têm ganho força no campo da prostética. Por conseguinte, nos últimos anos os investigadores têm impresso inúmeros modelos com diferentes materiais que vão desde o uso de termoplásticos, ao uso de materiais flexíveis. A utilização deste tipo de tecnologia permite, para além de uma rápida entrega do produto ao utilizador, uma diminuição no tempo de construção de uma prótese tornando-a mais leve e barata. Além do mais, a impressão 3D permite criar protótipos mais sustentáveis, uma vez que existe uma redução na quantidade de material desperdiçado. Embora já existam inúmeras soluções, a taxa de rejeição deste tipo de dispositivos é ainda bastante elevada, uma vez que a maioria das próteses disponíveis no mercado, nomeadamente as mioelétricas, são lentas, caras e pesadas. Ainda que existam alguns estudos que se debrucem neste tipo de tecnologias, bem como na sua evolução científica, o número de elétrodos utilizados é ainda significativo. Desta forma, e, tendo em conta que a maioria dos utilizadores não possuí uma área de superfície do antebraço suficiente para ser feita a aquisição dos sinais mioelétricos, o trabalho feito pela academia não se revelou tão contributivo para a indústria prostética como este prometia inicialmente. Este trabalho pretende resolver alguns desses problemas e responder às questões mais impostas pela indústria e investigadores, para que, no futuro, o número de utilizadores possa aumentar, assim como o seu índice de satisfação relativamente ao produto. Para tal, recolheram-se os sinais mioelétricos de quarenta voluntários, através do sistema de aquisição BIOPAC. Após a recolha, filtraram-se os sinais de seis voluntários através de dois processos diferentes. No primeiro, utilizaram-se filtros digitais e no segundo aplicou-se a transformada de onda para a redução do ruído. De seguida, o sinal foi segmentado em janelas mais pequenas de 100 e 250 milissegundos e extraíram-se treze features no domínio temporal. Para que o processo de classificação fosse otimizado, foram aplicados três métodos de seleção de features. A classificação foi dividida em três níveis diferentes nos quais dois algoritmos de aprendizagem automática foram implementados, individualmente. No primeiro nível, o objetivo foi a distinção entre os momentos em que o voluntário fazia movimento ou que estava em repouso. Caso o output do classificador fosse a classe movimento, este teria de, num segundo nível, tentar perceber se o voluntário estaria a mexer apenas um dedo ou a realizar um movimento que envolvesse a flexão de mais de que um dedo (preensão). No caso de uma preensão, passava-se ao terceiro nível onde o classificador teria de identificar se o voluntário estaria a realizar a preensão esférica ou em tríade. Para todos os níveis de classificação, obtiveram-se resultados para o método de validação cruzada e o método de teste e treino, sendo que neste, 70% dos dados foram utilizados como conjunto de treino e 30% como teste. Efetuada a análise dos resultados, escolheu-se um dos modelos da comunidade e-NABLE. O modelo foi impresso na impressora The Original Prusa i3 MK3S e o material escolhido foi o ácido poliláctico (PLA). Para que fosse possível testar a implementação de um microcontrolador num modelo que originalmente depende da flexão do cotovelo realizada pelo utilizador, foi necessário desenhar uma estrutura de suporte que suportasse, não só os motores utilizados para flexionar os dedos, como, também, o Arduíno. O suporte desenhado foi impresso com o mesmo material e com a mesma impressora. Os resultados obtidos mostraram que a janela de 250 milissegundo foi a melhor e que, regra geral, o melhor classificador é o K-Nearest Neighbors (KNN) com k=2, com exceção do primeiro nível, em que o melhor classificador foi o Linear Discriminant Analysis (LDA). Os melhores resultados obtiveram-se no primeiro nível de classificação onde a accuracy foi superior a 90%. Embora os resultados obtidos para o segundo e terceiro nível tenham sido próximos de 80%, concluiu-se que não era possível desenvolver um microcontrolador dependente apenas de um canal de aquisição. Tal era expectável, uma vez que os movimentos estudados são originados pelo mesmo grupo muscular e a intervariabilidade dos sujeitos um fator significativo. Os resultados da validação cruzada foram menos precisos do que os obtidos para a metodologia de treino-teste, o que permitiu concluir que a intervariabilidade existente entre os voluntários afeta significativamente o processo de classificação. Para além disso, os voluntários utilizaram o braço dominante e o braço não dominante, o que acabou por aumentar a discrepância entre os sinais recolhidos. Com efeito, os resultados mostraram que não é possível desenvolver um microcontrolador que seja adaptável a todos os utilizadores e, portanto, no futuro, o melhor caminho será optar pela personalização do protótipo. Tendo o conhecimento prévio desta evidência, o protótipo desenvolvido neste trabalho apenas servirá como protótipo de treino para o utilizador. Ainda assim, este é bem mais leve que os existentes no mercado e muito mais barato. Nele é ainda possível testar e controlar alguns dos componentes que no futuro irão fazer parte da prótese completa, prevenindo acidentes. Não obstante o cumprimento dos objetivos deste trabalho e das muitas respostas que por ele foram dadas, existe sempre espaço para melhorias. Dado à limitação de tempo, não foi possível testar o microcontrolador em tempo-real nem efetuar testes mecânicos de flexibilidade e resistência dos materiais da prótese. Deste modo, seria interessante no futuro fazer testes de performance em tempo real e submeter a prótese a condições extremas, para que a tensão elástica e a tensão dos pins sejam testadas. Para além disso, testar os mecanismos de segurança da prótese quando o utilizador tem de fazer muita força é fundamental. O teste destes parâmetros evitará a ocorrência de falhas que poderão magoar o utilizador, bem como estragar os objetos com os quais a prótese poderá interagir. Por fim, é necessário melhorar o aspeto cosmético das próteses. Para que isso aconteça, poderão ser utilizados polímeros com uma coloração próxima do tom da pele do utilizador. Uma outra forma de melhorar este aspeto, seria fazer o scanning do braço saudável do utilizador e usar materiais flexíveis para as articulações e dedos que, juntamente com uma palma de termoplásticos resistentes e um microcontrolador, permitissem um movimento bastante natural próximo do biológico. Em suma, apesar de algumas limitações, este trabalho contribuiu para o esclarecimento de muitas das dúvidas que ainda existiam na comunidade científica e ajudará a desenvolver a indústria prostética

Predicting Continuous Locomotion Modes via Multidimensional Feature Learning from sEMG

Walking-assistive devices require adaptive control methods to ensure smooth transitions between various modes of locomotion. For this purpose, detecting human locomotion modes (e.g., level walking or stair ascent) in advance is crucial for improving the intelligence and transparency of such robotic systems. This study proposes Deep-STF, a unified end-to-end deep learning model designed for integrated feature extraction in spatial, temporal, and frequency dimensions from surface electromyography (sEMG) signals. Our model enables accurate and robust continuous prediction of nine locomotion modes and 15 transitions at varying prediction time intervals, ranging from 100 to 500 ms. In addition, we introduced the concept of 'stable prediction time' as a distinct metric to quantify prediction efficiency. This term refers to the duration during which consistent and accurate predictions of mode transitions are made, measured from the time of the fifth correct prediction to the occurrence of the critical event leading to the task transition. This distinction between stable prediction time and prediction time is vital as it underscores our focus on the precision and reliability of mode transition predictions. Experimental results showcased Deep-STP's cutting-edge prediction performance across diverse locomotion modes and transitions, relying solely on sEMG data. When forecasting 100 ms ahead, Deep-STF surpassed CNN and other machine learning techniques, achieving an outstanding average prediction accuracy of 96.48%. Even with an extended 500 ms prediction horizon, accuracy only marginally decreased to 93.00%. The averaged stable prediction times for detecting next upcoming transitions spanned from 28.15 to 372.21 ms across the 100-500 ms time advances.Comment: 10 pages,7 figure

Stationary Wavelet Processing and Data Imputing in Myoelectric Pattern Recognition on a Low-Cost Embedded System

Pattern recognition-based decoding of surface electromyography allows for intuitive and flexible control of prostheses but comes at the cost of sensitivity to in-band noise and sensor faults. System robustness can be improved with wavelet-based signal processing and data imputing, but no attempt has been made to implement such algorithms on real-time, portable systems. The aim of this work was to investigate the feasibility of low-latency, wavelet-based processing and data imputing on an embedded device capable of controlling upper-arm prostheses. Nine able-bodied subjects performed Motion Tests while inducing transient disturbances. Additional investigation was performed on pre-recorded Motion Tests from 15 able-bodied subjects with simulated disturbances. Results from real-time tests were inconclusive, likely due to the low number of disturbance episodes, but simulated tests showed significant improvements in most metrics for both algorithms. However, both algorithms also showed reduced responsiveness during disturbance episodes. These results suggest wavelet-based processing and data imputing can be implemented in portable, real-time systems to potentially improve robustness to signal distortion in prosthetic devices with the caveat of reduced responsiveness for the typically short duration of signal disturbances. The trade-off between large-scale signal corruption robustness and system responsiveness warrants further studies in daily life activities

Virtual sensor of surface electromyography in a new extensive fault-tolerant classification system

A few prosthetic control systems in the scientific literature obtain pattern recognition algorithms adapted to changes that occur in the myoelectric signal over time and, frequently, such systems are not natural and intuitive. These are some of the several challenges for myoelectric prostheses for everyday use. The concept of the virtual sensor, which has as its fundamental objective to estimate unavailable measures based on other available measures, is being used in other fields of research. The virtual sensor technique applied to surface electromyography can help to minimize these problems, typically related to the degradation of the myoelectric signal that usually leads to a decrease in the classification accuracy of the movements characterized by computational intelligent systems. This paper presents a virtual sensor in a new extensive fault-tolerant classification system to maintain the classification accuracy after the occurrence of the following contaminants: ECG interference, electrode displacement, movement artifacts, power line interference, and saturation. The Time-Varying Autoregressive Moving Average (TVARMA) and Time-Varying Kalman filter (TVK) models are compared to define the most robust model for the virtual sensor. Results of movement classification were presented comparing the usual classification techniques with the method of the degraded signal replacement and classifier retraining The experimental results were evaluated for these five noise types in 16 surface electromyography (sEMG) channel degradation case studies. The proposed system without using classifier retraining techniques recovered of mean classification accuracy was of 4% to 38% for electrode displacement, movement artifacts, and saturation noise. The best mean classification considering all signal contaminants and channel combinations evaluated was the classification using the retraining method, replacing the degraded channel by the virtual sensor TVARMA model. This method recovered the classification accuracy after the degradations, reaching an average of 5.7% below the classification of the clean signal, that is the signal without the contaminants or the original signal. Moreover, the proposed intelligent technique minimizes the impact of the motion classification caused by signal contamination related to degrading events over time. There are improvements in the virtual sensor model and in the algorithm optimization that need further development to provide an increase the clinical application of myoelectric prostheses but already presents robust results to enable research with virtual sensors on biological signs with stochastic behavior

Stationary wavelet processing and data imputing in myoelectric pattern recognition on a low-cost embedded system

Pattern recognition-based decoding of surface electromyography allows for intuitive and flexible control of prostheses but comes at the cost of sensitivity to in-band noise and sensor faults. System robustness can be improved with wavelet-based signal processing and data imputing, but no attempt has been made to implement such algorithms on real-time, portable systems. The aim of this work was to investigate the feasibility of low-latency, wavelet-based processing and data imputing on an embedded device capable of controlling upper-arm prostheses. Nine able-bodied subjects performed Motion Tests while inducing transient disturbances. Additional investigation was performed on pre-recorded Motion Tests from 15 able-bodied subjects with simulated disturbances. Results from real-time tests were inconclusive, likely due to the low number of disturbance episodes, but simulated tests showed significant improvements in most metrics for both algorithms. However, both algorithms also showed reduced responsiveness during disturbance episodes. These results suggest wavelet-based processing and data imputing can be implemented in portable, real-time systems to potentially improve robustness to signal distortion in prosthetic devices with the caveat of reduced responsiveness for the typically short duration of signal disturbances. The trade-off between large-scale signal corruption robustness and system responsiveness warrants further studies in daily life activities