sEMG-based hand gesture recognition with deep learning

Hand gesture recognition based on surface electromyographic (sEMG) signals is a promising approach for the development of Human-Machine Interfaces (HMIs) with a natural control, such as intuitive robot interfaces or poly-articulated prostheses. However, real-world applications are limited by reliability problems due to motion artifacts, postural and temporal variability, and sensor re-positioning. This master thesis is the first application of deep learning on the Unibo-INAIL dataset, the first public sEMG dataset exploring the variability between subjects, sessions and arm postures, by collecting data over 8 sessions of each of 7 able-bodied subjects executing 6 hand gestures in 4 arm postures. In the most recent studies, the variability is addressed with training strategies based on training set composition, which improve inter-posture and inter-day generalization of classical (i.e. non-deep) machine learning classifiers, among which the RBF-kernel SVM yields the highest accuracy. The deep architecture realized in this work is a 1d-CNN implemented in Pytorch, inspired by a 2d-CNN reported to perform well on other public benchmark databases. On this 1d-CNN, various training strategies based on training set composition were implemented and tested. Multi-session training proves to yield higher inter-session validation accuracies than single-session training. Two-posture training proves to be the best postural training (proving the benefit of training on more than one posture), and yields 81.2% inter-posture test accuracy. Five-day training proves to be the best multi-day training, and yields 75.9% inter-day test accuracy. All results are close to the baseline. Moreover, the results of multi-day trainings highlight the phenomenon of user adaptation, indicating that training should also prioritize recent data. Though not better than the baseline, the achieved classification accuracies rightfully place the 1d-CNN among the candidates for further research

Temporal Variability Analysis in sEMG Hand Grasp Recognition using Temporal Convolutional Networks

Hand movement recognition via surface electromyographic (sEMG) signal is a promising approach for the advance in Human-Computer Interaction. However, this field has to deal with two main issues: (1) the long-term reliability of sEMG-based control is limited by the variability affecting the sEMG signal (especially, variability over time); (2) the classification algorithms need to be suitable for implementation on embedded devices, which have strict constraints in terms of power budget and computational resources. Current solutions present a performance over-time drop that makes them unsuitable for reliable gesture controller design. In this paper, we address temporal variability of sEMG-based grasp recognition, proposing a new approach based on Temporal Convolutional Networks, a class of deep learning algorithms particularly suited for time series analysis and temporal pattern recognition. Our approach improves by 7.6% the best results achieved in the literature on the NinaPro DB6, a reference dataset for temporal variability analysis of sEMG. Moreover, when targeting the much more challenging inter-session accuracy objective, our method achieves an accuracy drop of just 4.8% between intra- and inter-session validation. This proves the suitability of our setup for a robust, reliable long-term implementation. Furthermore, we distill the network using deep network quantization and pruning techniques, demonstrating that our approach can use down to 120x lower memory footprint than the initial network and 4x lower memory footprint than a baseline Support Vector Machine, with an inter-session accuracy degradation of only 2.5%, proving that the solution is suitable for embedded resource-constrained implementations

Neuromuscular Control Modeling: from Physics to Data-Driven Approaches

Il controllo neurale della postura umana è stato investigato a partire da un punto di vista fisico. Il paradigma di controllo intermittente è stato usato allo scopo di capire il peso di quest'ultimo nella generazione delle traiettorie del centro di pressione. Un primo contributo di questo lavoro rigurda quindi l'analisi del centro di pressione generato dal suddetto modello biomeccanico attraverso l'extended detrended fluctuation analysis, recentemente proposta in letteratura. Le proprietà di correlazione a lungo termine e disomogeneità sono risultate strettamente legate al guadagno derivativo del modello di controllo intermittente e anche al grado di intermittenza. Il paradigma di controllo è stato poi esteso verso un sistema biomeccanico più complesso, cioè un pendolo inverso doppio link con controllo intermittente alla caviglia. I contributi più significativi hanno riguardato la modellazione matematica del centro di pressione per una struttura multi-link e la verifica della sua plausibilità fisiologica. Si è poi preso in considerazione il caso della postura perturbata, integrando aspetti cinematici, dinamici e relativi all'attività muscolare. A tal fine, si è utilizzato sia un approccio fisico che basato su dati per l'identificazione dei modelli a struttura variabile Si sono prese in considerazione differenti condizioni di sperimentali, e in tutti i casi l'approccio utilizzato ha garantito un adeguato grado di interpretabilità riguardo il ruolo del sistema nervoso centrale nella regolazione del postura eretta in condizioni perturbate. La seconda parte della tesi ha riguardato la caratterizzazione del controllo motorio attraverso il segnale elettromiografico di superfice. Il primo contributo ha riguardato l'identifcazione dell'onset muscolare in condizioni di basso rapporto segnale rumore, sfruttando operatori energetico di tipo Teager-Kaiser al fine del precondizionamento del segnale mioelettrico. La versioe estesa di questo tipo di operatori è risultata particolarmente utile al miglioramento delle performance di numerosi algoritmi di detection. Si è poi proseguito con l'utilizzo di tali segnali al fine della classificazione dei gesti dell'arto superiore. In particoalre si è prerso in considerazione il problema della motion intention detection dei principali movimenti della spalla , utilizzando sia descrittori del segnale elettromiografico nel dominio del tempo e della frequenza. Quest'ultimo aspetto risulta essere un elemento di novità nel contesto scientifico in quanto si sono considerati il riconoscimento l'intezioni di movimento di otto gesti della spalla con particolare attenzione al ruolo dei descrittori del segnale per la classificazione. Infine, con approcci simili, si è preso in considerazione il problema del riconoscimento della scrittura manuale a partire dal dato elettromiografico. Tale aspetto risulta poco investigato sotto la prospettiva della pattern recognition mioelettrica, ma la sua valenza è data dalla crescente richiesta di interfacce uomo-macchina per compiti riabilitativi che coinvolgono una componente cognitiva significativa, Inoltre, vista la tendenza ad investigare il ruolo del polso per il prelievo del segnale elettromiografico al fine della realizzazione delle suddette interfacce, si è analizzato l'utilizzo dei segnali elettromiografici del polso rispetto a quelli dell'avambraccio al fine di predirre le cifre scritte dall'utente, noto che l'avambraccio risulta essere la zona di prelievo più comunemente utilizzata.The biomechanics and the neural control of the human stance was investigated starting from a physical point of view. In particular the intermittent motor control paradigm was investigated with the aim of understanding how such paradigm mirrors in the center of pressure (COP) trajectories. A first contribution given in this work of thesis regards the analysis of COP generated from intermittent controlled inverted pendulum through the extended detrended fluctuation analysis, which was recently introduced in the literature. It has been found that the long-term correlation and inohmogeniety properties of the COP time series strictly depend on the derivative gain term of the intermittent controller and on the degree of intermittency of the control action. Thus, , another contribution provided in this work of thesis regards the use of a more complex biomechanical model of the stance, e.g. a double-link inverted pendulum intermittently controlled at the ankle. In terms of novelty, it deserves to be pointed out the results regarding the mechanical modeling of the COP for a multi-link structure, and the assessment of its physiological plausibility. . On the other hand, when the perturbed posture motor task was taken into account, there was the need to enlarge the perspective, integrating kinematic, dynamic and muscle activity data. The idea of employing different sources of information to develop models of the CNS represents an important element that was investigated using tools related to hybrid system identification theory. Subjects underwent to impulsive support base translations in three different conditions: considering eyes open, closed, and performing mental counting. Although such data were essentially analyzed through a data-driven approach, the identified models guaranteed physical interpretations of the role played by the CNS in the three different conditions. The second main core of this thesis regards the characterization of the motor control using the surface electromyographic (sEMG) signals. A first contribution given in this work regarded the muscle onset detection considering low SNR scenarios. In this framework, energy operators such as the Teager-Kaiser energy operator (TKEO) and its extended version (ETKEO) were investigated as signal preconditioning steps before the application of state of the art onset detection algorithms. The latter have been significantly boosted when ETKEO was used with respect to TKEO. The use of extended energy operators for the sEMG signal preprocessing constitutes a novel element in this field that can be also further investigated in future studies. From the sEMG muscle, one can also predict which movement the subject is going to perform. This aspect can be enclosed in the motion intention detection (MID) field. In this thesis a MID problem was investigated by taking into account two important aspects: as first the study was centered on the shoulder joint movements. Secondly, the MID problem was faced under a pattern recognition perspective. This allowed to verify whether methodologies encountered in the myoelectric hand gesture recognition can be transferred in the affine field of MID In contrast to what reported in the literature, where MID problems generally consider only few movements, in this work of thesis up to eight shoulder movements have been investigated. Myoelectric pattern recognition architectures were also used in the assessment of the ten hand-written digits. Despite the handwriting can be considered a hand movement that involves fine muscular control actions, it has not been consistently investigated in the field of sEMG based hand gesture recognition. Further, since the literature supports the change from forearm to wrist in order to acquire EMG data for hand gesture recognition, it was investigated whether such exchange can be performed when a challenging classification task, as the handwriting recognition has to be performed

Recent Advances in Motion Analysis

The advances in the technology and methodology for human movement capture and analysis over the last decade have been remarkable. Besides acknowledged approaches for kinematic, dynamic, and electromyographic (EMG) analysis carried out in the laboratory, more recently developed devices, such as wearables, inertial measurement units, ambient sensors, and cameras or depth sensors, have been adopted on a wide scale. Furthermore, computational intelligence (CI) methods, such as artificial neural networks, have recently emerged as promising tools for the development and application of intelligent systems in motion analysis. Thus, the synergy of classic instrumentation and novel smart devices and techniques has created unique capabilities in the continuous monitoring of motor behaviors in different fields, such as clinics, sports, and ergonomics. However, real-time sensing, signal processing, human activity recognition, and characterization and interpretation of motion metrics and behaviors from sensor data still representing a challenging problem not only in laboratories but also at home and in the community. This book addresses open research issues related to the improvement of classic approaches and the development of novel technologies and techniques in the domain of motion analysis in all the various fields of application

Biceps brachii synergy and its contribution to target reaching tasks within a virtual cube

Ces dernières années, des travaux importants ont été observés dans le développement du contrôle prothétique afin d'aider les personnes amputées du membre supérieur à améliorer leur qualité de vie au quotidien. Certaines prothèses myoélectriques modernes des membres supérieurs disponibles dans le commerce ont de nombreux degrés de liberté et nécessitent de nombreux signaux de contrôle pour réaliser plusieurs tâches fréquemment utilisées dans la vie quotidienne. Pour obtenir plusieurs signaux de contrôle, de nombreux muscles sont requis mais pour les personnes ayant subi une amputation du membre supérieur, le nombre de muscles disponibles est plus ou moins réduit selon le niveau de l’amputation. Pour accroître le nombre de signaux de contrôle, nous nous sommes intéressés au biceps brachial, vu qu’anatomiquement il est formé de 2 chefs et que de la présence de compartiments a été observée sur sa face interne. Physiologiquement, il a été trouvé que les unités motrices du biceps sont activées à différents endroits du muscle lors de la production de diverses tâches fonctionnelles. De plus, il semblerait que le système nerveux central puisse se servir de la synergie musculaire pour arriver à facilement produire plusieurs mouvements. Dans un premier temps on a donc identifié que la synergie musculaire était présente chez le biceps de sujets normaux et on a montré que les caractéristiques de cette synergie permettaient d’identifier la posture statique de la main lorsque les signaux du biceps avaient été enregistrés. Dans un deuxième temps, on a réussi à démontrer qu’il était possible, dans un cube présenté sur écran, à contrôler la position d’une sphère en vue d’atteindre diverses cibles en utilisant la synergie musculaire du biceps. Les techniques de classification utilisées pourraient servir à faciliter le contrôle des prothèses myoélectriques.In recent years, important work has been done in the development of prosthetic control to help upper limb amputees improve their quality of life on a daily basis. Some modern commercially available upper limb myoelectric prostheses have many degrees of freedom and require many control signals to perform several tasks commonly used in everyday life. To obtain several control signals, many muscles are required, but for people with upper limb amputation, the number of muscles available is more or less reduced, depending on the level of amputation. To increase the number of control signals, we were interested in the biceps brachii, since it is anatomically composed of 2 heads and the presence of compartments was observed on its internal face. Physiologically, it has been found that the motor units of the biceps are activated at different places of the muscle during production of various functional tasks. In addition, it appears that the central nervous system can use muscle synergy to easily produce multiple movements. In this research, muscle synergy was first identified to be present in the biceps of normal subjects, and it was shown that the characteristics of this synergy allowed the identification of static posture of the hand when the biceps signals had been recorded. In a second investigation, we demonstrated that it was possible in a virtual cube presented on a screen to control online the position of a sphere to reach various targets by using muscle synergy of the biceps. Classification techniques have been used to improve the classification of muscular synergy features, and these classification techniques can be integrated with control algorithm that produces dynamic movement of myoelectric prostheses to facilitate the training of prosthetic control

Machine Learning for Hand Gesture Classification from Surface Electromyography Signals

Classifying hand gestures from Surface Electromyography (sEMG) is a process which has applications in human-machine interaction, rehabilitation and prosthetic control. Reduction in the cost and increase in the availability of necessary hardware over recent years has made sEMG a more viable solution for hand gesture classification. The research challenge is the development of processes to robustly and accurately predict the current gesture based on incoming sEMG data. This thesis presents a set of methods, techniques and designs that improve upon evaluation of, and performance on, the classification problem as a whole. These are brought together to set a new baseline for the potential classification. Evaluation is improved by careful choice of metrics and design of cross-validation techniques that account for data bias caused by common experimental techniques. A landmark study is re-evaluated with these improved techniques, and it is shown that data augmentation can be used to significantly improve upon the performance using conventional classification methods. A novel neural network architecture and supporting improvements are presented that further improve performance and is refined such that the network can achieve similar performance with many fewer parameters than competing designs. Supporting techniques such as subject adaptation and smoothing algorithms are then explored to improve overall performance and also provide more nuanced trade-offs with various aspects of performance, such as incurred latency and prediction smoothness. A new study is presented which compares the performance potential of medical grade electrodes and a low-cost commercial alternative showing that for a modest-sized gesture set, they can compete. The data is also used to explore data labelling in experimental design and to evaluate the numerous aspects of performance that must be traded off

User Training with Error Augmentation for Electromyogram-based Gesture Classification

We designed and tested a system for real-time control of a user interface by extracting surface electromyographic (sEMG) activity from eight electrodes in a wrist-band configuration. sEMG data were streamed into a machine-learning algorithm that classified hand gestures in real-time. After an initial model calibration, participants were presented with one of three types of feedback during a human-learning stage: veridical feedback, in which predicted probabilities from the gesture classification algorithm were displayed without alteration, modified feedback, in which we applied a hidden augmentation of error to these probabilities, and no feedback. User performance was then evaluated in a series of minigames, in which subjects were required to use eight gestures to manipulate their game avatar to complete a task. Experimental results indicated that, relative to baseline, the modified feedback condition led to significantly improved accuracy and improved gesture class separation. These findings suggest that real-time feedback in a gamified user interface with manipulation of feedback may enable intuitive, rapid, and accurate task acquisition for sEMG-based gesture recognition applications.Comment: 10 pages, 10 figure

Bioformers: Embedding Transformers for Ultra-Low Power sEMG-based Gesture Recognition

Human-machine interaction is gaining traction in rehabilitation tasks, such as controlling prosthetic hands or robotic arms. Gesture recognition exploiting surface electromyographic (sEMG) signals is one of the most promising approaches, given that sEMG signal acquisition is non-invasive and is directly related to muscle contraction. However, the analysis of these signals still presents many challenges since similar gestures result in similar muscle contractions. Thus the resulting signal shapes are almost identical, leading to low classification accuracy. To tackle this challenge, complex neural networks are employed, which require large memory footprints, consume relatively high energy and limit the maximum battery life of devices used for classification. This work addresses this problem with the introduction of the Bioformers. This new family of ultra-small attention-based architectures approaches state-of-the-art performance while reducing the number of parameters and operations of 4.9x. Additionally, by introducing a new inter-subjects pre-training, we improve the accuracy of our best Bioformer by 3.39%, matching state-of-the-art accuracy without any additional inference cost.Deploying our best performing Bioformer on a Parallel, Ultra-Low Power (PULP) microcontroller unit (MCU), the GreenWaves GAP8, we achieve an inference latency and energy of 2.72 ms and 0.14 mJ, respectively, 8.0x lower than the previous state-of-the-art neural network, while occupying just 94.2 kB of memory

Sistema de miografia óptica para reconhecimento de gestos e posturas de mão

Orientador: Éric FujiwaraDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia MecânicaResumo: Nesse projeto, demonstrou-se um sistema de miografia óptica como uma alternativa promissora para monitorar as posturas da mão e os gestos do usuário. Essa técnica se fundamenta em acompanhar as atividades musculares responsáveis pelos movimentos da mão com uma câmera externa, relacionando a distorção visual verificada no antebraço com a contração e o relaxamento necessários para dada postura. Três configurações de sensores foram propostas, estudadas e avaliadas. A primeira propôs monitorar a atividade muscular analisando a variação da frequência espacial de uma textura de listras uniformes impressa sobre a pele, enquanto que a segunda se caracteriza pela contagem de pixels de pele visível dentro da região de interesse. Ambas as configurações se mostraram inviáveis pela baixa robustez e alta demanda por condições experimentais controladas. Por fim, a terceira recupera o estado da mão acompanhando o deslocamento de uma série de marcadores coloridos distribuídos ao longo do antebraço. Com um webcam de 24 fps e 640 × 480 pixels, essa última configuração foi validada para oito posturas distintas, explorando principalmente a flexão e extensão dos dedos e do polegar, além da adução e abdução do último. Os dados experimentais, adquiridos off-line, são submetidos a uma rotina de processamento de imagens para extrair a informação espacial e de cor dos marcadores em cada quadro, dados esses utilizados para rastrear os mesmos marcadores ao longo de todos os quadros. Para reduzir a influência das vibrações naturais e inerentes ao corpo humano, um sistema de referencial local é ainda adotado dentro da própria região de interesse. Finalmente, os dados quadro a quadro com o ground truth são alimentados a uma rede neural artificial sequencial, responsável pela calibração supervisionada do sensor e posterior classificação das posturas. O desempenho do sistema para a classificação das oito posturas foi avaliado com base na validação cruzada com 10-folds, com a câmera monitorando o antebraço pela superfície interna ou externa. O sensor apresentou uma precisão de ?92.4% e exatidão de ?97.9% para o primeiro caso, e uma precisão de ?75.1% e exatidão de ?92.5% para o segundo, sendo comparável a outras técnicas de miografia, demonstrando a viabilidade do projeto e abrindo perspectivas para aplicações em interfaces humano-robôAbstract: In this work, an optical myography system is demonstrated as a promising alternative to monitor hand posture and gestures of the user. This technique is based on accompanying muscular activities responsible for hand motion with an external camera, and relating the visual deformation observed on the forearm to the muscular contractions/relaxations for a given posture. Three sensor designs were proposed, studied and evaluated. The first one intended to monitor muscular activity by analyzing the spatial frequency variation of a uniformly distributed stripe pattern stamped on the skin, whereas the second one is characterized by reckoning visible skin pixels inside the region of interest. Both designs are impracticable due to their low robustness and high demand for controlled experimental conditions. At last, the third design retrieves hand configuration by tracking visually the displacements of a series of color markers distributed over the forearm. With a webcam of 24 fps and 640 × 480 pixels, this design was validated for eight different postures, exploring fingers and thumb flexion/extension, plus thumb adduction/abduction. The experimental data are acquired offline and, then, submitted to an image processing routine to extract color and spatial information of the markers in each frame; the extracted data is subsequently used to track the same markers along all frames. To reduce the influence of human body natural and inherent vibrations, a local reference frame is yet adopted in the region of interest. Finally, the frame by frame data, along with the ground truth posture, are fed into a sequential artificial neural network, responsible for sensor supervised calibration and subsequent posture classification. The system performance was evaluated in terms of eight postures classification via 10-fold cross-validation, with the camera monitoring either the underside or the back of the forearm. The sensor presented a ?92.4% precision and ?97.9% accuracy for the former, and a ?75.1% precision and ?92.5% accuracy for the latter, being thus comparable to other myographic techniques; it also demonstrated that the project is feasible and offers prospects for human-robot interaction applicationsMestradoEngenharia MecanicaMestre em Engenharia Mecânica33003017CAPE