Real-Time Hand Gesture Recognition Using Temporal Muscle Activation Maps of Multi-Channel sEMG Signals

Accurate and real-time hand gesture recognition is essential for controlling advanced hand prostheses. Surface Electromyography (sEMG) signals obtained from the forearm are widely used for this purpose. Here, we introduce a novel hand gesture representation called Temporal Muscle Activation (TMA) maps which captures information about the activation patterns of muscles in the forearm. Based on these maps, we propose an algorithm that can recognize hand gestures in real-time using a Convolution Neural Network. The algorithm was tested on 8 healthy subjects with sEMG signals acquired from 8 electrodes placed along the circumference of the forearm. The average classification accuracy of the proposed method was 94%, which is comparable to state-of-the-art methods. The average computation time of a prediction was 5.5ms, making the algorithm ideal for the real-time gesture recognition applications.Comment: Paper accepted to IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 202

Multikernel convolutional neural network for sEMG based hand gesture classification

openIl riconoscimento dei gesti della mano è un argomento ampiamente discusso in letteratura, dove vengono analizzate diverse tecniche sia in termini di tipi di segnale in ingresso che di algoritmi. Tra i più utilizzati ci sono i segnali elettromiografici (sEMG), già ampiamente sfruttati nelle applicazioni di interazione uomo-macchina (HMI). Determinare come decodificare le informazioni contenute nei segnali EMG in modo robusto e accurato è un problema chiave per il quale è urgente trovare una soluzione. Recentemente, molti incarichi di riconoscimento dei pattern EMG sono stati affrontati utilizzando metodi di deep learning. Nonostante le elevate prestazioni di questi ultimi, le loro capacità di generalizzazione sono spesso limitate dall'elevata eterogeneità tra i soggetti, l'impedenza cutanea, il posizionamento dei sensori, ecc. Inoltre, poiché questo progetto è focalizzato sull'applicazione in tempo reale di protesi, ci sono maggiori vincoli sui tempi di risposta del sistema che riducono la complessità dei modelli. In questa tesi è stata testata una rete neurale convoluzionale multi-kernel su diversi dataset pubblici per verificare la sua generalizzabilità. Inoltre, è stata analizzata la capacità del modello di superare i limiti inter-soggetto e inter-sessione in giorni diversi, preservando i vincoli legati a un sistema embedded. I risultati confermano le difficoltà incontrate nell'estrazione di informazioni dai segnali emg; tuttavia, dimostrano la possibilità di ottenere buone prestazioni per un uso robusto di mani prostetiche. Inoltre, è possibile ottenere prestazioni migliori personalizzando il modello con tecniche di transfer learning e di adattamento al dominio.Hand gesture recognition is a widely discussed topic in the literature, where different techniques are analyzed in terms of both input signal types and algorithms. Among the most widely used are electromyographic signals (sEMG), which are already widely exploited in human-computer interaction (HMI) applications. Determining how to decode the information contained in EMG signals robustly and accurately is a key problem for which a solution is urgently needed. Recently, many EMG pattern recognition tasks have been addressed using deep learning methods. Despite their high performance, their generalization capabilities are often limited by high heterogeneity among subjects, skin impedance, sensor placement, etc. In addition, because this project is focused on the real-time application of prostheses, there are greater constraints on the system response times that reduce the complexity of the models. In this thesis, a multi-kernel convolutional neural network was tested on several public datasets to verify its generalizability. In addition, the model's ability to overcome inter-subject and inter-session constraints on different days while preserving the constraints associated with an embedded system was analyzed. The results confirm the difficulties encountered in extracting information from emg signals; however, they demonstrate the possibility of achieving good performance for robust use of prosthetic hands. In addition, better performance can be achieved by customizing the model with transfer learning and domain-adaptationtechniques

sEMG-based hand gesture recognition with deep learning

Hand gesture recognition based on surface electromyographic (sEMG) signals is a promising approach for the development of Human-Machine Interfaces (HMIs) with a natural control, such as intuitive robot interfaces or poly-articulated prostheses. However, real-world applications are limited by reliability problems due to motion artifacts, postural and temporal variability, and sensor re-positioning. This master thesis is the first application of deep learning on the Unibo-INAIL dataset, the first public sEMG dataset exploring the variability between subjects, sessions and arm postures, by collecting data over 8 sessions of each of 7 able-bodied subjects executing 6 hand gestures in 4 arm postures. In the most recent studies, the variability is addressed with training strategies based on training set composition, which improve inter-posture and inter-day generalization of classical (i.e. non-deep) machine learning classifiers, among which the RBF-kernel SVM yields the highest accuracy. The deep architecture realized in this work is a 1d-CNN implemented in Pytorch, inspired by a 2d-CNN reported to perform well on other public benchmark databases. On this 1d-CNN, various training strategies based on training set composition were implemented and tested. Multi-session training proves to yield higher inter-session validation accuracies than single-session training. Two-posture training proves to be the best postural training (proving the benefit of training on more than one posture), and yields 81.2% inter-posture test accuracy. Five-day training proves to be the best multi-day training, and yields 75.9% inter-day test accuracy. All results are close to the baseline. Moreover, the results of multi-day trainings highlight the phenomenon of user adaptation, indicating that training should also prioritize recent data. Though not better than the baseline, the achieved classification accuracies rightfully place the 1d-CNN among the candidates for further research

From Unimodal to Multimodal: improving the sEMG-Based Pattern Recognition via deep generative models

Multimodal hand gesture recognition (HGR) systems can achieve higher recognition accuracy. However, acquiring multimodal gesture recognition data typically requires users to wear additional sensors, thereby increasing hardware costs. This paper proposes a novel generative approach to improve Surface Electromyography (sEMG)-based HGR accuracy via virtual Inertial Measurement Unit (IMU) signals. Specifically, we trained a deep generative model based on the intrinsic correlation between forearm sEMG signals and forearm IMU signals to generate virtual forearm IMU signals from the input forearm sEMG signals at first. Subsequently, the sEMG signals and virtual IMU signals were fed into a multimodal Convolutional Neural Network (CNN) model for gesture recognition. To evaluate the performance of the proposed approach, we conducted experiments on 6 databases, including 5 publicly available databases and our collected database comprising 28 subjects performing 38 gestures, containing both sEMG and IMU data. The results show that our proposed approach outperforms the sEMG-based unimodal HGR method (with increases of 2.15%-13.10%). It demonstrates that incorporating virtual IMU signals, generated by deep generative models, can significantly enhance the accuracy of sEMG-based HGR. The proposed approach represents a successful attempt to transition from unimodal HGR to multimodal HGR without additional sensor hardware

On the Utility of Representation Learning Algorithms for Myoelectric Interfacing

Electrical activity produced by muscles during voluntary movement is a reflection of the firing patterns of relevant motor neurons and, by extension, the latent motor intent driving the movement. Once transduced via electromyography (EMG) and converted into digital form, this activity can be processed to provide an estimate of the original motor intent and is as such a feasible basis for non-invasive efferent neural interfacing. EMG-based motor intent decoding has so far received the most attention in the field of upper-limb prosthetics, where alternative means of interfacing are scarce and the utility of better control apparent. Whereas myoelectric prostheses have been available since the 1960s, available EMG control interfaces still lag behind the mechanical capabilities of the artificial limbs they are intended to steer—a gap at least partially due to limitations in current methods for translating EMG into appropriate motion commands. As the relationship between EMG signals and concurrent effector kinematics is highly non-linear and apparently stochastic, finding ways to accurately extract and combine relevant information from across electrode sites is still an active area of inquiry.This dissertation comprises an introduction and eight papers that explore issues afflicting the status quo of myoelectric decoding and possible solutions, all related through their use of learning algorithms and deep Artificial Neural Network (ANN) models. Paper I presents a Convolutional Neural Network (CNN) for multi-label movement decoding of high-density surface EMG (HD-sEMG) signals. Inspired by the successful use of CNNs in Paper I and the work of others, Paper II presents a method for automatic design of CNN architectures for use in myocontrol. Paper III introduces an ANN architecture with an appertaining training framework from which simultaneous and proportional control emerges. Paper Iv introduce a dataset of HD-sEMG signals for use with learning algorithms. Paper v applies a Recurrent Neural Network (RNN) model to decode finger forces from intramuscular EMG. Paper vI introduces a Transformer model for myoelectric interfacing that do not need additional training data to function with previously unseen users. Paper vII compares the performance of a Long Short-Term Memory (LSTM) network to that of classical pattern recognition algorithms. Lastly, paper vIII describes a framework for synthesizing EMG from multi-articulate gestures intended to reduce training burden

Interpreting Deep Learning Features for Myoelectric Control: A Comparison with Handcrafted Features

The research in myoelectric control systems primarily focuses on extracting discriminative representations from the electromyographic (EMG) signal by designing handcrafted features. Recently, deep learning techniques have been applied to the challenging task of EMG-based gesture recognition. The adoption of these techniques slowly shifts the focus from feature engineering to feature learning. However, the black-box nature of deep learning makes it hard to understand the type of information learned by the network and how it relates to handcrafted features. Additionally, due to the high variability in EMG recordings between participants, deep features tend to generalize poorly across subjects using standard training methods. Consequently, this work introduces a new multi-domain learning algorithm, named ADANN, which significantly enhances (p=0.00004) inter-subject classification accuracy by an average of 19.40% compared to standard training. Using ADANN-generated features, the main contribution of this work is to provide the first topological data analysis of EMG-based gesture recognition for the characterisation of the information encoded within a deep network, using handcrafted features as landmarks. This analysis reveals that handcrafted features and the learned features (in the earlier layers) both try to discriminate between all gestures, but do not encode the same information to do so. Furthermore, using convolutional network visualization techniques reveal that learned features tend to ignore the most activated channel during gesture contraction, which is in stark contrast with the prevalence of handcrafted features designed to capture amplitude information. Overall, this work paves the way for hybrid feature sets by providing a clear guideline of complementary information encoded within learned and handcrafted features.Comment: The first two authors shared first authorship. The last three authors shared senior authorship. 32 page

Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning

In recent years, deep learning algorithms have become increasingly more prominent for their unparalleled ability to automatically learn discriminant features from large amounts of data. However, within the field of electromyography-based gesture recognition, deep learning algorithms are seldom employed as they require an unreasonable amount of effort from a single person, to generate tens of thousands of examples. This work's hypothesis is that general, informative features can be learned from the large amounts of data generated by aggregating the signals of multiple users, thus reducing the recording burden while enhancing gesture recognition. Consequently, this paper proposes applying transfer learning on aggregated data from multiple users, while leveraging the capacity of deep learning algorithms to learn discriminant features from large datasets. Two datasets comprised of 19 and 17 able-bodied participants respectively (the first one is employed for pre-training) were recorded for this work, using the Myo Armband. A third Myo Armband dataset was taken from the NinaPro database and is comprised of 10 able-bodied participants. Three different deep learning networks employing three different modalities as input (raw EMG, Spectrograms and Continuous Wavelet Transform (CWT)) are tested on the second and third dataset. The proposed transfer learning scheme is shown to systematically and significantly enhance the performance for all three networks on the two datasets, achieving an offline accuracy of 98.31% for 7 gestures over 17 participants for the CWT-based ConvNet and 68.98% for 18 gestures over 10 participants for the raw EMG-based ConvNet. Finally, a use-case study employing eight able-bodied participants suggests that real-time feedback allows users to adapt their muscle activation strategy which reduces the degradation in accuracy normally experienced over time.Comment: Source code and datasets available: https://github.com/Giguelingueling/MyoArmbandDatase