Temporal Variability Analysis in sEMG Hand Grasp Recognition using Temporal Convolutional Networks

Hand movement recognition via surface electromyographic (sEMG) signal is a promising approach for the advance in Human-Computer Interaction. However, this field has to deal with two main issues: (1) the long-term reliability of sEMG-based control is limited by the variability affecting the sEMG signal (especially, variability over time); (2) the classification algorithms need to be suitable for implementation on embedded devices, which have strict constraints in terms of power budget and computational resources. Current solutions present a performance over-time drop that makes them unsuitable for reliable gesture controller design. In this paper, we address temporal variability of sEMG-based grasp recognition, proposing a new approach based on Temporal Convolutional Networks, a class of deep learning algorithms particularly suited for time series analysis and temporal pattern recognition. Our approach improves by 7.6% the best results achieved in the literature on the NinaPro DB6, a reference dataset for temporal variability analysis of sEMG. Moreover, when targeting the much more challenging inter-session accuracy objective, our method achieves an accuracy drop of just 4.8% between intra- and inter-session validation. This proves the suitability of our setup for a robust, reliable long-term implementation. Furthermore, we distill the network using deep network quantization and pruning techniques, demonstrating that our approach can use down to 120x lower memory footprint than the initial network and 4x lower memory footprint than a baseline Support Vector Machine, with an inter-session accuracy degradation of only 2.5%, proving that the solution is suitable for embedded resource-constrained implementations

Multikernel convolutional neural network for sEMG based hand gesture classification

openIl riconoscimento dei gesti della mano è un argomento ampiamente discusso in letteratura, dove vengono analizzate diverse tecniche sia in termini di tipi di segnale in ingresso che di algoritmi. Tra i più utilizzati ci sono i segnali elettromiografici (sEMG), già ampiamente sfruttati nelle applicazioni di interazione uomo-macchina (HMI). Determinare come decodificare le informazioni contenute nei segnali EMG in modo robusto e accurato è un problema chiave per il quale è urgente trovare una soluzione. Recentemente, molti incarichi di riconoscimento dei pattern EMG sono stati affrontati utilizzando metodi di deep learning. Nonostante le elevate prestazioni di questi ultimi, le loro capacità di generalizzazione sono spesso limitate dall'elevata eterogeneità tra i soggetti, l'impedenza cutanea, il posizionamento dei sensori, ecc. Inoltre, poiché questo progetto è focalizzato sull'applicazione in tempo reale di protesi, ci sono maggiori vincoli sui tempi di risposta del sistema che riducono la complessità dei modelli. In questa tesi è stata testata una rete neurale convoluzionale multi-kernel su diversi dataset pubblici per verificare la sua generalizzabilità. Inoltre, è stata analizzata la capacità del modello di superare i limiti inter-soggetto e inter-sessione in giorni diversi, preservando i vincoli legati a un sistema embedded. I risultati confermano le difficoltà incontrate nell'estrazione di informazioni dai segnali emg; tuttavia, dimostrano la possibilità di ottenere buone prestazioni per un uso robusto di mani prostetiche. Inoltre, è possibile ottenere prestazioni migliori personalizzando il modello con tecniche di transfer learning e di adattamento al dominio.Hand gesture recognition is a widely discussed topic in the literature, where different techniques are analyzed in terms of both input signal types and algorithms. Among the most widely used are electromyographic signals (sEMG), which are already widely exploited in human-computer interaction (HMI) applications. Determining how to decode the information contained in EMG signals robustly and accurately is a key problem for which a solution is urgently needed. Recently, many EMG pattern recognition tasks have been addressed using deep learning methods. Despite their high performance, their generalization capabilities are often limited by high heterogeneity among subjects, skin impedance, sensor placement, etc. In addition, because this project is focused on the real-time application of prostheses, there are greater constraints on the system response times that reduce the complexity of the models. In this thesis, a multi-kernel convolutional neural network was tested on several public datasets to verify its generalizability. In addition, the model's ability to overcome inter-subject and inter-session constraints on different days while preserving the constraints associated with an embedded system was analyzed. The results confirm the difficulties encountered in extracting information from emg signals; however, they demonstrate the possibility of achieving good performance for robust use of prosthetic hands. In addition, better performance can be achieved by customizing the model with transfer learning and domain-adaptationtechniques

Support vector machines to detect physiological patterns for EEG and EMG-based human-computer interaction:a review

Support vector machines (SVMs) are widely used classifiers for detecting physiological patterns in human-computer interaction (HCI). Their success is due to their versatility, robustness and large availability of free dedicated toolboxes. Frequently in the literature, insufficient details about the SVM implementation and/or parameters selection are reported, making it impossible to reproduce study analysis and results. In order to perform an optimized classification and report a proper description of the results, it is necessary to have a comprehensive critical overview of the applications of SVM. The aim of this paper is to provide a review of the usage of SVM in the determination of brain and muscle patterns for HCI, by focusing on electroencephalography (EEG) and electromyography (EMG) techniques. In particular, an overview of the basic principles of SVM theory is outlined, together with a description of several relevant literature implementations. Furthermore, details concerning reviewed papers are listed in tables and statistics of SVM use in the literature are presented. Suitability of SVM for HCI is discussed and critical comparisons with other classifiers are reported

Simultaneous assessment and training of an upper-limb amputee using incremental machine-learning-based myocontrol:a single-case experimental design

Background Machine-learning-based myocontrol of prosthetic devices suffers from a high rate of abandonment due to dissatisfaction with the training procedure and with the reliability of day-to-day control. Incremental myocontrol is a promising approach as it allows on-demand updating of the system, thus enforcing continuous interaction with the user. Nevertheless, a long-term study assessing the efficacy of incremental myocontrol is still missing, partially due to the lack of an adequate tool to do so. In this work we close this gap and report about a person with upper-limb absence who learned to control a dexterous hand prosthesis using incremental myocontrol through a novel functional assessment protocol called SATMC (Simultaneous Assessment and Training of Myoelectric Control). Methods The participant was fitted with a custom-made prosthetic setup with a controller based on Ridge Regression with Random Fourier Features (RR-RFF), a non-linear, incremental machine learning method, used to build and progressively update the myocontrol system. During a 13-month user study, the participant performed increasingly complex daily-living tasks, requiring fine bimanual coordination and manipulation with a multi-fingered hand prosthesis, in a realistic laboratory setup. The SATMC was used both to compose the tasks and continually assess the participant's progress. Patient satisfaction was measured using Visual Analog Scales. Results Over the course of the study, the participant progressively improved his performance both objectively, e.g., the time required to complete each task became shorter, and subjectively, meaning that his satisfaction improved. The SATMC actively supported the improvement of the participant by progressively increasing the difficulty of the tasks in a structured way. In combination with the incremental RR-RFF allowing for small adjustments when required, the participant was capable of reliably using four actions of the prosthetic hand to perform all required tasks at the end of the study. Conclusions Incremental myocontrol enabled an upper-limb amputee to reliably control a dexterous hand prosthesis while providing a subjectively satisfactory experience. The SATMC can be an effective tool to this aim

Guidage non-intrusif d'un bras robotique à l'aide d'un bracelet myoélectrique à électrode sèche

Depuis plusieurs années la robotique est vue comme une solution clef pour améliorer la qualité de vie des personnes ayant subi une amputation. Pour créer de nouvelles prothèses intelligentes qui peuvent être facilement intégrées à la vie quotidienne et acceptée par ces personnes, celles-ci doivent être non-intrusives, fiables et peu coûteuses. L’électromyographie de surface fournit une interface intuitive et non intrusive basée sur l’activité musculaire de l’utilisateur permettant d’interagir avec des robots. Cependant, malgré des recherches approfondies dans le domaine de la classification des signaux sEMG, les classificateurs actuels manquent toujours de fiabilité, car ils ne sont pas robustes face au bruit à court terme (par exemple, petit déplacement des électrodes, fatigue musculaire) ou à long terme (par exemple, changement de la masse musculaire et des tissus adipeux) et requiert donc de recalibrer le classifieur de façon périodique. L’objectif de mon projet de recherche est de proposer une interface myoélectrique humain-robot basé sur des algorithmes d’apprentissage par transfert et d’adaptation de domaine afin d’augmenter la fiabilité du système à long-terme, tout en minimisant l’intrusivité (au niveau du temps de préparation) de ce genre de système. L’aspect non intrusif est obtenu en utilisant un bracelet à électrode sèche possédant dix canaux. Ce bracelet (3DC Armband) est de notre (Docteur Gabriel Gagnon-Turcotte, mes co-directeurs et moi-même) conception et a été réalisé durant mon doctorat. À l’heure d’écrire ces lignes, le 3DC Armband est le bracelet sans fil pour l’enregistrement de signaux sEMG le plus performant disponible. Contrairement aux dispositifs utilisant des électrodes à base de gel qui nécessitent un rasage de l’avant-bras, un nettoyage de la zone de placement et l’application d’un gel conducteur avant l’utilisation, le brassard du 3DC peut simplement être placé sur l’avant-bras sans aucune préparation. Cependant, cette facilité d’utilisation entraîne une diminution de la qualité de l’information du signal. Cette diminution provient du fait que les électrodes sèches obtiennent un signal plus bruité que celle à base de gel. En outre, des méthodes invasives peuvent réduire les déplacements d’électrodes lors de l’utilisation, contrairement au brassard. Pour remédier à cette dégradation de l’information, le projet de recherche s’appuiera sur l’apprentissage profond, et plus précisément sur les réseaux convolutionels. Le projet de recherche a été divisé en trois phases. La première porte sur la conception d’un classifieur permettant la reconnaissance de gestes de la main en temps réel. La deuxième porte sur l’implémentation d’un algorithme d’apprentissage par transfert afin de pouvoir profiter des données provenant d’autres personnes, permettant ainsi d’améliorer la classification des mouvements de la main pour un nouvel individu tout en diminuant le temps de préparation nécessaire pour utiliser le système. La troisième phase consiste en l’élaboration et l’implémentation des algorithmes d’adaptation de domaine et d’apprentissage faiblement supervisé afin de créer un classifieur qui soit robuste au changement à long terme.For several years, robotics has been seen as a key solution to improve the quality of life of people living with upper-limb disabilities. To create new, smart prostheses that can easily be integrated into everyday life, they must be non-intrusive, reliable and inexpensive. Surface electromyography provides an intuitive interface based on a user’s muscle activity to interact with robots. However, despite extensive research in the field of sEMG signal classification, current classifiers still lack reliability due to their lack of robustness to short-term (e.g. small electrode displacement, muscle fatigue) or long-term (e.g. change in muscle mass and adipose tissue) noise. In practice, this mean that to be useful, classifier needs to be periodically re-calibrated, a time consuming process. The goal of my research project is to proposes a human-robot myoelectric interface based on transfer learning and domain adaptation algorithms to increase the reliability of the system in the long term, while at the same time reducing the intrusiveness (in terms of hardware and preparation time) of this kind of systems. The non-intrusive aspect is achieved from a dry-electrode armband featuring ten channels. This armband, named the 3DC Armband is from our (Dr. Gabriel Gagnon-Turcotte, my co-directors and myself) conception and was realized during my doctorate. At the time of writing, the 3DC Armband offers the best performance for currently available dry-electrodes, surface electromyographic armbands. Unlike gel-based electrodes which require intrusive skin preparation (i.e. shaving, cleaning the skin and applying conductive gel), the 3DC Armband can simply be placed on the forearm without any preparation. However, this ease of use results in a decrease in the quality of information. This decrease is due to the fact that the signal recorded by dry electrodes is inherently noisier than gel-based ones. In addition, other systems use invasive methods (intramuscular electromyography) to capture a cleaner signal and reduce the source of noises (e.g. electrode shift). To remedy this degradation of information resulting from the non-intrusiveness of the armband, this research project will rely on deep learning, and more specifically on convolutional networks. The research project was divided into three phases. The first is the design of a classifier allowing the recognition of hand gestures in real-time. The second is the implementation of a transfer learning algorithm to take advantage of the data recorded across multiple users, thereby improving the system’s accuracy, while decreasing the time required to use the system. The third phase is the development and implementation of a domain adaptation and self-supervised learning to enhance the classifier’s robustness to long-term changes

Human skill capturing and modelling using wearable devices

Industrial robots are delivering more and more manipulation services in manufacturing. However, when the task is complex, it is difficult to programme a robot to fulfil all the requirements because even a relatively simple task such as a peg-in-hole insertion contains many uncertainties, e.g. clearance, initial grasping position and insertion path. Humans, on the other hand, can deal with these variations using their vision and haptic feedback. Although humans can adapt to uncertainties easily, most of the time, the skilled based performances that relate to their tacit knowledge cannot be easily articulated. Even though the automation solution may not fully imitate human motion since some of them are not necessary, it would be useful if the skill based performance from a human could be firstly interpreted and modelled, which will then allow it to be transferred to the robot. This thesis aims to reduce robot programming efforts significantly by developing a methodology to capture, model and transfer the manual manufacturing skills from a human demonstrator to the robot. Recently, Learning from Demonstration (LfD) is gaining interest as a framework to transfer skills from human teacher to robot using probability encoding approaches to model observations and state transition uncertainties. In close or actual contact manipulation tasks, it is difficult to reliabley record the state-action examples without interfering with the human senses and activities. Therefore, wearable sensors are investigated as a promising device to record the state-action examples without restricting the human experts during the skilled execution of their tasks. Firstly to track human motions accurately and reliably in a defined 3-dimensional workspace, a hybrid system of Vicon and IMUs is proposed to compensate for the known limitations of the individual system. The data fusion method was able to overcome occlusion and frame flipping problems in the two camera Vicon setup and the drifting problem associated with the IMUs. The results indicated that occlusion and frame flipping problems associated with Vicon can be mitigated by using the IMU measurements. Furthermore, the proposed method improves the Mean Square Error (MSE) tracking accuracy range from 0.8˚ to 6.4˚ compared with the IMU only method. Secondly, to record haptic feedback from a teacher without physically obstructing their interactions with the workpiece, wearable surface electromyography (sEMG) armbands were used as an indirect method to indicate contact feedback during manual manipulations. A muscle-force model using a Time Delayed Neural Network (TDNN) was built to map the sEMG signals to the known contact force. The results indicated that the model was capable of estimating the force from the sEMG armbands in the applications of interest, namely in peg-in-hole and beater winding tasks, with MSE of 2.75N and 0.18N respectively. Finally, given the force estimation and the motion trajectories, a Hidden Markov Model (HMM) based approach was utilised as a state recognition method to encode and generalise the spatial and temporal information of the skilled executions. This method would allow a more representative control policy to be derived. A modified Gaussian Mixture Regression (GMR) method was then applied to enable motions reproduction by using the learned state-action policy. To simplify the validation procedure, instead of using the robot, additional demonstrations from the teacher were used to verify the reproduction performance of the policy, by assuming human teacher and robot learner are physical identical systems. The results confirmed the generalisation capability of the HMM model across a number of demonstrations from different subjects; and the reproduced motions from GMR were acceptable in these additional tests. The proposed methodology provides a framework for producing a state-action model from skilled demonstrations that can be translated into robot kinematics and joint states for the robot to execute. The implication to industry is reduced efforts and time in programming the robots for applications where human skilled performances are required to cope robustly with various uncertainties during tasks execution