Kinematic synergies of hand grasps ::a comprehensive study on a large publicly available dataset

Background: Hand grasp patterns require complex coordination. The reduction of the kinematic dimensionality is a key process to study the patterns underlying hand usage and grasping. It allows to define metrics for motor assessment and rehabilitation, to develop assistive devices and prosthesis control methods. Several studies were presented in this field but most of them targeted a limited number of subjects, they focused on postures rather than entire grasping movements and they did not perform separate analysis for the tasks and subjects, which can limit the impact on rehabilitation and assistive applications. This paper provides a comprehensive mapping of synergies from hand grasps targeting activities of daily living. It clarifies several current limits of the field and fosters the development of applications in rehabilitation and assistive robotics. Methods: In this work, hand kinematic data of 77 subjects, performing up to 20 hand grasps, were acquired with a data glove (a 22-sensor CyberGlove II data glove) and analyzed. Principal Component Analysis (PCA) and hierarchical cluster analysis were used to extract and group kinematic synergies that summarize the coordination patterns available for hand grasps. Results: Twelve synergies were found to account for > 80% of the overall variation. The first three synergies accounted for more than 50% of the total amount of variance and consisted of: the flexion and adduction of the Metacarpophalangeal joint (MCP) of fingers 3 to 5 (synergy #1), palmar arching and flexion of the wrist (synergy #2) and opposition of the thumb (synergy #3). Further synergies refine movements and have higher variability among subjects. Conclusion: Kinematic synergies are extracted from a large number of subjects (77) and grasps related to activities of daily living (20). The number of motor modules required to perform the motor tasks is higher than what previously described. Twelve synergies are responsible for most of the variation in hand grasping. The first three are used as primary synergies, while the remaining ones target finer movements (e.g. independence of thumb and index finger). The results generalize the description of hand kinematics, better clarifying several limits of the field and fostering the development of applications in rehabilitation and assistive robotics

Movement and gesture recognition using deep learning and wearable-sensor technology

Pattern recognition of time-series signals for movement and gesture analysis plays an important role in many fields as diverse as healthcare, astronomy, industry and entertainment. As a new technique in recent years, Deep Learning (DL) has made tremendous progress in computer vision and Natural Language Processing (NLP), but largely unexplored on its performance for movement and gesture recognition from noisy multi-channel sensor signals. To tackle this problem, this study was undertaken to classify diverse movements and gestures using four developed DL models: a 1-D Convolutional neural network (1-D CNN), a Recurrent neural network model with Long Short Term Memory (LSTM), a basic hybrid model containing one convolutional layer and one recurrent layer (C-RNN), and an advanced hybrid model containing three convolutional layers and three recurrent layers (3+3 C-RNN). The models will be applied on three different databases (DB) where the performances of models were compared. DB1 is the HCL dataset which includes 6 human daily activities of 30 subjects based on accelerometer and gyroscope signals. DB2 and DB3 are both based on the surface electromyography (sEMG) signal for 17 diverse movements. The evaluation and discussion for the improvements and limitations of the models were made according to the result

Subject-Independent Frameworks for Robotic Devices: Applying Robot Learning to EMG Signals

The capability of having human and robots cooperating together has increased the interest in the control of robotic devices by means of physiological human signals. In order to achieve this goal it is crucial to be able to catch the human intention of movement and to translate it in a coherent robot action. Up to now, the classical approach when considering physiological signals, and in particular EMG signals, is to focus on the specific subject performing the task since the great complexity of these signals. This thesis aims to expand the state of the art by proposing a general subject-independent framework, able to extract the common constraints of human movement by looking at several demonstration by many different subjects. The variability introduced in the system by multiple demonstrations from many different subjects allows the construction of a robust model of human movement, able to face small variations and signal deterioration. Furthermore, the obtained framework could be used by any subject with no need for long training sessions. The signals undergo to an accurate preprocessing phase, in order to remove noise and artefacts. Following this procedure, we are able to extract significant information to be used in online processes. The human movement can be estimated by using well-established statistical methods in Robot Programming by Demonstration applications, in particular the input can be modelled by using a Gaussian Mixture Model (GMM). The performed movement can be continuously estimated with a Gaussian Mixture Regression (GMR) technique, or it can be identified among a set of possible movements with a Gaussian Mixture Classification (GMC) approach. We improved the results by incorporating some previous information in the model, in order to enriching the knowledge of the system. In particular we considered the hierarchical information provided by a quantitative taxonomy of hand grasps. Thus, we developed the first quantitative taxonomy of hand grasps considering both muscular and kinematic information from 40 subjects. The results proved the feasibility of a subject-independent framework, even by considering physiological signals, like EMG, from a wide number of participants. The proposed solution has been used in two different kinds of applications: (I) for the control of prosthesis devices, and (II) in an Industry 4.0 facility, in order to allow human and robot to work alongside or to cooperate. Indeed, a crucial aspect for making human and robots working together is their mutual knowledge and anticipation of other’s task, and physiological signals are capable to provide a signal even before the movement is started. In this thesis we proposed also an application of Robot Programming by Demonstration in a real industrial facility, in order to optimize the production of electric motor coils. The task was part of the European Robotic Challenge (EuRoC), and the goal was divided in phases of increasing complexity. This solution exploits Machine Learning algorithms, like GMM, and the robustness was assured by considering demonstration of the task from many subjects. We have been able to apply an advanced research topic to a real factory, achieving promising results

Context-awareness for mobile sensing: a survey and future directions

The evolution of smartphones together with increasing computational power have empowered developers to create innovative context-aware applications for recognizing user related social and cognitive activities in any situation and at any location. The existence and awareness of the context provides the capability of being conscious of physical environments or situations around mobile device users. This allows network services to respond proactively and intelligently based on such awareness. The key idea behind context-aware applications is to encourage users to collect, analyze and share local sensory knowledge in the purpose for a large scale community use by creating a smart network. The desired network is capable of making autonomous logical decisions to actuate environmental objects, and also assist individuals. However, many open challenges remain, which are mostly arisen due to the middleware services provided in mobile devices have limited resources in terms of power, memory and bandwidth. Thus, it becomes critically important to study how the drawbacks can be elaborated and resolved, and at the same time better understand the opportunities for the research community to contribute to the context-awareness. To this end, this paper surveys the literature over the period of 1991-2014 from the emerging concepts to applications of context-awareness in mobile platforms by providing up-to-date research and future research directions. Moreover, it points out the challenges faced in this regard and enlighten them by proposing possible solutions

Kernel density estimation of electromyographic signals and ensemble learning for highly accurate classification of a large set of hand/wrist motions

The performance of myoelectric control highly depends on the features extracted from surface electromyographic (sEMG) signals. We propose three new sEMG features based on the kernel density estimation. The trimmed mean of density (TMD), the entropy of density, and the trimmed mean absolute value of derivative density were computed for each sEMG channel. These features were tested for the classification of single tasks as well as of two tasks concurrently performed. For single tasks, correlation-based feature selection was used, and the features were then classified using linear discriminant analysis (LDA), non-linear support vector machines, and multi-layer perceptron. The eXtreme gradient boosting (XGBoost) classifier was used for the classification of two movements simultaneously performed. The second and third versions of the Ninapro dataset (conventional control) and Ameri’s movement dataset (simultaneous control) were used to test the proposed features. For the Ninapro dataset, the overall accuracy of LDA using the TMD feature was 98.99 ± 1.36% and 92.25 ± 9.48% for able-bodied and amputee subjects, respectively. Using ensemble learning of the three classifiers, the average macro and micro-F-score, macro recall, and precision on the validation sets were 98.23 ± 2.02, 98.32 ± 1.93, 98.32 ± 1.93, and 98.88 ± 1.31%, respectively, for the intact subjects. The movement misclassification percentage was 1.75 ± 1.73 and 3.44 ± 2.23 for the intact subjects and amputees. The proposed features were significantly correlated with the movement classes [Generalized Linear Model (GLM); P-value < 0.05]. An accurate online implementation of the proposed algorithm was also presented. For the simultaneous control, the overall accuracy was 99.71 ± 0.08 and 97.85 ± 0.10 for the XGBoost and LDA classifiers, respectively. The proposed features are thus promising for conventional and simultaneous myoelectric control.Peer ReviewedPostprint (published version

Classification de mouvements fantômes du membre supérieur chez des amputés huméraux à l'aide de mesures électromyographiques et cinématiques

RÉSUMÉ La perte d’un membre supérieur engendre de nombreux déficits fonctionnels pour l’amputé dans sa vie de tous les jours. En effet, la plupart des activités de la vie quotidienne, telles qu’attacher ses souliers ou ouvrir une bouteille, sont complexes et difficiles à réaliser avec un seul bras fonctionnel. Les impacts de ces déficits augmentent à mesure que le niveau d’amputation est plus haut au niveau du bras. Pour toutes ces personnes, les nombreuses avancées dans le domaine des prothèses myoélectriques, c’est-à-dire commandées par l’activité musculaire des muscles restants après l’amputation, sont encourageantes parce qu’elles permettent d’entretenir l’espoir d’une prothèse à la commande intuitive. Un phénomène particulier, présent chez la majorité des amputés, est celui des sensations au membre fantôme. Ces sensations peuvent se manifester sous plusieurs formes : thermiques, douleurs, mobilités. Les mobilités du membre fantôme sont particulièrement intéressantes pour le développement des prothèses myoélectriques étant donné qu’il a été démontré que les mouvements fantômes produisent une activité électromyographique (EMG) au niveau du membre amputé. Cependant, les études s’intéressant à la détection des mouvements fantômes ont enregistré l’activité EMG provenant de muscles difficilement intégrables dans l’emboiture d’une prothèse myoélectriques, tels que ceux du dos, du torse et de l’épaule. La présente étude se concentre sur la classification des mouvements fantômes chez les amputés huméraux à l’aide de l’EMG dans l’optique de développer une prothèse myoélectrique commandée par reconnaissance de formes. Cinq adultes ayant subi une amputation unilatérale humérale suite à un trauma ont participé à cette étude. L’activité EMG des participants a été enregistrée exclusivement autour de leur moignon. Durant les enregistrements, il était demandé aux participants de réaliser l’un des principaux mouvements fantômes du membre supérieur : la flexion ou l’extension du coude, la pronation ou la supination de l’avant-bras, la flexion ou l’extension du poignet, l’ouverture ou la fermeture de la main et le repos. Chaque mouvement fantôme devait être réalisé symétriquement à l’aide du bras sain et la cinématique de ce dernier a été enregistrée à l’aide d’un système d’analyse du mouvement. Dix caractéristiques (ou « features » en anglais) temporels ont été extraites des signaux EMG et utilisées pour entraîner un réseau de neurones permettant de classifier les mouvements fantômes du membre supérieur.----------ABSTRACT Upper limb amputation creates substantial functional deficits for the amputee. Indeed, most activities of daily living, such as tying shoelaces or opening a bottle, are complex and hard to achieve with only one functional arm. These functional impairments increase as the level of amputation is higher up the arm. For these people, recent advances in the field of myoelectric prostheses, i.e. controlled by the activity of the remaining muscles after amputation, are encouraging because they help maintain the hope of an intuitive prosthesis. A particular phenomenon, occurring in the majority of amputees, is the presence of phantom limb sensations. Phantom limb sensations are of many types: thermal, pain, and mobility. Phantom limb mobilities are particularly interesting for the development of myoelectric prostheses since it has been shown that they produce an electromyographic (EMG) activity in the amputated limb. However, the studies focusing on the detection of phantom movements recorded EMG from muscles that are hard to integrate into the socket element of a myoelectric prosthesis, such as the back, chest and shoulder muscles. This study focuses on the classification of phantom movements in transhumeral amputees using EMG in the context of developing a myoelectric prosthesis controlled by pattern recognition. Five adults who underwent unilateral humeral amputation following a trauma participated in this study. The EMG activity of the participants was recorded exclusively around their stump. During the recordings, participants were asked to perform one of the main upper limb phantom movements: flexion or extension of the elbow, pronation or supination of the forearm, flexion or extension of the wrist, opening or closing the hand and rest. Each phantom movement was to be made symmetrical with the unaffected arm and the kinematics of the latter was recorded using a motion analysis system. Ten time-domain features were extracted from the EMG signals and used to train a neural network to classify the phantom limb movements. The performance of the classifier was evaluated based on the number of movements studied and an optimal set of four EMG features was determined. The impact of kinematic information on the classification performance was also evaluated. The accuracy of the classification varies from one amputee to another, but some trends are common: performance decreases if the number of degrees of freedom considered in the classification increases and/or if the phantom movements become more distal. Moreover, the optimal set of four EMG features provided a performance equivalent to that obtained with all ten EMG features. The addition of the kinematic information improved classification accuracy for all amputees

Novel Bidirectional Body - Machine Interface to Control Upper Limb Prosthesis

Objective. The journey of a bionic prosthetic user is characterized by the opportunities and limitations involved in adopting a device (the prosthesis) that should enable activities of daily living (ADL). Within this context, experiencing a bionic hand as a functional (and, possibly, embodied) limb constitutes the premise for mitigating the risk of its abandonment through the continuous use of the device. To achieve such a result, different aspects must be considered for making the artificial limb an effective support for carrying out ADLs. Among them, intuitive and robust control is fundamental to improving amputees’ quality of life using upper limb prostheses. Still, as artificial proprioception is essential to perceive the prosthesis movement without constant visual attention, a good control framework may not be enough to restore practical functionality to the limb. To overcome this, bidirectional communication between the user and the prosthesis has been recently introduced and is a requirement of utmost importance in developing prosthetic hands. Indeed, closing the control loop between the user and a prosthesis by providing artificial sensory feedback is a fundamental step towards the complete restoration of the lost sensory-motor functions. Within my PhD work, I proposed the development of a more controllable and sensitive human-like hand prosthesis, i.e., the Hannes prosthetic hand, to improve its usability and effectiveness. Approach. To achieve the objectives of this thesis work, I developed a modular and scalable software and firmware architecture to control the Hannes prosthetic multi-Degree of Freedom (DoF) system and to fit all users’ needs (hand aperture, wrist rotation, and wrist flexion in different combinations). On top of this, I developed several Pattern Recognition (PR) algorithms to translate electromyographic (EMG) activity into complex movements. However, stability and repeatability were still unmet requirements in multi-DoF upper limb systems; hence, I started by investigating different strategies to produce a more robust control. To do this, EMG signals were collected from trans-radial amputees using an array of up to six sensors placed over the skin. Secondly, I developed a vibrotactile system to implement haptic feedback to restore proprioception and create a bidirectional connection between the user and the prosthesis. Similarly, I implemented an object stiffness detection to restore tactile sensation able to connect the user with the external word. This closed-loop control between EMG and vibration feedback is essential to implementing a Bidirectional Body - Machine Interface to impact amputees’ daily life strongly. For each of these three activities: (i) implementation of robust pattern recognition control algorithms, (ii) restoration of proprioception, and (iii) restoration of the feeling of the grasped object's stiffness, I performed a study where data from healthy subjects and amputees was collected, in order to demonstrate the efficacy and usability of my implementations. In each study, I evaluated both the algorithms and the subjects’ ability to use the prosthesis by means of the F1Score parameter (offline) and the Target Achievement Control test-TAC (online). With this test, I analyzed the error rate, path efficiency, and time efficiency in completing different tasks. Main results. Among the several tested methods for Pattern Recognition, the Non-Linear Logistic Regression (NLR) resulted to be the best algorithm in terms of F1Score (99%, robustness), whereas the minimum number of electrodes needed for its functioning was determined to be 4 in the conducted offline analyses. Further, I demonstrated that its low computational burden allowed its implementation and integration on a microcontroller running at a sampling frequency of 300Hz (efficiency). Finally, the online implementation allowed the subject to simultaneously control the Hannes prosthesis DoFs, in a bioinspired and human-like way. In addition, I performed further tests with the same NLR-based control by endowing it with closed-loop proprioceptive feedback. In this scenario, the results achieved during the TAC test obtained an error rate of 15% and a path efficiency of 60% in experiments where no sources of information were available (no visual and no audio feedback). Such results demonstrated an improvement in the controllability of the system with an impact on user experience. Significance. The obtained results confirmed the hypothesis of improving robustness and efficiency of a prosthetic control thanks to of the implemented closed-loop approach. The bidirectional communication between the user and the prosthesis is capable to restore the loss of sensory functionality, with promising implications on direct translation in the clinical practice