Deep Learning Based Abnormal Gait Classification System Study with Heterogeneous Sensor Network

Gait is one of the important biological characteristics of the human body. Abnormal gait is mostly related to the lesion site and has been demonstrated to play a guiding role in clinical research such as medical diagnosis and disease prevention. In order to promote the research of automatic gait pattern recognition, this paper introduces the research status of abnormal gait recognition and systems analysis of the common gait recognition technologies. Based on this, two gait information extraction methods, sensor-based and vision-based, are studied, including wearable system design and deep neural network-based algorithm design. In the sensor-based study, we proposed a lower limb data acquisition system. The experiment was designed to collect acceleration signals and sEMG signals under normal and pathological gaits. Specifically, wearable hardware-based on MSP430 and upper computer software based on Labview is designed. The hardware system consists of EMG foot ring, high-precision IMU and pressure-sensitive intelligent insole. Data of 15 healthy persons and 15 hemiplegic patients during walking were collected. The classification of gait was carried out based on sEMG and the average accuracy rate can reach 92.8% for CNN. For IMU signals five kinds of abnormal gait are trained based on three models: BPNN, LSTM, and CNN. The experimental results show that the system combined with the neural network can classify different pathological gaits well, and the average accuracy rate of the six-classifications task can reach 93%. In vision-based research, by using human keypoint detection technology, we obtain the precise location of the key points through the fusion of thermal mapping and offset, thus extracts the space-time information of the key points. However, the results show that even the state-of-the-art is not good enough for replacing IMU in gait analysis and classification. The good news is the rhythm wave can be observed within 2 m, which proves that the temporal and spatial information of the key points extracted is highly correlated with the acceleration information collected by IMU, which paved the way for the visual-based abnormal gait classification algorithm.步态指人走路时表现出来的姿态，是人体重要生物特征之一。异常步态多与病变部位有关，作为反映人体健康状况和行为能力的重要特征，其被论证在医疗诊断、疾病预防等临床研究中具有指导作用。为了促进步态模式自动识别的研究，本文介绍了异常步态识别的研究现状，系统地分析了常见步态识别技术以及算法，以此为基础研究了基于传感器与基于视觉两种步态信息提取方法，内容包括可穿戴系统设计与基于深度神经网络的算法设计。 在基于传感器的研究中，本工作开发了下肢步态信息采集系统，并利用该信息采集系统设计实验，采集正常与不同病理步态下的加速度信号与肌电信号，搭建深度神经网络完成分类任务。具体的，在系统搭建部分设计了基于MSP430的可穿戴硬件设备以及基于Labview的上位机软件，该硬件系统由肌电脚环，高精度IMU以及压感智能鞋垫组成，该上位机软件接收、解包蓝牙数据并计算出步频步长等常用步态参数。 在基于运动信号与基于表面肌电的研究中，采集了15名健康人与15名偏瘫病人的步态数据，并针对表面肌电信号训练卷积神经网络进行帕金森步态的识别与分类，平均准确率可达92.8%。针对运动信号训练了反向传播神经网络，LSTM以及卷积神经网络三种模型进行五种异常步态的分类任务。实验结果表明，本工作中步态信息采集系统结合神经网络模型，可以很好地对不同病理步态进行分类，六分类平均正确率可达93%。 在基于视觉的研究中，本文利用人体关键点检测技术，首先检测出图片中的一个或多个人，接着对边界框做图像分割，接着采用全卷积resnet对每一个边界框中的人物的主要关节点做热力图并分析偏移量，最后通过热力图与偏移的融合得到关键点的精确定位。通过该算法提取了不同步态下姿态关键点时空信息，为基于视觉的步态分析系统提供了基础条件。但实验结果表明目前最高准确率的人体关键点检测算法不足以替代IMU实现步态分析与分类。但在2m之内可以观察到节律信息，证明了所提取的关键点时空信息与IMU采集的加速度信息呈现较高相关度，为基于视觉的异常步态分类算法铺平了道路

Evaluation of Surgical Skill Using Machine Learning with Optimal Wearable Sensor Locations

Evaluation of surgical skills during minimally invasive surgeries is needed when recruiting new surgeons. Although surgeons’ differentiation by skill level is highly complex, performance in specific clinical tasks such as pegboard transfer and knot tying could be determined using wearable EMG and accelerometer sensors. A wireless wearable platform has made it feasible to collect movement and muscle activation signals for quick skill evaluation during surgical tasks. However, it is challenging since the placement of multiple wireless wearable sensors may interfere with their performance in the assessment. This study utilizes machine learning techniques to identify optimal muscles and features critical for accurate skill evaluation. This study enrolled a total of twenty-six surgeons of different skill levels: novice (n = 11), intermediaries (n = 12), and experts (n = 3). Twelve wireless wearable sensors consisting of surface EMGs and accelerometers were placed bilaterally on bicep brachii, tricep brachii, anterior deltoid, flexor carpi ulnaris (FCU), extensor carpi ulnaris (ECU), and thenar eminence (TE) muscles to assess muscle activations and movement variability profiles. We found features related to movement complexity such as approximate entropy, sample entropy, and multiscale entropy played a critical role in skill level identification. We found that skill level was classified with highest accuracy by i) ECU for Random Forest Classifier (RFC), ii) deltoid for Support Vector Machines (SVM) and iii) biceps for Naïve Bayes Classifier with classification accuracies 61%, 57% and 47%. We found RFC classifier performed best with highest classification accuracy when muscles are combined i) ECU and deltoid (58%), ii) ECU and biceps (53%), and iii) ECU, biceps and deltoid (52%). Our findings suggest that quick surgical skill evaluation is possible using wearables sensors, and features from ECU, deltoid, and biceps muscles contribute an important role in surgical skill evaluation

Recent Advances in Motion Analysis

The advances in the technology and methodology for human movement capture and analysis over the last decade have been remarkable. Besides acknowledged approaches for kinematic, dynamic, and electromyographic (EMG) analysis carried out in the laboratory, more recently developed devices, such as wearables, inertial measurement units, ambient sensors, and cameras or depth sensors, have been adopted on a wide scale. Furthermore, computational intelligence (CI) methods, such as artificial neural networks, have recently emerged as promising tools for the development and application of intelligent systems in motion analysis. Thus, the synergy of classic instrumentation and novel smart devices and techniques has created unique capabilities in the continuous monitoring of motor behaviors in different fields, such as clinics, sports, and ergonomics. However, real-time sensing, signal processing, human activity recognition, and characterization and interpretation of motion metrics and behaviors from sensor data still representing a challenging problem not only in laboratories but also at home and in the community. This book addresses open research issues related to the improvement of classic approaches and the development of novel technologies and techniques in the domain of motion analysis in all the various fields of application

Wearable fusion system for assessment of motor function in lesion-symptom mapping studies

Lesion-symptom mapping studies are a critical component of addressing the relationship between brain and behaviour. Recent developments have yielded significant improvements in the imaging and detection of lesion profiles, but the quantification of motor outcomes is still largely performed by subjective and low-resolution standard clinical rating scales. This mismatch means than lesion-symptom mapping studies are limited in scope by scores which lack the necessary accuracy to fully quantify the subcomponents of motor function. The first study conducted aimed to develop a new automated system of motor function which addressed the limitations inherent in the clinical rating scales. A wearable fusion system was designed that included the attachment of inertial sensors to record the kinematics of upper extremity. This was combined with the novel application of mechanomyographic sensors in this field, to enable the quantification of hand/wrist function. Novel outputs were developed for this system which aimed to combine the validity of the clinical rating scales with the high accuracy of measurements possible with a wearable sensor system. This was achieved by the development of a sophisticated classification model which was trained on series of kinematic and myographic measures to classify the clinical rating scale. These classified scores were combined with a series of fine-grained clinical features derived from higher-order sensor metrics. The developed automated system graded the upper-extremity tasks of the Fugl-Meyer Assessment with a mean accuracy of 75\% for gross motor tasks and 66\% for the wrist/hand tasks. This accuracy increased to 85\% and 74\% when distinguishing between healthy and impaired function for each of these tasks. Several clinical features were computed to describe the subcomponents of upper extremity motor function. This fine-grained clinical feature set offers a novel means to complement the low resolution but well-validated standardised clinical rating scales. A second study was performed to utilise the fine-grained clinical feature set calculated in the previous study in a large-scale region-of-interest lesion-symptom mapping study. Statistically significant regions of motor dysfunction were found in the corticospinal tract and the internal capsule, which are consistent with other motor-based lesion-symptom mapping studies. In addition, the cortico-ponto-cerebellar tract was found to be statistically significant when testing with a clinical feature of hand/wrist motor function. This is a novel finding, potentially due to prior studies being limited to quantifying this subcomponent of motor function using standard clinical rating scales. These results indicate the validity and potential of the clinical feature set to provide a more detailed picture of motor dysfunction in lesion-symptom mapping studies.Open Acces

Modeling and Simulation of Lower Limb Spasticity in Motor-Impaired Individuals

Spasticity is a symptom that impairs the ability to freely move and control one’s limbs through increased tone and involuntary activations in the muscles. It can cause pain and discomfort and interfere with daily life and activities such as walking. Spasticity is a result of upper motor neuron lesions and is seen commonly in survivors of stroke and brain trauma, and individuals with cerebral palsy, multiple sclerosis, and spinal cord injuries. Despite its ubiquity the phenomena is not well understood. However, the most referred to definition describes spasticity as “a velocity-dependent increase in tonic stretch reflexes with exaggerated tendon jerks, resulting from hyper-excitability of the stretch reflexes.” Qualitative, subjective measures are commonly used in the clinical setting to assess spasticity, most notably the Modified Ashworth score, which has been shown to have inconsistent reliability, relying heavily on the examiner’s experience, and is inaccurate for the lower limbs. Furthermore, these subjective scores do not account for the velocity-dependence of spasticity, which is a key differentiator against other symptoms such as rigidity. Consequently, there is a need for an objective measure of spasticity that can provide a more accurate and reliable alternative or supplement to the current clinical practice, in order to improve the evaluation of treatment and rehabilitation for spasticity. To address this need, a system was developed, validated and applied for modeling the spasticity in the lower-limbs of an affected individual. An experimental setup consisted of a brace-handle system with integrated force sensors for passive actuation of each leg segment, stretching spastic muscles to assess the severity of the condition. The setup included wearable sensors sEMG and IMUs – recording muscular activity and limb segment kinematics respectively during these motions. From the data, onsets of muscular activity and subsequently the trigger points of spastic reflexes were identified, which were mapped onto the calculated joint kinematics. Based on threshold-control theory, stretch reflex threshold (SRT) models of spasticity were created for each muscle by plotting the joint velocities and positions and using regression analysis to create a dynamic threshold in the kinematic space that divided the regimes of spastic and non-spastic motion. These muscle-specific models were combined by muscle groups, leading to the creation of a novel, data-based measure that characterizes the severity of spasticity of a group of muscles. The models and measures were found to agree with the expected changes from different conditions of muscle stretch, and different levels of spasticity in the included subjects, but required more data for statistical validation. The muscle-specific models were then implemented in a spasticity controller developed for use in neuromuscular simulations, in addition to further modeling of spastic reflex characteristics. The controller was applied in a scenario simulation of the same passive movement spasticity assessments used to collect the original data, which provided additional validation of the methodology and results of the modeling. The spasticity controller was also applied in a previously developed reinforcement-learning walking agent, to see the effects of spasticity on simulated gait. Following modification and training of the new agents, the spatio-temporal parameters of gait were analyzed to determine the differences in healthy and spastic gait, which agreed with expectations and further validated the spasticity modeling. This thesis presents a system to accurately and reliably model spasticity, establishing a novel, objective measure to better characterize spasticity, validating it through demonstrations of its use that may be extended in future work to accomplish better understanding of spasticity and provide invaluable improvements to the lives of affected individuals through practical applications

Développement d’un outil automatique d’aide au diagnostic pour les enfants atteints de paralysie cérébrale en réadaptation robotique

La paralysie cérébrale représente l’infirmité la plus courante chez les enfants, affectant le mouvement, la coordination et la tonicité musculaire. Cette atteinte peut avoir des effets dévastateurs sur le développement des enfants, s’accompagnant d’une grande difficulté pour accomplir les tâches de la vie quotidienne. Les interventions médicamenteuses, psychothérapeutiques et l’adoption de nouvelles technologies d’assistance robotisée, sont des moyens qui permettent d’améliorer la qualité de vie et procurer une indépendance maximale aux enfants dont les capacités mentales le permettent. Les résultats de ces prises en charge sont généralement basés sur des outils d’évaluation subjectives qui dépendent grandement de l’avis de l’évaluateur et des facteurs environnementaux. Dans le but d’améliorer l’efficacité de ces interventions, l’objectif de cette thèse est de développer un outil de catégorisation des comportements musculaires dynamiques et les habilités motrices des enfants atteints de paralysie cérébrale. Cet objectif global est subdivisé en trois objectifs spécifiques: (1) tester la validité d’une méthode assistée par ordinateur pour la classification des niveaux fonctionnels des enfants avec paralysie cérébrale à partir des mouvements simples d’extension-flexion et de supination-pronation; (2) explorer d’autres techniques d’apprentissage machine plus avancées pour la catégorisation des habilités motrices à partir des mouvements effectués avec un dispositif d’assistance robotisé REAplan; et (3) comparer les comportements musculaires dynamiques aux membres supérieurs entre les enfants atteints de paralysie cérébrale avant et après la rééducation afin de valider l’efficacité du REAPlan comme outil de réadaptation. Parmi les résultats, une bonne corrélation a été trouvée entre les niveaux de sévérité établis par l’échelle « Manuel Ability Classification System » (MACS) et les niveaux de sévérité issus de la méthode de classification. En outre, il a été possible de différencier les enfants avec un développement normal, des enfants avec paralysie cérébrale pré-thérapie et post-thérapie, avec une précision globale de 97,6%. Par la suite, un indice quantitatif « Upper Limb Motor Function Index » (ULMFI) a été calculé à partir des paramètres électromyographiques et accélérométriques les plus pertinents pour distinguer les trois groupes d’enfants. L'ULMFI a montré des différences significatives entre le groupe avec un développement normal et les enfants avec paralysie cérébrale pré et post-thérapie assistée par robot. Les résultats de cette thèse suggèrent que les coûts et les efforts nécessaires pour évaluer et caractériser le niveau de limitation d'un enfant atteint de paralysie cérébrale, peuvent être davantage réduits avec des techniques d'apprentissage machine. Comme perspective, cette méthode pourra aussi être appliquée à d’autres populations atteintes de maladies neuromusculaires et de déficits moteurs cérébraux, afin de mieux cibler les muscles atteints avec des traitements spécifiques et d'améliorer le diagnostic médical.----------ABSTRACT Cerebral palsy is the most common disability in children, affecting movement, coordination and muscle tone. This disability has a devastating effect on children development, on their quality of life and impacts their ability to perform everyday tasks. The use of appropriate combinations of medical and psychotherapeutic interventions and the adoption of the assistive robotic devices could improve the independence and quality of life of children with cerebral palsy whose mental abilities allow it. The effects of these interventions are generally assessed based on the perspective of the therapist evaluating the child and the environmental factors. To improve the effectiveness of these interventions, the aim of this thesis was to develop a computerized method to estimate the disability levels of children with cerebral palsy. To do so our three specific objectives were: (1) validating a computerized method to classify disability levels of children with cerebral palsy during two main movements of upper extremity: extension-flexion and supination-pronation; (2) testing a more advanced machine learning techniques to categorize motor skills using an assistive robotic device REAplan; and (3) comparing dynamic muscle behavior in upper limbs between children with cerebral palsy before and after the intervention to validate the effectiveness of REAPlan as a rehabilitation tool. A good correlation was found between the severity levels fixed by the « Manual Ability Classification System » (MACS) and the obtained classes. In addition, it was possible to differentiate children with typical development from children with cerebral palsy pre-therapy and post-therapy, with an overall accuracy of 97.6%. Thereafter, a quantitative « Upper Limb Motor Function Index » (ULMFI) was calculated from the most relevant electromyographic and accelerometric parameters to distinguish between the three groups of children. The ULMFI was able to differentiate between children with typical development and children with cerebral palsy pre and post Robot-Assisted Therapy (Robot-AT). The results of this thesis suggest that the cost and effort needed to assess and characterize the disability level of a child with cerebral palsy can be further reduced using machine learning techniques. As a perspective, the proposed assessment method can also be applied to other populations with neuromuscular diseases and cerebral motor deficits, to identify the muscles to target with specific treatments and improve medical diagnosis

A Regression-Based Framework for Quantitative Assessment of Muscle Spasticity Using Combined EMG and Inertial Data From Wearable Sensors

There have always been practical demands for objective and accurate assessment of muscle spasticity beyond its clinical routine. A novel regression-based framework for quantitative assessment of muscle spasticity is proposed in this paper using wearable surface electromyogram (EMG) and inertial sensors combined with a simple examination procedure. Sixteen subjects with elbow flexor or extensor (i.e., biceps brachii muscle or triceps brachii muscle) spasticity and eight healthy subjects were recruited for the study. The EMG and inertial data were recorded from each subject when a series of passive elbow stretches with different stretch velocities were conducted. In the proposed framework, both lambda model and kinematic model were constructed from the recorded data, and biomarkers were extracted respectively from the two models to describe the neurogenic component and biomechanical component of the muscle spasticity, respectively. Subsequently, three evaluation methods using supervised machine learning algorithms including single-/multi-variable linear regression and support vector regression (SVR) were applied to calibrate biomarkers from each single model or combination of two models into evaluation scores. Each of these evaluation scores can be regarded as a prediction of the modified Ashworth scale (MAS) grade for spasticity assessment with the same meaning and clinical interpretation. In order to validate performance of three proposed methods within the framework, a 24-fold leave-one-out cross validation was conducted for all subjects. Both methods with each individual model achieved satisfactory performance, with low mean square error (MSE, 0.14 and 0.47) between the resultant evaluation score and the MAS. By contrast, the method using SVR to fuse biomarkers from both models outperformed other two methods with the lowest MSE at 0.059. The experimental results demonstrated the usability and feasibility of the proposed framework, and it provides an objective, quantitative and convenient solution to spasticity assessment, suitable for clinical, community, and home-based rehabilitation

Gait analysis in neurological populations: Progression in the use of wearables

Gait assessment is an essential tool for clinical applications not only to diagnose different neurological conditions but also to monitor disease progression as it contributes to the understanding of underlying deficits. There are established methods and models for data collection and interpretation of gait assessment within different pathologies. This narrative review aims to depict the evolution of gait assessment from observation and rating scales to wearable sensors and laboratory technologies, and provide possible future directions. In this context, we first present an extensive review of current clinical outcomes and gait models. Then, we demonstrate commercially available wearable technologies with their technical capabilities along with their use in gait assessment studies for various neurological conditions. In the next sections, a descriptive knowledge for existing inertial based algorithms and a sign based guide that shows the outcomes of previous neurological gait assessment studies are presented. Finally, we state a discussion for the use of wearables in gait assessment and speculate the possible research directions by revealing the limitations and knowledge gaps in the literature