Deep Learning Based Abnormal Gait Classification System Study with Heterogeneous Sensor Network

Gait is one of the important biological characteristics of the human body. Abnormal gait is mostly related to the lesion site and has been demonstrated to play a guiding role in clinical research such as medical diagnosis and disease prevention. In order to promote the research of automatic gait pattern recognition, this paper introduces the research status of abnormal gait recognition and systems analysis of the common gait recognition technologies. Based on this, two gait information extraction methods, sensor-based and vision-based, are studied, including wearable system design and deep neural network-based algorithm design. In the sensor-based study, we proposed a lower limb data acquisition system. The experiment was designed to collect acceleration signals and sEMG signals under normal and pathological gaits. Specifically, wearable hardware-based on MSP430 and upper computer software based on Labview is designed. The hardware system consists of EMG foot ring, high-precision IMU and pressure-sensitive intelligent insole. Data of 15 healthy persons and 15 hemiplegic patients during walking were collected. The classification of gait was carried out based on sEMG and the average accuracy rate can reach 92.8% for CNN. For IMU signals five kinds of abnormal gait are trained based on three models: BPNN, LSTM, and CNN. The experimental results show that the system combined with the neural network can classify different pathological gaits well, and the average accuracy rate of the six-classifications task can reach 93%. In vision-based research, by using human keypoint detection technology, we obtain the precise location of the key points through the fusion of thermal mapping and offset, thus extracts the space-time information of the key points. However, the results show that even the state-of-the-art is not good enough for replacing IMU in gait analysis and classification. The good news is the rhythm wave can be observed within 2 m, which proves that the temporal and spatial information of the key points extracted is highly correlated with the acceleration information collected by IMU, which paved the way for the visual-based abnormal gait classification algorithm.步态指人走路时表现出来的姿态，是人体重要生物特征之一。异常步态多与病变部位有关，作为反映人体健康状况和行为能力的重要特征，其被论证在医疗诊断、疾病预防等临床研究中具有指导作用。为了促进步态模式自动识别的研究，本文介绍了异常步态识别的研究现状，系统地分析了常见步态识别技术以及算法，以此为基础研究了基于传感器与基于视觉两种步态信息提取方法，内容包括可穿戴系统设计与基于深度神经网络的算法设计。 在基于传感器的研究中，本工作开发了下肢步态信息采集系统，并利用该信息采集系统设计实验，采集正常与不同病理步态下的加速度信号与肌电信号，搭建深度神经网络完成分类任务。具体的，在系统搭建部分设计了基于MSP430的可穿戴硬件设备以及基于Labview的上位机软件，该硬件系统由肌电脚环，高精度IMU以及压感智能鞋垫组成，该上位机软件接收、解包蓝牙数据并计算出步频步长等常用步态参数。 在基于运动信号与基于表面肌电的研究中，采集了15名健康人与15名偏瘫病人的步态数据，并针对表面肌电信号训练卷积神经网络进行帕金森步态的识别与分类，平均准确率可达92.8%。针对运动信号训练了反向传播神经网络，LSTM以及卷积神经网络三种模型进行五种异常步态的分类任务。实验结果表明，本工作中步态信息采集系统结合神经网络模型，可以很好地对不同病理步态进行分类，六分类平均正确率可达93%。 在基于视觉的研究中，本文利用人体关键点检测技术，首先检测出图片中的一个或多个人，接着对边界框做图像分割，接着采用全卷积resnet对每一个边界框中的人物的主要关节点做热力图并分析偏移量，最后通过热力图与偏移的融合得到关键点的精确定位。通过该算法提取了不同步态下姿态关键点时空信息，为基于视觉的步态分析系统提供了基础条件。但实验结果表明目前最高准确率的人体关键点检测算法不足以替代IMU实现步态分析与分类。但在2m之内可以观察到节律信息，证明了所提取的关键点时空信息与IMU采集的加速度信息呈现较高相关度，为基于视觉的异常步态分类算法铺平了道路

Recent Advances in Motion Analysis

The advances in the technology and methodology for human movement capture and analysis over the last decade have been remarkable. Besides acknowledged approaches for kinematic, dynamic, and electromyographic (EMG) analysis carried out in the laboratory, more recently developed devices, such as wearables, inertial measurement units, ambient sensors, and cameras or depth sensors, have been adopted on a wide scale. Furthermore, computational intelligence (CI) methods, such as artificial neural networks, have recently emerged as promising tools for the development and application of intelligent systems in motion analysis. Thus, the synergy of classic instrumentation and novel smart devices and techniques has created unique capabilities in the continuous monitoring of motor behaviors in different fields, such as clinics, sports, and ergonomics. However, real-time sensing, signal processing, human activity recognition, and characterization and interpretation of motion metrics and behaviors from sensor data still representing a challenging problem not only in laboratories but also at home and in the community. This book addresses open research issues related to the improvement of classic approaches and the development of novel technologies and techniques in the domain of motion analysis in all the various fields of application

An automatic wearable multi-sensor based gait analysis system for older adults.

Gait abnormalities in older adults are very common in clinical practice. They lead to serious adverse consequences such as falls and injury resulting in increased care cost. There is therefore a national imperative to address this challenge. Currently gait assessment is done using standardized clinical tools dependent on subjective evaluation. More objective gold standard methods (motion capture systems such as Qualisys and Vicon) to analyse gait rely on access to expensive complex equipment based in gait laboratories. These are not widely available for several reasons including a scarcity of equipment, need for technical staff, need for patients to attend in person, complicated time consuming procedures and overall expense. To broaden the use of accurate quantitative gait monitoring and assessment, the major goal of this thesis is to develop an affordable automatic gait analysis system that will provide comprehensive gait information and allow use in clinic or at home. It will also be able to quantify and visualize gait parameters, identify gait variables and changes, monitor abnormal gait patterns of older people in order to reduce the potential for falling and support falls risk management. A research program based on conducting experiments on volunteers is developed in collaboration with other researchers in Bournemouth University, The Royal Bournemouth Hospital and care homes. This thesis consists of five different studies toward addressing our major goal. Firstly, a study on the effects on sensor output from an Inertial Measurement Unit (IMU) attached to different anatomical foot locations. Placing an IMU over the bony prominence of the first cuboid bone is the best place as it delivers the most accurate data. Secondly, an automatic gait feature extraction method for analysing spatiotemporal gait features which shows that it is possible to extract gait features automatically outside of a gait laboratory. Thirdly, user friendly and easy to interpret visualization approaches are proposed to demonstrate real time spatiotemporal gait information. Four proposed approaches have the potential of helping professionals detect and interpret gait asymmetry. Fourthly, a validation study of spatiotemporal IMU extracted features compared with gold standard Motion Capture System and Treadmill measurements in young and older adults is conducted. The results obtained from three experimental conditions demonstrate that our IMU gait extracted features are highly valid for spatiotemporal gait variables in young and older adults. In the last study, an evaluation system using Procrustes and Euclidean distance matrix analysis is proposed to provide a comprehensive interpretation of shape and form differences between individual gaits. The results show that older gaits are distinguishable from young gaits. A pictorial and numerical system is proposed which indicates whether the assessed gait is normal or abnormal depending on their total feature values. This offers several advantages: 1) it is user friendly and is easy to set up and implement; 2) it does not require complex equipment with segmentation of body parts; 3) it is relatively inexpensive and therefore increases its affordability decreasing health inequality; and 4) its versatility increases its usability at home supporting inclusivity of patients who are home bound. A digital transformation strategy framework is proposed where stakeholders such as patients, health care professionals and industry partners can collaborate through development of new technologies, value creation, structural change, affordability and sustainability to improve the diagnosis and treatment of gait abnormalities

Recognition of gait patterns in human motor disorders using a machine learning approach

Dissertação de mestrado em Industrial Electronics and Computers EngineeringWith advanced age, the occurrence of motor disturbances becomes more prevalent and can lead to gait pathologies, increasing the risk of falls. Currently, there are many available gait monitoring systems that can aid in gait disorder diagnosis by extracting relevant data from a subject’s gait. This increases the amount of data to be processed in working time. To accelerate this process and provide an objective tool for a systematic clinical diagnosis support, Machine Learning methods are a powerful addition capable of processing great amounts of data and uncover non-linear relationships in data. The purpose of this dissertation is the development of a gait pattern recognition system based on a Machine Learning approach for the support of clinical diagnosis of post-stroke gait. This includes the development of a data estimation tool capable of computing several features from inertial sensors. Four different neural networks were be added to the classification tool: Feed-Forward (FFNN), convolutional (CNN) and two recurrent neural networks (LSTM and CLSTM). The performance of all classification models was analyzed and compared in order to select the most effective method of gait analysis. The performance metric used is Matthew’s Correlation Coefficient. The classifiers that exhibit the best performance where Support Vector Machines (SVM), k-Nearest Neighbors (KNN), CNN, LSTM and CLSTM, with a Matthew’s correlation coeficient of 1 in the test set. Despite the first two classifiers reaching the same performance of the three neural networks, the later reached this performance systematically and without the need of explicit dimensionality reduction methods.Com o avançar da idade, a ocorrência de distúrbios motores torna-se mais prevalente, conduzindo a patologias na marcha e aumentando o risco de quedas. Atualmente, muitos sistemas de monitorização de marcha extraem grandes quantidades de dados biomecânicos para apoio ao diagnóstico clínico, aumentando a quantidade de dados a ser processados em tempo útil. Para acelerar esse processo e proporcionar uma ferramenta objetiva de apoio sistemático ao diagnóstico clínico, métodos de Machine Learning são uma poderosa adição, processando grandes quantidades de dados e descobrindo relações não-lineares entre dados. Esta dissertação tem o objetivo de desenvolver um sistema de reconhecimento de padrões de marcha com uma abordagem de Machine Learning para apoio ao diagnóstico clínico da marcha de vitimas de AVC. Isso inclui o desenvolvimento de uma ferramenta de estimação de dados biomecânicos e cálculo de features, a partir de sensores inerciais. Quatro redes neuronais foram implementadas numa ferramenta de classificação: uma rede Feed-Forward (FFNN), uma convolucinal (CNN), e duas redes recorrentes (LSTM e CLSTM). O desempenho de todos os modelos de classificação foi analisado. A métrica de desempenho usada é o coeficiente de correlação de Matthew. Os classificadores com melhor performance foram: Support Vector Machines (SVM), k-Nearest Neighbors (KNN), CNN, LSTM e CLSTM. Todos com uma performance igual a 1 no conjunto de teste. Apesar de os dois primeiros classificadores atingirem a mesma performance das redes neuronais, estas atingiram esta performance repetidamente e sem necessitar de métodos de redução de dimensionalidade

Inertial Measurement Unit-Based Gait Event Detection in Healthy and Neurological Cohorts: A Walk in the Dark

A deep learning (DL)-based network is developed to determine gait events from IMU data from a shank- or foot-worn device. The DL network takes as input the raw IMU data and predicts for each time step the probability that it corresponds to an initial or final contact. The algorithm is validated for walking at different self-selected speeds across multiple neurological diseases and both in clinical research settings and the habitual environment. The algorithms shows a high detection rate for initial and final contacts, and a small time error when compared to reference events obtained with an optical motion capture system or pressure insoles. Based on the excellent performance, it is concluded that the DL algorithm is well suited for continuous long-term monitoring of gait in the habitual environment

Extraction of biomedical indicators from gait videos

Gait has been an extensively investigated topic in recent years. Through the analysis of gait it is possible to detect pathologies, which makes this analysis very important to assess anomalies and, consequently, help in the diagnosis and rehabilitation of patients. There are some systems for analyzing gait, but they are usually either systems with subjective evaluations or systems used in specialized laboratories with complex equipment, which makes them very expensive and inaccessible. However, there has been a significant effort of making available simpler and more accurate systems for gait analysis and classification. This dissertation reviews recent gait analysis and classification systems, presents a new database with videos of 21 subjects, simulating 4 different pathologies as well as normal gait, and also presents a web application that allows the user to remotely access an automatic classification system and thus obtain the expected classification and heatmaps for the given input. The classification system is based on the use of gait representation images such as the Gait Energy Image (GEI) and the Skeleton Gait Energy Image (SEI), which are used as input to a VGG-19 Convolutional Neural Network (CNN) that is used to perform classification. This classification system is a vision-based system. To sum up, the developed web application aims to show the usefulness of the classification system, making it possible for anyone to access it.A marcha tem sido um tema muito investigado nos últimos anos. Através da análise da marcha é possível detetar patologias, o que torna esta análise muito importante para avaliar anómalias e consequentemente, ajudar no diagnóstico e na reabilitação dos pacientes. Existem alguns sistemas para analisar a marcha, mas habitualmente, ou estão sujeitos a uma interpretação subjetiva, ou são sistemas usados em laboratórios especializados com equipamento complexo, o que os torna muito dispendiosos e inacessíveis. No entanto, tem havido um esforço significativo com o objectivo de disponibilizar sistemas mais simples e mais precisos para análise e classificação da marcha. Esta dissertação revê os sistemas de análise e classificação da marcha desenvolvidos recentemente, apresenta uma nova base de dados com vídeos de 21 sujeitos, a simular 4 patologias diferentes bem como marcha normal, e apresenta também uma aplicação web que permite ao utilizador aceder remotamente a um sistema automático de classificação e assim, obter a classificação prevista e mapas de características respectivos de acordo com a entrada dada. O sistema de classificação baseia-se no uso de imagens de representação da marcha como a "Gait Energy Image" (GEI) e "Skeleton Gait Energy Image" (SEI), que são usadas como entrada numa rede neuronal convolucional VGG-19 que é usada para realizar a classificação. Este sistema de classificação corresponde a um sistema baseado na visão. Em suma, a aplicação web desenvolvida tem como finalidade mostrar a utilidade do sistema de classificação, tornando possível o acesso a qualquer pessoa

Gait analysis in neurological populations: Progression in the use of wearables

Gait assessment is an essential tool for clinical applications not only to diagnose different neurological conditions but also to monitor disease progression as it contributes to the understanding of underlying deficits. There are established methods and models for data collection and interpretation of gait assessment within different pathologies. This narrative review aims to depict the evolution of gait assessment from observation and rating scales to wearable sensors and laboratory technologies, and provide possible future directions. In this context, we first present an extensive review of current clinical outcomes and gait models. Then, we demonstrate commercially available wearable technologies with their technical capabilities along with their use in gait assessment studies for various neurological conditions. In the next sections, a descriptive knowledge for existing inertial based algorithms and a sign based guide that shows the outcomes of previous neurological gait assessment studies are presented. Finally, we state a discussion for the use of wearables in gait assessment and speculate the possible research directions by revealing the limitations and knowledge gaps in the literature

Wearable Movement Sensors for Rehabilitation: From Technology to Clinical Practice

This Special Issue shows a range of potential opportunities for the application of wearable movement sensors in motor rehabilitation. However, the papers surely do not cover the whole field of physical behavior monitoring in motor rehabilitation. Most studies in this Special Issue focused on the technical validation of wearable sensors and the development of algorithms. Clinical validation studies, studies applying wearable sensors for the monitoring of physical behavior in daily life conditions, and papers about the implementation of wearable sensors in motor rehabilitation are under-represented in this Special Issue. Studies investigating the usability and feasibility of wearable movement sensors in clinical populations were lacking. We encourage researchers to investigate the usability, acceptance, feasibility, reliability, and clinical validity of wearable sensors in clinical populations to facilitate the application of wearable movement sensors in motor rehabilitation

Smart Technology for Telerehabilitation: A Smart Device Inertial-sensing Method for Gait Analysis

The aim of this work was to develop and validate an iPod Touch (4th generation) as a potential ambulatory monitoring system for clinical and non-clinical gait analysis. This thesis comprises four interrelated studies, the first overviews the current available literature on wearable accelerometry-based technology (AT) able to assess mobility-related functional activities in subjects with neurological conditions in home and community settings. The second study focuses on the detection of time-accurate and robust gait features from a single inertial measurement unit (IMU) on the lower back, establishing a reference framework in the process. The third study presents a simple step length algorithm for straight-line walking and the fourth and final study addresses the accuracy of an iPod’s inertial-sensing capabilities, more specifically, the validity of an inertial-sensing method (integrated in an iPod) to obtain time-accurate vertical lower trunk displacement measures. The systematic review revealed that present research primarily focuses on the development of accurate methods able to identify and distinguish different functional activities. While these are important aims, much of the conducted work remains in laboratory environments, with relatively little research moving from the “bench to the bedside.” This review only identified a few studies that explored AT’s potential outside of laboratory settings, indicating that clinical and real-world research significantly lags behind its engineering counterpart. In addition, AT methods are largely based on machine-learning algorithms that rely on a feature selection process. However, extracted features depend on the signal output being measured, which is seldom described. It is, therefore, difficult to determine the accuracy of AT methods without characterizing gait signals first. Furthermore, much variability exists among approaches (including the numbers of body-fixed sensors and sensor locations) to obtain useful data to analyze human movement. From an end-user’s perspective, reducing the amount of sensors to one instrument that is attached to a single location on the body would greatly simplify the design and use of the system. With this in mind, the accuracy of formerly identified or gait events from a single IMU attached to the lower trunk was explored. The study’s analysis of the trunk’s vertical and anterior-posterior acceleration pattern (and of their integrands) demonstrates, that a combination of both signals may provide more nuanced information regarding a person’s gait cycle, ultimately permitting more clinically relevant gait features to be extracted. Going one step further, a modified step length algorithm based on a pendulum model of the swing leg was proposed. By incorporating the trunk’s anterior-posterior displacement, more accurate predictions of mean step length can be made in healthy subjects at self-selected walking speeds. Experimental results indicate that the proposed algorithm estimates step length with errors less than 3% (mean error of 0.80 ± 2.01cm). The performance of this algorithm, however, still needs to be verified for those suffering from gait disturbances. Having established a referential framework for the extraction of temporal gait parameters as well as an algorithm for step length estimations from one instrument attached to the lower trunk, the fourth and final study explored the inertial-sensing capabilities of an iPod Touch. With the help of Dr. Ian Sheret and Oxford Brookes’ spin-off company ‘Wildknowledge’, a smart application for the iPod Touch was developed. The study results demonstrate that the proposed inertial-sensing method can reliably derive lower trunk vertical displacement (intraclass correlations ranging from .80 to .96) with similar agreement measurement levels to those gathered by a conventional inertial sensor (small systematic error of 2.2mm and a typical error of 3mm). By incorporating the aforementioned methods, an iPod Touch can potentially serve as a novel ambulatory monitor system capable of assessing gait in clinical and non-clinical environments

Studies on gait control using a portable pneumatically powered ankle-foot orthosis (PPAFO) during human walking

A powered ankle-foot orthosis (AFO) can be very useful for people with neuromuscular injury. Control of powered AFOs will be more efficient to provide assistance to individuals with lower limb muscle impairments if we can identify different gait events during walking. A walking or gait cycle can be divided into multiple phases and sub-phases by proper gait event detection, and these phases/sub-phases are associated with one of the three main functional tasks during the gait cycle: loading response, forward propulsion, and limb advancement. The gait cycle of one limb can also be characterized by examining the limb’s behavior over one stride, which can be quantified as 0% to 100% of a gait cycle (GC). One easy approach to identify gait events is by checking whether sensor signals go above/below a predetermined threshold. By estimation of a walker’s instantaneous state, as represented by a specific percentage of the gait cycle (from states 0 to 100, which correlate with 0% to 100% GC), we can efficiently detect the various gait events more accurately. Our Human Dynamics and Controls Laboratory previously developed the portable pneumatically powered ankle-foot orthosis (PPAFO), which was capable of providing torque in both plantarflexion and dorsiflexion directions at the ankle. There were three types of sensor attached with the PPAFO (two force sensitive resistors and an angle sensor). In this dissertation, three aspects of effective control strategies for the PPAFO have been proposed. In the first study, two improved and reliable state estimators (Modified Fractional Time (MFT) and Artificial Neural Network (ANN)) were proposed for identifying when the limb with the PPAFO was at a certain percentage of the gait cycle. A correct estimation of percentage of gait cycle will assist with detecting specific gait events more accurately. The performance of new estimators was compared to a previously developed Fractional Time state estimation technique. To control a powered AFO using these estimators, however, detection of proper actuation timing is necessary. In the second study, a supervised learning algorithm to classify the appropriate start timing for plantarflexor actuation was proposed. Proper actuation timing has only been addressed in the literature in terms of functional efficiency or metabolic cost during walking. In this study, we will explore identifying the plantarflexor actuation timing in terms of biomechanics outcomes of human walking using a machine learning based algorithm. The third study investigated the recognition of different gait modes encountered during walking. The actuation scheme plays a significant role in walking on level ground, stair descent or stair ascent modes. The wrong actuation scheme for a given mode can cause falls or trips. A gait mode recognition technique was developed for detecting these different modes by attaching an inertial measurement unit and using a classifier based on artificial neural networks. This new algorithm improves upon the current one step delay limitation found as a drawback of a previously developed technique. Overall, this dissertation focused on addressing some important issues related to control of powered AFO that ultimately will help to assist people wearing the device in daily life situations during walking. The proposed approaches and algorithms introduced in this dissertation showed very promising results that proved that these methods can successfully improve the control system of powered AFOs