Development of Waterloo Robotic Rollator (WATRR)

One of the major risk factors for impaired mobility is aging, and with the aging population on the rise, the demand for assistive technologies for individuals with mobility impairment is at an all time high. Impaired mobility can lead to loss of independence, increased chance of mortality, deterioration of health, decreased cognitive function and a poor quality of life. Moreover, individuals with impaired mobility also tend to have higher hospital utilization costs. Mobility capability can be (re)built through the use of assistive technologies. Rollators/Walkers are a commonly used mobility aid that has shown to help with mobility by providing support, particularly transferring a portion of the lower limb loads to the upper limbs. However, safety has been a concern with rollators, with thousands of accidents occurring every year. Currently, many research projects are investigating methods to improve rollators, particularly surrounding the use of robotic rollators. At University of Waterloo Neural and Rehabilitation lab (NRE lab), our goal is to develop technology to improve lives of people, with development of robotic rollators being one of our research foci. The Waterloo Robotic Rollators (WATRR) is an active rollator system with built-in sensing and actuation systems. It is believed that the user experience and safety of rollators can be improved through the use of smart control algorithms. The purpose of this thesis was to develop methods to address safety and user experience concerns by proposing a hybrid control approach, where distance and orientation control are key control parameters, including automatic braking. First, the Waterloo Robotic Rollator (WATRR), a low weight robotic rollator platform, representative of current rollators with sensors and actuators is presented. I describe key design decisions for the platform, offer an overview of the software architecture, and discuss further research development goals. The proposed hybrid controller is then described and both simulation and experimental data for controller design is presented. To enable the envisioned hybrid control systems, a human state estimator and a robot state estimator are required. The human state estimator uses computer vision and machine learning in a hourglass network structure to predict shoulder locations. Using the estimated location and depth data, human velocity, distance and orientation relative to the rollator are estimated. For the robot state estimator, a new velocity estimator based on learning methods is proposed. As rollator lateral velocity can be difficult to estimate with traditional methods, we propose an augmented learning-aided state estimator. This estimator is a Long- Short-Term Memory (LSTM) based estimator, augmented with an Unscented Kalman Filter (UKF). The proposed estimator was validated through experimental data. The main contribution of this thesis was a new lightweight rollator system with sensors and actuators that enabled development of advanced controls. Next, previous control systems are not only improved upon by using a new hybrid controller but also implemented on our platform. A new robot state estimator is developed that relies solely on the kinematics and is able to estimate lateral velocity with a mean error of $<10mm/s$ without requiring additional instrumentation or knowledge of the rollator's time varying parameters. Finally, a new human state estimator is designed which does not require instrumentation on the human and outperforms current estimators

A Deep Learning approach to prevent problematic movements of industrial workers based on inertial sensors

Nowadays, manufacturing industries still face difficulties applying traditional Work-related MusculoSkeletal Disorders (WMSDs) risk assessment methods due to the high effort required by a continuous data collection when using observational methods. An interesting solution is to adopt Inertial Measurement Units (IMUs) to automate the data collection, thus supporting occupational health professionals. In this paper, we propose a deep learning approach to predict human motion based on IMU data with the goal of preventing industrial worker problematic movements that can arise during repetitive actions. The proposed system includes an initial Madgwick filter to merge the raw inertial tri-axis sensor data into a single angle orientation time series. Then, a Machine Learning (ML) algorithm is trained with the obtained time series, allowing to build a forecasting model. The effectiveness of the developed system was validated by using an open-source dataset composed of different motions for the upper body collected in a laboratory environment, aiming to monitor the abduction/adduction angle of the arm. Firstly, distinct ML algorithms were compared for a single angle orientation time series prediction, including: three Long Short-Term Memory (LSTM) methods - a one layer, a stacked layer and a Sequence to Sequence (Seq2Seq) model; and three non deep learning methods - a Multiple Linear Regression, a Random Forest and a Support Vector Machine. The best results were provided by the Seq2Seq LSTM model, which was further evaluated for WMSD prevention by considering 11 human subject datasets and two evaluation procedures (single person and multiple person training and testing). Overall, interesting results were achieved, particularly for multiple person evaluation, where the proposed Seq2Seq LSTM has shown an excellent capability to anticipate problematic movements.This article is a result of the project STVgoDigital - Digitalization of the T&C sector (POCI-01-0247-FEDER-046086), supported by COMPETE 2020, under the PORTUGAL 2020 Partnership Agreement, through the European Regional De velopment Fund (ERDF)

A deep learning solution for real-time human motion decoding in smart walkers

Dissertação de mestrado integrado em Engenharia Biomédica (especialização em Eletrónica Médica)The treatment of gait impairments has increasingly relied on rehabilitation therapies which benefit from the use of smart walkers. These walkers still lack advanced and seamless Human-Robot Interaction, which intuitively understands the intentions of human motion, empowering the user’s recovery state and autonomy, while reducing the physician’s effort. This dissertation proposes the development of a deep learning solution to tackle the human motion decoding problematic in smart walkers, using only lower body vision information from a camera stream, mounted on the WALKit Smart Walker, a smart walker prototype for rehabilitation purposes. Different deep learning frameworks were designed for early human motion recognition and detec tion. A custom acquisition method, including a smart walker’s automatic driving algorithm and labelling procedure, was also designed to enable further training and evaluation of the proposed frameworks. Facing a 4-class (stop, walk, turn right/left) classification problem, a deep learning convolutional model with an attention mechanism achieved the best results: an offline f1-score of 99.61%, an online calibrated instantaneous precision higher than 97% and a human-centred focus slightly higher than 30%. Promising results were attained for early human motion detection, with enhancements in the focus of the proposed architectures. However, further improvements are still needed to achieve a more reliable solution for integration in a smart walker’s control strategy, based in the human motion intentions.O tratamento de distúrbios da marcha tem apostado cada vez mais em terapias de reabilitação que beneficiam do uso de andarilhos inteligentes. Estes ainda carecem de uma Interação Humano-Robô avançada e eficaz, capaz de entender, intuitivamente, as intenções do movimento humano, fortalecendo a recuperação autónoma do paciente e reduzindo o esforço médico. Esta dissertação propõe o desenvolvimento de uma solução de aprendizagem para o problema de descodificação de movimento humano em andarilhos inteligentes, usando apenas vídeos recolhidos pelo WALKit Smart Walker, um protótipo de andarilho inteligente usado para reabilitação. Foram desenvolvidos algoritmos de aprendizagem para o reconhecimento e detecção precoces de movimento humano. Um método de aquisição personalizado, incluindo um algoritmo de condução e labelização automatizados, foi projetado para permitir o conseguinte treino e avaliação dos algoritmos propostos. Perante a classificação de 4 ações (parar, andar, virar à direita/esquerda), um modelo convolucional com um mecanismo de atenção alcançou os melhores resultados: f1-score offline de 99,61%, precisão instantânea calibrada online de superior a 97 % e um foco centrado no ser humano ligeiramente superior a 30%. Com esta dissertação alcançaram-se resultados promissores para a detecção precoce de movimento humano, com aprimoramentos no foco dos algoritmos propostos. No entanto, ainda são necessárias melhorias adicionais para alcançar uma solução mais robusta para a integração na estratégia de controlo de um andarilho inteligente, com base nas intenções de movimento do utilizador

First advances in near fall detection and prediction when using a walker

Dissertação de mestrado integrado em Engenharia Biomédica (área de especialização em Eletrónica Médica)Falls are a major concern to society. Several injuries associated with falls need medical care, and in the worst-case scenario, a fall can lead to death. These consequences have a high cost for the population. In order to overcome these problems, a diversity of approaches for detection, prediction, and prevention of falls have been tackled. Walkers are often prescribed to subjects who present a higher risk of falling. Thus, it is essential to develop strategies to enhance the user's safety in an imminent danger situation. In this sense, this dissertation aims to develop a strategy to detect a near fall (NF) and its direction as well as the detection of incipient near fall (INF) while the subject uses a walker. Furthermore, it has the purpose of detecting two gait events, the heel strike (HS) and the toe-off (TO). The strategies established, in this work, were based on the data gathered through an inertial sensor placed on the lower trunk and force sensors placed on the insoles. Following data collection, the methodology adopted to identify the situations aforementioned was based on machine learning algorithms. In order to reach the model with best performance, many combinations of different classifiers were tested with three feature selection methods. Regarding the detection of NF, the results achieved presented a Matthews Correlation Coefficient (MCC) of 79.99% being possible to detect a NF 1.76±0.76s before its end. With the implementation of the post-processing algorithm, a large part of the false positives was eliminated being able to detect all NF 1.48±0.68s before its end. Concerning the models built to distinguish the direction of the NF, the best model presented accuracy of 58.97% being unable to reliably distinguish the three fall directions. The methodology followed, in this work, was unsuccessful to detect an INF. The best model presented MCC=23.87%, in this case. Lastly, with respect to the detection of HS and TO events the best model reached MCC=86.94%. With the application of the post-processing algorithm, part of misclassified samples was eliminated, however, a delay in the detection of the HS and TO events was introduced. With the post-processing it was possible to reach MCC=88.82%, not including the imposed delay.As quedas representam uma grande preocupação para a sociedade. Várias lesões associadas às quedas necessitam de cuidados médicos e, no pior dos casos, uma queda pode levar à morte. Estas consequências traduzem-se em custos elevados para a população. A fim de ultrapassar estes problemas, várias abordagens têm sido endereçadas para deteção, previsão e prevenção das quedas. Os andarilhos são muitas vezes prescritos a sujeitos que apresentam um risco de queda maior. Desta forma, é essencial desenvolver estratégias para aumentar a segurança do utilizador perante uma situação de perigo iminente. Neste sentido, esta dissertação visa desenvolver uma estratégia que permita a deteção de uma quase queda (NF) e a sua direção, assim como a deteção incipiente de uma NF (INF). Para além disso, tem o objetivo de detetar dois eventos de marcha, o heel strike (HS) e o toe-off (TO). As estratégias definidas, neste trabalho, basearam-se nos dados recolhidos através de um sensor inercial posicionado no tronco inferior e de sensores de força colocados nas palmilhas. Após a aquisição dos dados, a metodologia adotada para identificar as situações anteriormente referidas foi baseada em algoritmos de machine learning. Com o intuito de obter o modelo com melhor desempenho, várias combinações de diferentes classificadores foram testadas com três métodos de seleção de features. No que concerne à deteção da NF, os resultados alcançados apresentaram um Matthews Correlation Coefficient (MCC) de 79.99% sendo possível detetar uma NF 1.76±0.76s antes do seu final. Com a implementação do algoritmo de pós-processamento, grande parte dos falsos positivos foram eliminados, sendo possível detetar todas as NF 1.48±0.68s antes do seu final. Em relação aos modelos construídos para distinguir a direção da NF, o melhor modelo apresentou uma precisão (ACC) de 59.97%. A metodologia seguida neste trabalho não foi bem sucedida na deteção INF. O melhor modelo apresentou um MCC=23.87%. Relativamente à deteção dos eventos, HS e TO, o melhor modelo atingiu um MCC=86.94%. Com a aplicação do algoritmo de pós-processamento parte das amostras mal classificadas foram eliminadas, no entanto, foi introduzido um atraso na deteção do HS e do TO. Com o pós-processamento foi possível obter um MCC=88.82%, não incluindo o atraso imposto pelo pós-processamento

Walk-IT: An Open-Source Modular Low-Cost Smart Rollator

Rollators are widely used in clinical rehabilitation for gait assessment, but gait analysis usually requires a great deal of expertise and focus from medical staff. Smart rollators can capture gait parameters autonomously while avoiding complex setups. However, commercial smart rollators, as closed systems, can not be modified; plus, they are often expensive and not widely available. This work presents a low cost open-source modular rollator for monitorization of gait parameters and support. The whole system is based on commercial components and its software architecture runs over ROS2 to allow further customization and expansion. This paper describes the overall software and hardware architecture and, as an example of extended capabilities, modules for monitoring dynamic partial weight bearing and for estimation of spatiotemporal gait parameters of clinical interest. All presented tests are coherent from a clinical point of view and consistent with input dataThis work is funded by Programa Proyectos RETOS del Ministerio de Ciencia, Innovación y Universidades, Ref: RTI2018-096701-B-C21 (SAVIA: Sistema de Autonomía Variable para movIlidad Asistida) and Plan Propio de la Universidad de Málaga (E3-PROYECTOS DE PRUEBA DE CONCEPTO (E3/02/18)) at Málaga University, Spain. Partial funding for open access charge: Universidad de Málaga

Wearable and BAN Sensors for Physical Rehabilitation and eHealth Architectures

The demographic shift of the population towards an increase in the number of elderly citizens, together with the sedentary lifestyle we are adopting, is reflected in the increasingly debilitated physical health of the population. The resulting physical impairments require rehabilitation therapies which may be assisted by the use of wearable sensors or body area network sensors (BANs). The use of novel technology for medical therapies can also contribute to reducing the costs in healthcare systems and decrease patient overflow in medical centers. Sensors are the primary enablers of any wearable medical device, with a central role in eHealth architectures. The accuracy of the acquired data depends on the sensors; hence, when considering wearable and BAN sensing integration, they must be proven to be accurate and reliable solutions. This book is a collection of works focusing on the current state-of-the-art of BANs and wearable sensing devices for physical rehabilitation of impaired or debilitated citizens. The manuscripts that compose this book report on the advances in the research related to different sensing technologies (optical or electronic) and body area network sensors (BANs), their design and implementation, advanced signal processing techniques, and the application of these technologies in areas such as physical rehabilitation, robotics, medical diagnostics, and therapy