Musculoskeletal Estimation Using Inertial Measurement Units and Single Video Image

International audienceWe address the problem of estimating the physical burden of a human body. This translates to monitor and estimate muscle tension and joint reaction forces of a mus-culoskeletal model in real-time. The system should minimize the discomfort generating by any sensors that needs to be fixed on the user. Our system combines a 3D pose estimation from vision and IMU sensors. We aim to minimize the number of IMU fixed to the subject while compensating the remaining lack of information with vision

인간공학적 자세 평가를 위한 비디오 기반의 작업 자세 입력 시스템 개발

학위논문(석사) -- 서울대학교대학원 : 공과대학 산업공학과, 2022. 8. 윤명환.Work-related musculoskeletal disorders are a crucial problem for the worker’s safety and productivity of the workplace. The purpose of this study is to propose and develop a video-based work pose entry system for ergonomic postural assessment methods, Rapid Upper Limb Assessment(RULA) and Rapid Entire Body Assessment(REBA). This study developed a work pose entry system using the YOLOv3 algorithm for human tracking and the SPIN approach for 3D human pose estimation. The work pose entry system takes in a 2D video and scores of few evaluation items as input and outputs a final RULA or REBA score and the corresponding action level. An experiment for validation was conducted to 20 evaluators which were classified into two groups, experienced and novice, based on their level of knowledge or experience on ergonomics and musculoskeletal disorders. Participants were asked to manually evaluate working postures of 20 working videos taken at an automobile assembly plant, recording their scores on an Excel worksheet. Scores were generated by the work pose entry system based on individual items that need to be inputted, and the results of manual evaluation and results from the work pose entry system were compared. Descriptive statistics and Mann-Whitney U test showed that using the proposed work pose entry system decreased the difference and standard deviation between the groups. Also, findings showed that experienced evaluators tend to score higher than novice evaluators. Fisher’s exact test was also conducted on evaluation items that are inputted into the work pose entry system, and results have shown that some items that may seem apparent can be perceived differently between groups as well. The work pose entry system developed in this study can contribute to increasing consistency of ergonomic risk assessment and reducing time and effort of ergonomic practitioners during the process. Directions for future research on developing work pose entry systems for ergonomic posture assessment using computer vision are also suggested in the current study.작업 관련 근골격계 질환은 근로자의 안전과 작업장의 생산성 향상에 중요한 문제다. 본 연구의 목적은 인간공학적 자세 분석에 사용되는 대표적인 방법인 Rapid Upper Limb Assessment(RULA) 및 Rapid Entire Body Assessment(REBA)를 위한 비디오 기반의 작업 자세 입력 시스템을 제안하는 것이다. 본 연구는 영상 내 사람 탐지 및 추적을 위한 YOLOv3 알고리즘과 3차원 사람 자세 추정을 위한 SPIN 접근법을 사용하는 시스템을 개발했다. 해당 작업 자세 입력 시스템은 2차원 영상과 몇 개의 평가 항목 점수를 입력으로 받아 최종 RULA 또는 REBA 점수와 해당 조치수준(Action level)을 출력한다. 본 연구에서 제안하는 작업 자세 입력 시스템이 일관적인 결과를 산출하는지 알아보기 위해 인간공학 및 근골격계 질환에 대한 지식이나 경험을 기준으로 숙련된 평가자와 초보 평가자의 두 그룹으로 분류된 평가자 20명을 대상으로 검증 실험을 진행했다. 참가자들은 국내 자동차 조립 공장에서 찍은 20개의 작업 영상의 작업 자세를 수동으로 평가하여 Excel 워크시트에 점수를 기록하였다. 시스템 사용 시 입력해야 하는 개별 항목을 기준으로 시스템을 통한 점수를 생성하고 기존의 전통적인 방법으로 평가한 결과와 시스템에서 얻은 결과를 비교하였으며, 기술 통계와 Mann-Whitney U test는 제안된 시스템을 사용하면 그룹 간의 차이와 표준편차가 감소한다는 것을 보여주었다. 또한, 경험이 많은 평가자들이 초보 평가자들보다 더 높은 점수를 받는 경향이 있다는 것을 보여주었다. 시스템에 입력되는 평가 항목과 경험 정도와의 관계를 확인하기 위해 Fisher’s exact test를 수행하였으며, 결과는 명백해 보일 수 있는 일부 항목도 그룹 간에 다르게 인식될 수 있음을 보여주었다. 이 도구에서 개발된 작업 자세 입력 시스템은 인간공학적 자세 평가의 일관성을 높이고 평가 과정 중 중에 인간공학적 평가자의 시간과 노력을 줄이는 데 기여할 수 있다. 또한 컴퓨터 비전을 활용한 인간공학적 자세 평가를 위한 작업 자세 입력 시스템 개발에 대한 향후 연구 방향도 이번 연구에서 제시된다.Chapter 1 Introduction 1 1.1 Background 1 1.2 Research Objectives 4 1.3 Organization of the Thesis 5 Chapter 2 Literature Review 6 2.1 Overview 6 2.2 Work-related Musculoskeletal Disorders 6 2.3 Ergonomic Posture Analysis 7 2.3.1 Self-reports 7 2.3.2 Observational Methods 7 2.3.3 Direct Methods 15 2.3.4 Vision-based Methods 17 2.4 3D Human Pose Estimation 19 2.4.1 Model-free Approaches 20 2.4.2 Model-based Approaches 21 Chapter 3 Proposed System Design 23 3.1 Overview 23 3.2 Human Tracking 24 3.3 3D Human Pose Estimation 24 3.4 Score Calculation 26 3.4.1 Posture Score Calculation 26 3.4.2 Output of the Proposed System 31 Chapter 4 Validation Experiment 32 4.1 Hypotheses 32 4.2 Methods 32 4.2.1 Participants 32 4.2.2 Apparatus 33 4.2.3 Procedure 33 4.2.4 Data Analysis 37 4.3 Results 38 4.3.1 RULA 38 4.3.2 REBA 46 4.3.3 Evaluation Items for Manual Input 54 Chapter 5 Discussion 56 5.1 Group Difference 56 5.1.1 RULA 57 5.1.2 REBA 57 5.2 Evaluation Items for Manual Input 58 5.3 Proposed Work Pose Entry System 59 Chapter 6 Conclusion 62 6.1 Conclusion 62 6.2 Limitation, Contribution, and Future Direction 62 Bibliography 65 국문초록 77석

Pilot Validation Study of Inertial Measurement Units and Markerless Methods for 3D Neck and Trunk Kinematics during a Simulated Surgery Task

Surgeons are at high risk for developing musculoskeletal symptoms (MSS), like neck and back pain. Quantitative analysis of 3D neck and trunk movements during surgery can help to develop preventive devices such as exoskeletons. Inertial Measurement Units (IMU) and markerless motion capture methods are allowed in the operating room (OR) and are a good alternative for bulky optoelectronic systems. We aim to validate IMU and markerless methods against an optoelectronic system during a simulated surgery task. Intraclass correlation coefficient (ICC (2,1)), root mean square error (RMSE), range of motion (ROM) difference and Bland–Altman plots were used for evaluating both methods. The IMU-based motion analysis showed good-to-excellent (ICC 0.80–0.97) agreement with the gold standard within 2.3 to 3.9 degrees RMSE accuracy during simulated surgery tasks. The markerless method shows 5.5 to 8.7 degrees RMSE accuracy (ICC 0.31–0.70). Therefore, the IMU method is recommended over the markerless motion capture

Assessing the Utility of a Video-Based Motion Capture Alternative in the Assessment of Lumbar Spine Planar Angular Joint Kinematics

Markerless motion capture is a novel technique to measure human movement kinematics. The purpose of this research is to evaluate the markerless algorithm, DeepLabCut (DLC) against a 3D motion capture system (Vicon Motion Systems Ltd., Oxford, UK) in the analysis of planar spine and elbow flexion-extension movement. Data were acquired concurrently from DLC and Vicon for all movements. A novel DLC model was trained using data derived from a subset of participants (training group). Accuracy and precision were assessed from data derived from the training group as well as in a new set of participants (testing group). Two-way SPM ANOVAs were used to detect significant differences between the training vs. testing sets, capture methods (Vicon vs. DLC), as well as potential higher order interaction effect between these independent variables in the estimation of flexion extension angles and variability. No significant differences were observed in any planar angles, nor were any higher order interactions observed between each motion capture modality and the training vs. testing datasets. Bland Altman plots were also generated to depict the mean bias and level of agreement between DLC and Vicon for both training, and testing datasets. Supplemental analyses, suggest that these results are partially affected by the alignment of each participant’s body segments with respect to each planar reference frame. This research suggests that DLC-derived planar kinematics of both the elbow and lumbar spine are of acceptable accuracy and precision when compared to conventional laboratory gold-standards (Vicon)

IMU-based Modularized Wearable Device for Human Motion Classification

Human motion analysis is used in many different fields and applications. Currently, existing systems either focus on one single limb or one single class of movements. Many proposed systems are designed to be used in an indoor controlled environment and must possess good technical know-how to operate. To improve mobility, a less restrictive, modularized, and simple Inertial Measurement units based system is proposed that can be worn separately and combined. This allows the user to measure singular limb movements separately and also monitor whole body movements over a prolonged period at any given time while not restricted to a controlled environment. For proper analysis, data is conditioned and pre-processed through possible five stages namely power-based, clustering index-based, Kalman filtering, distance-measure-based, and PCA-based dimension reduction. Different combinations of the above stages are analyzed using machine learning algorithms for selected case studies namely hand gesture recognition and environment and shoe parameter-based walking pattern analysis to validate the performance capability of the proposed wearable device and multi-stage algorithms. The results of the case studies show that distance-measure-based and PCA-based dimension reduction will significantly improve human motion identification accuracy. This is further improved with the introduction of the Kalman filter. An LSTM neural network is proposed as an alternate classifier and the results indicate that it is a robust classifier for human motion recognition. As the results indicate, the proposed wearable device architecture and multi-stage algorithms are cable of distinguishing between subtle human limb movements making it a viable tool for human motion analysis.Comment: 10 pages, 12 figures, 28 reference

Explaining the Ergonomic Assessment of Human Movement in Industrial Contexts

Manufacturing processes are based on human labour and the symbiosis between human operators and machines. The operators are required to follow predefined sequences of movements. The operations carried out at assembly lines are repetitive, being identified as a risk factor for the onset of musculoskeletal disorders. Ergonomics plays a big role in preventing occupational diseases. Ergonomic risk scores measure the overall risk exposure of operators however these methods still present challenges: the scores are often associated to a given workstation, being agnostic to the variability among operators. Observation methods are most often employed yet require a significant amount of effort, preventing an accurate and continuous ergonomic evaluation to the entire population of operators. Finally, the risk’s results are rendered as index scores, hindering a more comprehensive interpretation by occupational physicians. This dissertation developed a solution for automatic operator risk exposure in assembly lines. Three main contributions were presented: (1) an upper limb and torso motion tracking algorithm which relies on inertial sensors to estimate the orientation of anatomical joints; (2) an adjusted ergonomic risk score; (3) an ergonomic risk explanation approach based on the analysis of the angular risk factors. Throughout the research, two experimental assessments were conducted: laboratory validation and field evaluation. The laboratory tests enabled the creation of a movements’ dataset and used an optical motion capture system as reference. The field evaluation dataset was acquired on an automotive assembly line and serve as the basis for an ergonomic risk evaluation study. The experimental results revealed that the proposed solution has the potential to be applied in a real environment. Through direct measures, the ergonomic feedback is fastened, and consequently, the evaluation can be extended to more operators, ultimately preventing, in long-term, work-related injuries

Classification of trunk motion based on inertial sensors

55 páginasPor mucho tiempo en la ingeniería se ha trabajado para la valoración y prevención de desórdenes musculo esqueléticos de la espalda. El uso de sistemas de captura de movimiento es una de las técnicas más utilizadas en este ámbito, considerada el estándar, es un sistema de alto costo y limitaciones considerables. El uso de sensores de inercia en los últimos años ha llegado a ser muchos más frecuente gracias a los avances tecnológicos que lo han llevado a ser más accesible y compacto. Este proyecto se basa en el uso de sensores de inercia para la clasificación de movimientos de la espalda, lo cual ayuda a la valoración y prevención de desórdenes musculo esqueléticos. En el presente texto, se presenta todo el proceso de desarrollo de un sistema para la identificación de movimientos de la espalda, mediante el uso de las señales de los sensores de inercia. Se presentaran experimentos y resultados del uso de un sistema de captura de movimiento comparado con el sistema de sensores de inercia. El programa propuesto para la clasificación de los tres movimientos del tronco (flexión, lateral y rotación) trae consigo importantes ventajas de fiabilidad y libertad espacial. El sistema desarrollado permite la valoración de los movimientos del tronco. La clasificación de estos movimientos utilizando sensores de inercia es un método considerablemente mucho más portable comparado con un sistema de captura de movimiento. También, este sistema se puede usar en diferentes espacios sin mayores esfuerzos y los límites transnacionales de movimientos son mucho más amplios en comparación con sistemas de captura de movimientos. La clasificación permite la adición de nuevas características para la identificación más precisa de movimientos o posturas que permitirán una mejor valoración para la prevención de desórdenes musculo esqueléticos del trono.PregradoIngeniero(a) Biomédico(a

An inertial motion capture framework for constructing body sensor networks

Motion capture is the process of measuring and subsequently reconstructing the movement of an animated object or being in virtual space. Virtual reconstructions of human motion play an important role in numerous application areas such as animation, medical science, ergonomics, etc. While optical motion capture systems are the industry standard, inertial body sensor networks are becoming viable alternatives due to portability, practicality and cost. This thesis presents an innovative inertial motion capture framework for constructing body sensor networks through software environments, smartphones and web technologies. The first component of the framework is a unique inertial motion capture software environment aimed at providing an improved experimentation environment, accompanied by programming scaffolding and a driver development kit, for users interested in studying or engineering body sensor networks. The software environment provides a bespoke 3D engine for kinematic motion visualisations and a set of tools for hardware integration. The software environment is used to develop the hardware behind a prototype motion capture suit focused on low-power consumption and hardware-centricity. Additional inertial measurement units, which are available commercially, are also integrated to demonstrate the functionality the software environment while providing the framework with additional sources for motion data. The smartphone is the most ubiquitous computing technology and its worldwide uptake has prompted many advances in wearable inertial sensing technologies. Smartphones contain gyroscopes, accelerometers and magnetometers, a combination of sensors that is commonly found in inertial measurement units. This thesis presents a mobile application that investigates whether the smartphone is capable of inertial motion capture by constructing a novel omnidirectional body sensor network. This thesis proposes a novel use for web technologies through the development of the Motion Cloud, a repository and gateway for inertial data. Web technologies have the potential to replace motion capture file formats with online repositories and to set a new standard for how motion data is stored. From a single inertial measurement unit to a more complex body sensor network, the proposed architecture is extendable and facilitates the integration of any inertial hardware configuration. The Motion Cloud’s data can be accessed through an application-programming interface or through a web portal that provides users with the functionality for visualising and exporting the motion data