Trunk Rehabilitation Using Cable-Driven Robotic Systems

Upper body control is required to complete many daily tasks. One needs to stabilize the head and trunk over the pelvis, as one shifts the center of mass to interact with the world. While healthy individuals can perform activities that require leaning, reaching, and grasping readily, those with neurological and musculoskeletal disorders present with control deficits. These deficits can lead to difficulty in shifting the body center of mass away from the stable midline, leading to functional limitations and a decline in the quality of activity. Often these patient groups use canes, walkers, and wheelchairs for support, leading to occasional strapping or joint locking of the body for trunk stabilization. Current rehabilitation strategies focus on isolated components of stability. This includes strengthening, isometric exercises, hand-eye coordination tasks, isolated movement, and proprioceptive training. Although all these components are evidence based and directly correlate to better stability, motor learning theories such as those by Nikolai Bernstein, suggest that task and context specific training can lead to better outcomes. In specific, based on our experimentation, we believe functional postural exploration, while encompassing aspects of strengthening, hand-eye coordination, and proprioceptive feedback can provide better results. In this work, we present two novel cable robotic platforms for seated and standing posture training. The Trunk Support Trainer (TruST) is a platform for seated posture rehabilitation that provides controlled external wrench on the human trunk in any direction in real-time. The Stand Trainer is a platform for standing posture rehabilitation that can control the trunk, pelvis, and knees, simultaneously. The system works through the use of novel force-field algorithms that are modular and user-specific. The control uses an assist-as-needed strategy to apply forces on the user during regions of postural instability. The device also allows perturbations for postural reactive training. We have conducted several studies using healthy adult populations and pilot studies on patient groups including cerebral palsy, cerebellar ataxia, and spinal cord injury. We propose new training methods that incorporate motor learning theory and objective interventions for improving posture control. We identify novel methods to characterize posture in form of the “8-point star test”. This is to assess the postural workspace. We also demonstrate novel methods for functional training of posture and balance. Our results show that training with our robotic platforms can change the trunk kinematics. Specifically, healthy adults are able to translate the trunk further and rotate the trunk more anteriorly in the seated position. In the standing position, they can alter their reach strategy to maintain the upper trunk more vertically while reaching. Similarly, Cerebral Palsy patients improve their trunk translations, reaching workspace, and maintain a more vertical posture after training, in the seated position. Our results also showed that an Ataxia patient was able to improve their reaching workspace and trunk translations in the standing position. Finally, our results show that the robotic platforms can successfully reduce trunk and pelvis sway in spinal cord injury patients. The results of the pilot studies suggest that training with our robotic platforms and methods is beneficial in improving trunk control

Human Health Engineering Volume II

In this Special Issue on “Human Health Engineering Volume II”, we invited submissions exploring recent contributions to the field of human health engineering, i.e., technology for monitoring the physical or mental health status of individuals in a variety of applications. Contributions could focus on sensors, wearable hardware, algorithms, or integrated monitoring systems. We organized the different papers according to their contributions to the main parts of the monitoring and control engineering scheme applied to human health applications, namely papers focusing on measuring/sensing physiological variables, papers highlighting health-monitoring applications, and examples of control and process management applications for human health. In comparison to biomedical engineering, we envision that the field of human health engineering will also cover applications for healthy humans (e.g., sports, sleep, and stress), and thus not only contribute to the development of technology for curing patients or supporting chronically ill people, but also to more general disease prevention and optimization of human well-being

Humanoid Robots

For many years, the human being has been trying, in all ways, to recreate the complex mechanisms that form the human body. Such task is extremely complicated and the results are not totally satisfactory. However, with increasing technological advances based on theoretical and experimental researches, man gets, in a way, to copy or to imitate some systems of the human body. These researches not only intended to create humanoid robots, great part of them constituting autonomous systems, but also, in some way, to offer a higher knowledge of the systems that form the human body, objectifying possible applications in the technology of rehabilitation of human beings, gathering in a whole studies related not only to Robotics, but also to Biomechanics, Biomimmetics, Cybernetics, among other areas. This book presents a series of researches inspired by this ideal, carried through by various researchers worldwide, looking for to analyze and to discuss diverse subjects related to humanoid robots. The presented contributions explore aspects about robotic hands, learning, language, vision and locomotion

Understanding motor control in humans to improve rehabilitation robots

Recent reviews highlighted the limited results of robotic rehabilitation and the low quality of evidences in this field. Despite the worldwide presence of several robotic infrastructures, there is still a lack of knowledge about the capabilities of robotic training effect on the neural control of movement. To fill this gap, a step back to motor neuroscience is needed: the understanding how the brain works in the generation of movements, how it adapts to changes and how it acquires new motor skills is fundamental. This is the rationale behind my PhD project and the contents of this thesis: all the studies included in fact examined changes in motor control due to different destabilizing conditions, ranging from external perturbations, to self-generated disturbances, to pathological conditions. Data on healthy and impaired adults have been collected and quantitative and objective information about kinematics, dynamics, performance and learning were obtained for the investigation of motor control and skill learning. Results on subjects with cervical dystonia show how important assessment is: possibly adequate treatments are missing because the physiological and pathological mechanisms underlying sensorimotor control are not routinely addressed in clinical practice. These results showed how sensory function is crucial for motor control. The relevance of proprioception in motor control and learning is evident also in a second study. This study, performed on healthy subjects, showed that stiffness control is associated with worse robustness to external perturbations and worse learning, which can be attributed to the lower sensitiveness while moving or co-activating. On the other hand, we found that the combination of higher reliance on proprioception with \u201cdisturbance training\u201d is able to lead to a better learning and better robustness. This is in line with recent findings showing that variability may facilitate learning and thus can be exploited for sensorimotor recovery. Based on these results, in a third study, we asked participants to use the more robust and efficient strategy in order to investigate the control policies used to reject disturbances. We found that control is non-linear and we associated this non-linearity with intermittent control. As the name says, intermittent control is characterized by open loop intervals, in which movements are not actively controlled. We exploited the intermittent control paradigm for other two modeling studies. In these studies we have shown how robust is this model, evaluating it in two complex situations, the coordination of two joints for postural balance and the coordination of two different balancing tasks. It is an intriguing issue, to be addressed in future studies, to consider how learning affects intermittency and how this can be exploited to enhance learning or recovery. The approach, that can exploit the results of this thesis, is the computational neurorehabilitation, which mathematically models the mechanisms underlying the rehabilitation process, with the aim of optimizing the individual treatment of patients. Integrating models of sensorimotor control during robotic neurorehabilitation, might lead to robots that are fully adaptable to the level of impairment of the patient and able to change their behavior accordingly to the patient\u2019s intention. This is one of the goals for the development of rehabilitation robotics and in particular of Wristbot, our robot for wrist rehabilitation: combining proper assessment and training protocols, based on motor control paradigms, will maximize robotic rehabilitation effects

Human-Inspired Balancing and Recovery Stepping for Humanoid Robots

Robustly maintaining balance on two legs is an important challenge for humanoid robots. The work presented in this book represents a contribution to this area. It investigates efficient methods for the decision-making from internal sensors about whether and where to step, several improvements to efficient whole-body postural balancing methods, and proposes and evaluates a novel method for efficient recovery step generation, leveraging human examples and simulation-based reinforcement learning

동영상 속 사람 동작의 물리 기반 재구성 및 분석

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 컴퓨터공학부, 2021. 2. 이제희.In computer graphics, simulating and analyzing human movement have been interesting research topics started since the 1960s. Still, simulating realistic human movements in a 3D virtual world is a challenging task in computer graphics. In general, motion capture techniques have been used. Although the motion capture data guarantees realistic result and high-quality data, there is lots of equipment required to capture motion, and the process is complicated. Recently, 3D human pose estimation techniques from the 2D video are remarkably developed. Researchers in computer graphics and computer vision have attempted to reconstruct the various human motions from video data. However, existing methods can not robustly estimate dynamic actions and not work on videos filmed with a moving camera. In this thesis, we propose methods to reconstruct dynamic human motions from in-the-wild videos and to control the motions. First, we developed a framework to reconstruct motion from videos using prior physics knowledge. For dynamic motions such as backspin, the poses estimated by a state-of-the-art method are incomplete and include unreliable root trajectory or lack intermediate poses. We designed a reward function using poses and hints extracted from videos in the deep reinforcement learning controller and learned a policy to simultaneously reconstruct motion and control a virtual character. Second, we simulated figure skating movements in video. Skating sequences consist of fast and dynamic movements on ice, hindering the acquisition of motion data. Thus, we extracted 3D key poses from a video to then successfully replicate several figure skating movements using trajectory optimization and a deep reinforcement learning controller. Third, we devised an algorithm for gait analysis through video of patients with movement disorders. After acquiring the patients joint positions from 2D video processed by a deep learning network, the 3D absolute coordinates were estimated, and gait parameters such as gait velocity, cadence, and step length were calculated. Additionally, we analyzed the optimization criteria of human walking by using a 3D musculoskeletal humanoid model and physics-based simulation. For two criteria, namely, the minimization of muscle activation and joint torque, we compared simulation data with real human data for analysis. To demonstrate the effectiveness of the first two research topics, we verified the reconstruction of dynamic human motions from 2D videos using physics-based simulations. For the last two research topics, we evaluated our results with real human data.컴퓨터 그래픽스에서 인간의 움직임 시뮬레이션 및 분석은 1960 년대부터 다루어진 흥미로운 연구 주제이다. 몇 십년 동안 활발하게 연구되어 왔음에도 불구하고, 3차원 가상 공간 상에서 사실적인 인간의 움직임을 시뮬레이션하는 연구는 여전히 어렵고 도전적인 주제이다. 그동안 사람의 움직임 데이터를 얻기 위해서 모션 캡쳐 기술이 사용되어 왔다. 모션 캡처 데이터는 사실적인 결과와 고품질 데이터를 보장하지만 모션 캡쳐를 하기 위해서 필요한 장비들이 많고, 그 과정이 복잡하다. 최근에 2차원 영상으로부터 사람의 3차원 자세를 추정하는 연구들이 괄목할 만한 결과를 보여주고 있다. 이를 바탕으로 컴퓨터 그래픽스와 컴퓨터 비젼 분야의 연구자들은 비디오 데이터로부터 다양한 인간 동작을 재구성하려는 시도를 하고 있다. 그러나 기존의 방법들은 빠르고 다이나믹한 동작들은 안정적으로 추정하지 못하며 움직이는 카메라로 촬영한 비디오에 대해서는 작동하지 않는다. 본 논문에서는 비디오로부터 역동적인 인간 동작을 재구성하고 동작을 제어하는 방법을 제안한다. 먼저 사전 물리학 지식을 사용하여 비디오에서 모션을 재구성하는 프레임 워크를 제안한다. 공중제비와 같은 역동적인 동작들에 대해서 최신 연구 방법을 동원하여 추정된 자세들은 캐릭터의 궤적을 신뢰할 수 없거나 중간에 자세 추정에 실패하는 등 불완전하다. 우리는 심층강화학습 제어기에서 영상으로부터 추출한 포즈와 힌트를 활용하여 보상 함수를 설계하고 모션 재구성과 캐릭터 제어를 동시에 하는 정책을 학습하였다. 둘 째, 비디오에서 피겨 스케이팅 기술을 시뮬레이션한다. 피겨 스케이팅 기술들은 빙상에서 빠르고 역동적인 움직임으로 구성되어 있어 모션 데이터를 얻기가 까다롭다. 비디오에서 3차원 키 포즈를 추출하고 궤적 최적화 및 심층강화학습 제어기를 사용하여 여러 피겨 스케이팅 기술을 성공적으로 시연한다. 셋 째, 파킨슨 병이나 뇌성마비와 같은 질병으로 인하여 움직임 장애가 있는 환자의 보행을 분석하기 위한 알고리즘을 제안한다. 2차원 비디오로부터 딥러닝을 사용한 자세 추정기법을 사용하여 환자의 관절 위치를 얻어낸 다음, 3차원 절대 좌표를 얻어내어 이로부터 보폭, 보행 속도와 같은 보행 파라미터를 계산한다. 마지막으로, 근골격 인체 모델과 물리 시뮬레이션을 이용하여 인간 보행의 최적화 기준에 대해 탐구한다. 근육 활성도 최소화와 관절 돌림힘 최소화, 두 가지 기준에 대해 시뮬레이션한 후, 실제 사람 데이터와 비교하여 결과를 분석한다. 처음 두 개의 연구 주제의 효과를 입증하기 위해, 물리 시뮬레이션을 사용하여 이차원 비디오로부터 재구성한 여러 가지 역동적인 사람의 동작들을 재현한다. 나중 두 개의 연구 주제는 사람 데이터와의 비교 분석을 통하여 평가한다.1 Introduction 1 2 Background 9 2.1 Pose Estimation from 2D Video . . . . . . . . . . . . . . . . . . . . 9 2.2 Motion Reconstruction from Monocular Video . . . . . . . . . . . . 10 2.3 Physics-Based Character Simulation and Control . . . . . . . . . . . 12 2.4 Motion Reconstruction Leveraging Physics . . . . . . . . . . . . . . 13 2.5 Human Motion Control . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.1 Figure Skating Simulation . . . . . . . . . . . . . . . . . . . 16 2.6 Objective Gait Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.7 Optimization for Human Movement Simulation . . . . . . . . . . . . 17 2.7.1 Stability Criteria . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Human Dynamics from Monocular Video with Dynamic Camera Movements 19 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.3 Pose and Contact Estimation . . . . . . . . . . . . . . . . . . . . . . 21 3.4 Learning Human Dynamics . . . . . . . . . . . . . . . . . . . . . . . 24 3.4.1 Policy Learning . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.4.2 Network Training . . . . . . . . . . . . . . . . . . . . . . . . 28 3.4.3 Scene Estimator . . . . . . . . . . . . . . . . . . . . . . . . 29 3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.5.1 Video Clips . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.5.2 Comparison of Contact Estimators . . . . . . . . . . . . . . . 33 3.5.3 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.5.4 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4 Figure Skating Simulation from Video 42 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3 Skating Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3.1 Non-holonomic Constraints . . . . . . . . . . . . . . . . . . 46 4.3.2 Relaxation of Non-holonomic Constraints . . . . . . . . . . . 47 4.4 Data Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.5 Trajectory Optimization and Control . . . . . . . . . . . . . . . . . . 50 4.5.1 Trajectory Optimization . . . . . . . . . . . . . . . . . . . . 50 4.5.2 Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5 Gait Analysis Using Pose Estimation Algorithm with 2D-video of Patients 61 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 5.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.2.1 Patients and video recording . . . . . . . . . . . . . . . . . . 63 5.2.2 Standard protocol approvals, registrations, and patient consents 66 5.2.3 3D Pose estimation from 2D video . . . . . . . . . . . . . . . 66 5.2.4 Gait parameter estimation . . . . . . . . . . . . . . . . . . . 67 5.2.5 Statistical analysis . . . . . . . . . . . . . . . . . . . . . . . 68 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.3.1 Validation of video-based analysis of the gait . . . . . . . . . 68 5.3.2 gait analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.4.1 Validation with the conventional sensor-based method . . . . 75 5.4.2 Analysis of gait and turning in TUG . . . . . . . . . . . . . . 75 5.4.3 Correlation with clinical parameters . . . . . . . . . . . . . . 76 5.4.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.5 Supplementary Material . . . . . . . . . . . . . . . . . . . . . . . . . 77 6 Control Optimization of Human Walking 80 6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.2.1 Musculoskeletal model . . . . . . . . . . . . . . . . . . . . . 82 6.2.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.2.3 Control co-activation level . . . . . . . . . . . . . . . . . . . 83 6.2.4 Push-recovery experiment . . . . . . . . . . . . . . . . . . . 84 6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 6.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 7 Conclusion 90 7.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Docto

NOVEL BIO-AWARE TECHNOLOGIES AND ALGORITHMS FOR HANDS AND HAPTICS

This thesis focuses on novel algorithms and interfaces, arising from inspection and comprehension of the human nature. In the first part I deal with new mechanical designs and concepts for building and controlling robotic hands. In particular I show how the sensorimotor synergies of the human hand can be useful not only for controlling but also for building robotic hands, suggesting novel design paradigms. Despite the synergy model is useful for designing and controlling robotic hands, it is incomplete to explain the hand behavior during grasp (both for humans and robots). To solve this problem, it is needed to consider compliant articulations introducing the "soft synergies" concept. Consequently, the compliance and the soft synergies lead to the concepts of muscle and mechanical impedance. Thus, in the second part of this thesis I present an observer for estimating the time varying mechanical impedance of a Variable Stiffness Actuator (VSA), i.e. a novel kind of actuator whose performances and capabilities are very close to the human muscles. Another important feature, both for human and robotic hands, is the sense of touch. Indeed in the third and last part of this thesis, I deal with the haptics and haptic interfaces. I show two new haptic devices with their applications on humans. Moreover, I present a tactile sensing algorithm toolbox for computing the contact point of a robotic fingertip interacting with an object

Object Handovers: a Review for Robotics

This article surveys the literature on human-robot object handovers. A handover is a collaborative joint action where an agent, the giver, gives an object to another agent, the receiver. The physical exchange starts when the receiver first contacts the object held by the giver and ends when the giver fully releases the object to the receiver. However, important cognitive and physical processes begin before the physical exchange, including initiating implicit agreement with respect to the location and timing of the exchange. From this perspective, we structure our review into the two main phases delimited by the aforementioned events: 1) a pre-handover phase, and 2) the physical exchange. We focus our analysis on the two actors (giver and receiver) and report the state of the art of robotic givers (robot-to-human handovers) and the robotic receivers (human-to-robot handovers). We report a comprehensive list of qualitative and quantitative metrics commonly used to assess the interaction. While focusing our review on the cognitive level (e.g., prediction, perception, motion planning, learning) and the physical level (e.g., motion, grasping, grip release) of the handover, we briefly discuss also the concepts of safety, social context, and ergonomics. We compare the behaviours displayed during human-to-human handovers to the state of the art of robotic assistants, and identify the major areas of improvement for robotic assistants to reach performance comparable to human interactions. Finally, we propose a minimal set of metrics that should be used in order to enable a fair comparison among the approaches.Comment: Review paper, 19 page

Study and development of sensorimotor interfaces for robotic human augmentation

This thesis presents my research contribution to robotics and haptics in the context of human augmentation. In particular, in this document, we are interested in bodily or sensorimotor augmentation, thus the augmentation of humans by supernumerary robotic limbs (SRL). The field of sensorimotor augmentation is new in robotics and thanks to the combination with neuroscience, great leaps forward have already been made in the past 10 years. All of the research work I produced during my Ph.D. focused on the development and study of fundamental technology for human augmentation by robotics: the sensorimotor interface. This new concept is born to indicate a wearable device which has two main purposes, the first is to extract the input generated by the movement of the user's body, and the second to provide the somatosensory system of the user with an haptic feedback. This thesis starts with an exploratory study of integration between robotic and haptic devices, intending to combine state-of-the-art devices. This allowed us to realize that we still need to understand how to improve the interface that will allow us to feel the agency when using an augmentative robot. At this point, the path of this thesis forks into two alternative ways that have been adopted to improve the interaction between the human and the robot. In this regard, the first path we presented tackles two aspects conerning the haptic feedback of sensorimotor interfaces, which are the choice of the positioning and the effectiveness of the discrete haptic feedback. In the second way we attempted to lighten a supernumerary finger, focusing on the agility of use and the lightness of the device. One of the main findings of this thesis is that haptic feedback is considered to be helpful by stroke patients, but this does not mitigate the fact that the cumbersomeness of the devices is a deterrent to their use. Preliminary results here presented show that both the path we chose to improve sensorimotor augmentation worked: the presence of the haptic feedback improves the performance of sensorimotor interfaces, the co-positioning of haptic feedback and the input taken from the human body can improve the effectiveness of these interfaces, and creating a lightweight version of a SRL is a viable solution for recovering the grasping function