Prediction and control in human neuromusculoskeletal models

Computational neuromusculoskeletal modelling enables the generation and testing of hypotheses about human movement on a large scale, in silico. Humanoid models, which increasingly aim to replicate the full complexity of the human nervous and musculoskeletal systems, are built on extensive prior knowledge, extracted from anatomical imaging, kinematic and kinetic measurement, and codified as model description. Where inverse dynamic analysis is applied, its basis is in Newton's laws of motion, and in solving for muscular redundancy it is necessary to invoke knowledge of central nervous motor strategy. This epistemological approach contrasts strongly with the models of machine learning, which are generally over-parameterised and largely data-driven. Even as spectacular performance has been delivered by the application of these models in a number of discrete domains of artificial intelligence, work towards general human-level intelligence has faltered, leading many to wonder if the data-driven approach is fundamentally limited, and spurring efforts to combine machine learning with knowledge-based modelling. Through a series of five studies, this thesis explores the combination of neuromusculoskeletal modelling with machine learning in order to enhance the core tasks of prediction and control. Several principles for the development of clinically useful artificially intelligent systems emerge: stability, computational efficiency and incorporation of prior knowledge. The first study concerns the use of neural network function approximators for the prediction of internal forces during human movement, an important task with many clinical applications, but one for which the standard tools of modelling are slow and cumbersome. By training on a large dataset of motions and their corresponding forces, state of the art performance is demonstrated, with many-fold increases in inference speed enabling the deployment of trained models for use in a real time biofeedback system. Neural networks trained in this way, to imitate some optimal controller, encode a mapping from high-level movement descriptors to actuator commands, and may thus be deployed in simulation as \textit{policies} to control the actions of humanoid models. Unfortunately, the high complexity of realistic simulation makes stable control a challenging task, beyond the capabilities of such naively trained models. The objective of the second study was to improve performance and stability of policy-based controllers for humanoid models in simulation. A novel technique was developed, borrowing from established unsupervised adversarial methods in computer vision. This technique enabled significant gains in performance relative to a neural network baseline, without the need for additional access to the optimal controller. For the third study, increases in the capabilities of these policy-based controllers were sought. Reinforcement learning is widely considered the most powerful means of optimising such policies, but it is computationally inefficient, and this inefficiency limits its clinical utility. To mitigate this problem, a novel framework, making use of domain-specific knowledge present in motion data, and in an inverse model of the biomechanical system, was developed. Training on simple desktop hardware, this framework enabled rapid initialisation of humanoid models that were able to move naturally through a 3-dimensional simulated environment, with 900-fold improvements in sample efficiency relative to a related technique based on pure reinforcement learning. After training with subject-specific anatomical parameters, and motion data, learned policies represent personalised models of motor control that may be further interrogated to test hypotheses about movement. For the fourth study, subject-specific controllers were taken and used as the substrate for transfer learning, by removing kinematic constraints and optimising with respect to the magnitude of the medial knee joint reaction force, an important biomechanical variable in osteoarthritis of the knee. Models learned new kinematic strategies for the reduction of this biomarker, which were subsequently validated by their use, in the real world, to construct subject-specific routines for real time gait retraining. Six out of eight subjects were able to reduce medial knee joint loading by pursuing the personalised kinematic targets found in simulation. Personalisation of assistive devices, such as limb prostheses, is another area of growing interest, and one for which computational frameworks promise cost-effective solutions. Reinforcement learning provides powerful techniques for this task but the expansion of the scope of optimisation, to include previously static elements of a prosthesis, is problematic for its complexity and resulting sample inefficiency. The fifth and final study demonstrates a new algorithm that leverages the methods described in the previous studies, and additional techniques for variance control, to surmount this problem, improving sample efficiency and simultaneously, through the use of prior knowledge encoded in motion data, providing a rational means of determining optimality in the prosthesis. Trained models were able to jointly optimise motor control and prosthesis design to enable improved performance in a walking task, and optimised designs were robust to both random seed and reward specification. This algorithm could be used to speed the design and production of real personalised prostheses, representing a potent realisation of the potential benefits of combined reinforcement learning and realistic neuromusculoskeletal modelling.Open Acces

Closed-loop Central Pattern Generator Control of Human Gaits in OpenSim Simulator

International audienceIn this paper, a new neuro-musculoskeletal gait simulation platform is presented. This platform is developed to reproduce healthy or altered walking gaits. It is based on an original model of central pattern generator able to generate variable rhythmic signals for controlling biological human leg joints. Output signals of motoneurons are applied to excitation inputs of modelled muscles of the human lower limbs model. Eight central pattern generators control a musculoskeletal model made up of three joints per leg actuated by 44 Hill-type muscle models. Forward dynamics simulation in OpenSim show that it is possible to generate different stable walking gaits by changing parameters of controller. Further work is aimed on development of stable human standing by implementing reflexes

Otimização muscle-in-the-loop em tempo real para reabilitação física com um exosqueleto ativo: uma mudança de paradigma

Assisting human locomotion with a wearable robotic orthosis is still quite challenging, largely due to the complexity of the neuromusculoskeletal system, the time-varying dynamics that accompany motor adaptation, and the uniqueness of every individual’s response to the assistance given by the robot. To this day, these devices have not met their well-known promise yet, mostly due to the fact that they are not perfectly suitable for the rehabilitation of neuropathologic patients. One of the main challenges hampering this goal still relies on the interface and co-dependency between the human and the machine. Nowadays, most commercial exoskeletons replay pre-defined gait patterns, whereas research exoskeletons are switching to controllers based on optimized torque profiles. In most cases, the dynamics of the human musculoskeletal system are still ignored and do not take into account the optimal conditions for inducing a positive modulation of neuromuscular activity. This is because both rehabilitation strategies are still emphasized on the macro level of the whole joint instead of focusing on the muscles’ dynamics and activity, which are the actual anatomical elements that may need to be rehabilitated. Strategies to keep the human in the loop of the exoskeleton’s control laws in real-time may help to overcome these challenges. The main purpose of the present dissertation is to make a paradigm shift in the approach on how the assistance that is given to a subject by an exoskeleton is modelled and controlled during physical rehabilitation. Therefore, in the scope of the present work, it was intended to design, concede, implement, and validate a real-time muscle-in-the-loop optimization model to find the best assistive support ratio that would induce optimal rehabilitation conditions to a specific group of impaired muscles while having a minimum impact on the other healthy muscles. The developed optimization model was implemented in the form of a plugin and was integrated on a neuromechanical model-based interface for driving a bilateral ankle exoskeleton. Experimental pilot tests evaluated the feasibility and effectiveness of the model. Results of the most significant pilots achieved EMG reductions up to 61 ± 3 % in Soleus and 41 ± 10 % in Gastrocnemius Lateralis. Moreover, results also demonstrated the efficiency of the optimization’s specific reduction on rehabilitation by looking into the muscular fatigue after each experiment. Finally, two parallel preliminary studies emerged from the pilots, which looked at muscle adaptation, after a new assistive condition had been applied, over time and at the effect of the lateral positioning of the exoskeleton’s actuators on the leg muscles.Auxiliar a locomoção humana com uma ortose robótica ainda é bastante desafiante, em grande parte devido à complexidade do sistema neuromusculoesquelético, à dinâmica variável no tempo que acompanha a adaptação motora e à singularidade da resposta de cada indivíduo à assistência dada pelo robô. Até hoje, está por cumprir a promessa inicial destes dispositivos, principalmente devido ao facto de não serem perfeitamente adequados para a reabilitação de pacientes neuropatológicos. Um dos principais desafios que dificultam esse objetivo foca-se ainda na interface e na co-dependência entre o ser humano e a máquina. Hoje em dia, a maioria dos exoesqueletos comerciais reproduz padrões de marcha predefinidos, enquanto que os exoesqueletos em investigação estão só agora a mudar para controladores com base em perfis de binário otimizados. Na maioria dos casos, a dinâmica do sistema musculoesquelético humano ainda é ignorada e não tem em consideração as condições ideais para induzir uma modulação positiva da atividade neuromuscular. Isso ocorre porque ambas as estratégias de reabilitação ainda são enfatizadas no nível macro de toda a articulação, em vez de se concentrar na dinâmica e atividade dos músculos, que são os elementos anatómicos que realmente precisam de ser reabilitados. Estratégias para manter o ser humano em loop nos comandos que controlam o exoesqueleto em tempo real podem ajudar a superar estes desafios. O principal objetivo desta dissertação é fazer uma mudança de paradigma na abordagem em como a assistência que é dada a um sujeito por um exosqueleto é modelada e controlada durante a reabilitação física. Portanto, no contexto do presente trabalho, pretendeu-se projetar, conceder, implementar e validar um modelo de otimização muscle-in-the-loop em tempo real para encontrar a melhor relação de suporte capaz de induzir as condições ideais de reabilitação para um grupo específico de músculos fragilizados, tendo um impacto mínimo nos outros músculos saudáveis. O modelo de otimização desenvolvido foi implementado na forma de um plugin e foi integrado numa interface baseada num modelo neuromecânico para o controlo de um exoesqueleto bilateral de tornozelo. Testes experimentais piloto avaliaram a viabilidade e a eficácia do modelo. Os resultados dos testes mais significativos demonstraram reduções de EMG de até 61 ± 3 % no Soleus e 41 ± 10 % no Gastrocnemius Lateral. Adicionalmente, os resultados demonstraram também a eficiência em reabilitação da redução específica no EMG devido à otimização tendo em conta a fadiga muscular após cada teste. Finalmente, dois estudos preliminares paralelos emergiram dos testes piloto, que analisaram a adaptação muscular após uma nova condição assistiva ter sido definida ao longo do tempo e o efeito do posicionamento lateral dos atuadores do exoesqueleto nos músculos da perna.Mestrado em Engenharia Biomédic

Closed-loop Central Pattern Generator Control of Human Gaits in OpenSim Simulator

International audienceIn this paper, a new neuro-musculoskeletal gait simulation platform is presented. This platform is developed to reproduce healthy or altered walking gaits. It is based on an original model of central pattern generator able to generate variable rhythmic signals for controlling biological human leg joints. Output signals of motoneurons are applied to excitation inputs of modelled muscles of the human lower limbs model. Eight central pattern generators control a musculoskeletal model made up of three joints per leg actuated by 44 Hill-type muscle models. Forward dynamics simulation in OpenSim show that it is possible to generate different stable walking gaits by changing parameters of controller. Further work is aimed on development of stable human standing by implementing reflexes

Simulating a Flexible Robotic System based on Musculoskeletal Modeling

Humanoid robotics offers a unique research tool for understanding the human brain and body. The synthesis of human motion is a complex procedure that involves accurate reconstruction of movement sequences, modeling of musculoskeletal kinematics, dynamics and actuation, and characterization of reliable performance criteria. Many of these processes have much in common with the problems found in robotics research, with the recent advent of complex humanoid systems. This work presents the design and development of a new-generation bipedal robot. Its modeling and simulation has been realized by using an open-source software to create and analyze dynamic simulation of movement: OpenSim. Starting from a study by Fuben He, our model aims to be used as an innovative approach to the study of a such type of robot in which there are series elastic actuators represented by active and passive spring components in series with motors. It has provided of monoarticular and biarticular joint in a very similar manner to human musculoskeletal model. This thesis is only the starting point of a wide range of other possible future works: from the control structure completion and whole-body control application, to imitation learning and reinforcement learning for human locomotion, from motion test on at ground to motion test on rough ground, and obviously the transition from simulation to practice with a real elastic bipedal robot biologically-inspired that can move like a human bein

Underactuated approach for the control-based forward dynamic analysis of acquired gait motions

Peer ReviewedPostprint (published version

PREDICTIVE SIMULATION OF HUMAN MOVEMENT AND APPLICATIONS TO ASSISTIVE DEVICE DESIGN AND CONTROL

Predictive simulation based on dynamic optimization using musculoskeletal models is a powerful approach for studying biomechanics of human gait. Predictive simulation can be used for a variety of applications from designing assistive devices to testing theories of motor controls. However, one of the challenges in formulating the predictive dynamic optimization problem is that the cost function, which represents the underlying goal of the walking task (e.g., minimal energy consumption) is generally unknown and is assumed a priori. While different studies used different cost functions, the qualities of the gaits with those cost functions were often not provided. Therefore, this dissertation evaluates and examines different cost function forms for dynamic simulation of human walking. The problem of the walking cost function determination was cast as a bilevel optimization, which was solved using a nested evolutionary approach. The results showed cost functions based on a weighted combination of muscle-based performance criteria (e.g., metabolic cost, muscle fatigue), gait smoothness, and gait stability led to better walking solutions compared to any cost functions only based on muscle performance criteria. Further evaluations of the walking cost functions were done in the simulation cases of human walking augmented with assistive devices such as prosthesis and exoskeleton. The simulations of augmented walking were comparable to the experimental results, which suggests the potential of using the simulation approach to address problems of finding assistive device design and control

The Benefit of Combining Neuronal Feedback and Feed-Forward Control for Robustness in Step Down Perturbations of Simulated Human Walking Depends on the Muscle Function

It is often assumed that the spinal control of human locomotion combines feed-forward central pattern generation with sensory feedback via muscle reflexes. However, the actual contribution of each component to the generation and stabilization of gait is not well understood, as direct experimental evidence for either is difficult to obtain. We here investigate the relative contribution of the two components to gait stability in a simulation model of human walking. Specifically, we hypothesize that a simple linear combination of feedback and feed-forward control at the level of the spinal cord improves the reaction to unexpected step down perturbations. In previous work, we found preliminary evidence supporting this hypothesis when studying a very reduced model of rebounding behaviors. In the present work, we investigate if the evidence extends to a more realistic model of human walking. We revisit a model that has previously been published and relies on spinal feedback control to generate walking. We extend the control of this model with a feed-forward muscle activation pattern. The feed-forward pattern is recorded from the unperturbed feedback control output. We find that the improvement in the robustness of the walking model with respect to step down perturbations depends on the ratio between the two strategies and on the muscle to which they are applied. The results suggest that combining feed-forward and feedback control is not guaranteed to improve locomotion, as the beneficial effects are dependent on the muscle and its function during walking

Respiratory, postural and spatio-kinetic motor stabilization, internal models, top-down timed motor coordination and expanded cerebello-cerebral circuitry: a review

Human dexterity, bipedality, and song/speech vocalization in Homo are reviewed within a motor evolution perspective in regard to 

(i) brain expansion in cerebello-cerebral circuitry, 
(ii) enhanced predictive internal modeling of body kinematics, body kinetics and action organization, 
(iii) motor mastery due to prolonged practice, 
(iv) task-determined top-down, and accurately timed feedforward motor adjustment of multiple-body/artifact elements, and 
(v) reduction in automatic preflex/spinal reflex mechanisms that would otherwise restrict such top-down processes. 

Dual-task interference and developmental neuroimaging research argues that such internal modeling based motor capabilities are concomitant with the evolution of 
(vi) enhanced attentional, executive function and other high-level cognitive processes, and that 
(vii) these provide dexterity, bipedality and vocalization with effector nonspecific neural resources. 

The possibility is also raised that such neural resources could 
(viii) underlie human internal model based nonmotor cognitions. 
&#xa