Chaotic exploration and learning of locomotor behaviours

Recent developments in the embodied approach to understanding the generation of adaptive behaviour, suggests that the design of adaptive neural circuits for rhythmic motor patterns should not be done in isolation from an appreciation, and indeed exploitation, of neural-body-environment interactions. Utilising spontaneous mutual entrainment between neural systems and physical bodies provides a useful passage to the regions of phase space which are naturally structured by the neuralbody- environmental interactions. A growing body of work has provided evidence that chaotic dynamics can be useful in allowing embodied systems to spontaneously explore potentially useful motor patterns. However, up until now there has been no general integrated neural system that allows goal-directed, online, realtime exploration and capture of motor patterns without recourse to external monitoring, evaluation or training methods. For the first time, we introduce such a system in the form of a fully dynamic neural system, exploiting intrinsic chaotic dynamics, for the exploration and learning of the possible locomotion patterns of an articulated robot of an arbitrary morphology in an unknown environment. The controller is modelled as a network of neural oscillators which are coupled only through physical embodiment, and goal directed exploration of coordinated motor patterns is achieved by a chaotic search using adaptive bifurcation. The phase space of the indirectly coupled neural-body-environment system contains multiple transient or permanent self-organised dynamics each of which is a candidate for a locomotion behaviour. The adaptive bifurcation enables the system orbit to wander through various phase-coordinated states using its intrinsic chaotic dynamics as a driving force and stabilises the system on to one of the states matching the given goal criteria. In order to improve the sustainability of useful transient patterns, sensory homeostasis has been introduced which results in an increased diversity of motor outputs, thus achieving multi-scale exploration. A rhythmic pattern discovered by this process is memorised and sustained by changing the wiring between initially disconnected oscillators using an adaptive synchronisation method. The dynamical nature of the weak coupling through physical embodiment allows this adaptive weight learning to be easily integrated, thus forming a continuous exploration-learning system. Our result shows that the novel neuro-robotic system is able to create and learn a number of emergent locomotion behaviours for a wide range of body configurations and physical environment, and can re-adapt after sustaining damage. The implications and analyses of these results for investigating the generality and limitations of the proposed system are discussed

A Review of Control Strategies in Closed-Loop Neuroprosthetic Systems

It has been widely recognized that closed-loop neuroprosthetic systems achieve more favourable outcomes for users then equivalent open-loop devices. Improved performance of tasks, better usability and greater embodiment have all been reported in systems utilizing some form of feedback. However the interdisciplinary work on neuroprosthetic systems can lead to miscommunication due to similarities in well established nomenclature in different fields. Here we present a review of control strategies in existing experimental, investigational and clinical neuroprosthetic systems in order to establish a baseline and promote a common understanding of different feedback modes and closed loop controllers. The first section provides a brief discussion of feedback control and control theory. The second section reviews the control strategies of recent Brain Machine Interfaces, neuromodulatory implants, neuroprosthetic systems and assistive neurorobotic devices. The final section examines the different approaches to feedback in current neuroprosthetic and neurorobotic systems

Neuro-fuzzy modeling of multi-field surface neuroprostheses for hand grasp

154 p.Las neuroprótesis aplican pulsos eléctricos a los nervios periféricos con el objetivo de sustituir funciones motrices/sensoriales perdidas, dando asistencia e influyendo positivamente en la rehabilitación motriz de personas con disfunciones motrices causadas por trastornos neurológicos. La complejidad de la neuroanatomía del antebrazo y la mano, su dimensionalidad, las diversas tareas no-cíclicas, la variabilidad de movimientos entre sujetos y la reducida selectividad de las neuroprótesis superficiales, ha dado lugar al diseño de un número reducido de neuroprótesis orientadas a agarres básicos. La posibilidad de hacer más selectiva la estimulación mediante los electrodos multi-campo, junto con el conocimiento sobre la incomodidad y los movimientos que genera la aplicación de la estimulación eléctrica funcional (FES por sus siglas en inglés) en miembro superior, podrían ser base fundamental para el desarrollo de neuroprótesis de agarre más avanzadas. La presente tesis describe un análisis de incomodidad como resultado de FES en el miembro superior, y propone modelos neuro-difusos para neuroprótesis de agarre tanto para personas sanas como para personas con trastornos neurológicos. El conocimiento generado respecto a la incomodidad puede ser utilizado como guía para desarrollar aplicaciones de FES de miembro superior más cómodas. Del mismo modo, los modelos propuestos en esta tesis pueden ser utilizados para apoyar el diseño y la validación de sistemas de control avanzados en neuroprótesis dirigidas a la función de agarre.Tecnalia; Intelligent Control Research Grou

Learning to Ascend Stairs and Ramps:Deep Reinforcement Learning for a Physics-Based Human Musculoskeletal Model

This paper proposes to use deep reinforcement learning to teach a physics-based human musculoskeletal model to ascend stairs and ramps. The deep reinforcement learning architecture employs the proximal policy optimization algorithm combined with imitation learning and is trained with experimental data of a public dataset. The human model is developed in the open-source simulation software OpenSim, together with two objects (i.e., the stairs and ramp) and the elastic foundation contact dynamics. The model can learn to ascend stairs and ramps with muscle forces comparable to healthy subjects and with a forward dynamics comparable to the experimental training data, achieving an average correlation of 0.82 during stair ascent and of 0.58 during ramp ascent across both the knee and ankle joints

Prediction and control in human neuromusculoskeletal models

Computational neuromusculoskeletal modelling enables the generation and testing of hypotheses about human movement on a large scale, in silico. Humanoid models, which increasingly aim to replicate the full complexity of the human nervous and musculoskeletal systems, are built on extensive prior knowledge, extracted from anatomical imaging, kinematic and kinetic measurement, and codified as model description. Where inverse dynamic analysis is applied, its basis is in Newton's laws of motion, and in solving for muscular redundancy it is necessary to invoke knowledge of central nervous motor strategy. This epistemological approach contrasts strongly with the models of machine learning, which are generally over-parameterised and largely data-driven. Even as spectacular performance has been delivered by the application of these models in a number of discrete domains of artificial intelligence, work towards general human-level intelligence has faltered, leading many to wonder if the data-driven approach is fundamentally limited, and spurring efforts to combine machine learning with knowledge-based modelling. Through a series of five studies, this thesis explores the combination of neuromusculoskeletal modelling with machine learning in order to enhance the core tasks of prediction and control. Several principles for the development of clinically useful artificially intelligent systems emerge: stability, computational efficiency and incorporation of prior knowledge. The first study concerns the use of neural network function approximators for the prediction of internal forces during human movement, an important task with many clinical applications, but one for which the standard tools of modelling are slow and cumbersome. By training on a large dataset of motions and their corresponding forces, state of the art performance is demonstrated, with many-fold increases in inference speed enabling the deployment of trained models for use in a real time biofeedback system. Neural networks trained in this way, to imitate some optimal controller, encode a mapping from high-level movement descriptors to actuator commands, and may thus be deployed in simulation as \textit{policies} to control the actions of humanoid models. Unfortunately, the high complexity of realistic simulation makes stable control a challenging task, beyond the capabilities of such naively trained models. The objective of the second study was to improve performance and stability of policy-based controllers for humanoid models in simulation. A novel technique was developed, borrowing from established unsupervised adversarial methods in computer vision. This technique enabled significant gains in performance relative to a neural network baseline, without the need for additional access to the optimal controller. For the third study, increases in the capabilities of these policy-based controllers were sought. Reinforcement learning is widely considered the most powerful means of optimising such policies, but it is computationally inefficient, and this inefficiency limits its clinical utility. To mitigate this problem, a novel framework, making use of domain-specific knowledge present in motion data, and in an inverse model of the biomechanical system, was developed. Training on simple desktop hardware, this framework enabled rapid initialisation of humanoid models that were able to move naturally through a 3-dimensional simulated environment, with 900-fold improvements in sample efficiency relative to a related technique based on pure reinforcement learning. After training with subject-specific anatomical parameters, and motion data, learned policies represent personalised models of motor control that may be further interrogated to test hypotheses about movement. For the fourth study, subject-specific controllers were taken and used as the substrate for transfer learning, by removing kinematic constraints and optimising with respect to the magnitude of the medial knee joint reaction force, an important biomechanical variable in osteoarthritis of the knee. Models learned new kinematic strategies for the reduction of this biomarker, which were subsequently validated by their use, in the real world, to construct subject-specific routines for real time gait retraining. Six out of eight subjects were able to reduce medial knee joint loading by pursuing the personalised kinematic targets found in simulation. Personalisation of assistive devices, such as limb prostheses, is another area of growing interest, and one for which computational frameworks promise cost-effective solutions. Reinforcement learning provides powerful techniques for this task but the expansion of the scope of optimisation, to include previously static elements of a prosthesis, is problematic for its complexity and resulting sample inefficiency. The fifth and final study demonstrates a new algorithm that leverages the methods described in the previous studies, and additional techniques for variance control, to surmount this problem, improving sample efficiency and simultaneously, through the use of prior knowledge encoded in motion data, providing a rational means of determining optimality in the prosthesis. Trained models were able to jointly optimise motor control and prosthesis design to enable improved performance in a walking task, and optimised designs were robust to both random seed and reward specification. This algorithm could be used to speed the design and production of real personalised prostheses, representing a potent realisation of the potential benefits of combined reinforcement learning and realistic neuromusculoskeletal modelling.Open Acces

Benchmarking Cerebellar Control

Cerebellar models have long been advocated as viable models for robot dynamics control. Building on an increasing insight in and knowledge of the biological cerebellum, many models have been greatly refined, of which some computational models have emerged with useful properties with respect to robot dynamics control. Looking at the application side, however, there is a totally different picture. Not only is there not one robot on the market which uses anything remotely connected with cerebellar control, but even in research labs most testbeds for cerebellar models are restricted to toy problems. Such applications hardly ever exceed the complexity of a 2 DoF simulated robot arm; a task which is hardly representative for the field of robotics, or relates to realistic applications. In order to bring the amalgamation of the two fields forwards, we advocate the use of a set of robotics benchmarks, on which existing and new computational cerebellar models can be comparatively tested. It is clear that the traditional approach to solve robotics dynamics loses ground with the advancing complexity of robotic structures; there is a desire for adaptive methods which can compete as traditional control methods do for traditional robots. In this paper we try to lay down the successes and problems in the fields of cerebellar modelling as well as robot dynamics control. By analyzing the common ground, a set of benchmarks is suggested which may serve as typical robot applications for cerebellar models

Neurofly 2008 abstracts : the 12th European Drosophila neurobiology conference 6-10 September 2008 Wuerzburg, Germany

This volume consists of a collection of conference abstracts

Multi-expert learning of adaptive legged locomotion

Achieving versatile robot locomotion requires motor skills which can adapt to previously unseen situations. We propose a Multi-Expert Learning Architecture (MELA) that learns to generate adaptive skills from a group of representative expert skills. During training, MELA is first initialised by a distinct set of pre-trained experts, each in a separate deep neural network (DNN). Then by learning the combination of these DNNs using a Gating Neural Network (GNN), MELA can acquire more specialised experts and transitional skills across various locomotion modes. During runtime, MELA constantly blends multiple DNNs and dynamically synthesises a new DNN to produce adaptive behaviours in response to changing situations. This approach leverages the advantages of trained expert skills and the fast online synthesis of adaptive policies to generate responsive motor skills during the changing tasks. Using a unified MELA framework, we demonstrated successful multi-skill locomotion on a real quadruped robot that performed coherent trotting, steering, and fall recovery autonomously, and showed the merit of multi-expert learning generating behaviours which can adapt to unseen scenarios