Sample Efficient Optimization for Learning Controllers for Bipedal Locomotion

Learning policies for bipedal locomotion can be difficult, as experiments are expensive and simulation does not usually transfer well to hardware. To counter this, we need al- gorithms that are sample efficient and inherently safe. Bayesian Optimization is a powerful sample-efficient tool for optimizing non-convex black-box functions. However, its performance can degrade in higher dimensions. We develop a distance metric for bipedal locomotion that enhances the sample-efficiency of Bayesian Optimization and use it to train a 16 dimensional neuromuscular model for planar walking. This distance metric reflects some basic gait features of healthy walking and helps us quickly eliminate a majority of unstable controllers. With our approach we can learn policies for walking in less than 100 trials for a range of challenging settings. In simulation, we show results on two different costs and on various terrains including rough ground and ramps, sloping upwards and downwards. We also perturb our models with unknown inertial disturbances analogous with differences between simulation and hardware. These results are promising, as they indicate that this method can potentially be used to learn control policies on hardware.Comment: To appear in International Conference on Humanoid Robots (Humanoids '2016), IEEE-RAS. (Rika Antonova and Akshara Rai contributed equally

Cycle-to-cycle control of swing phase of paraplegic gait induced by surface electrical stimulation

Parameterised swing phase of gait in paraplegics was obtained using surface electrical stimulation of the hip flexors, hamstrings and quadriceps; the hip flexors were stimulated to obtain a desired hip angle range, the hamstrings to provide foot clearance in the forward swing, and the quadriceps to acquire knee extension at the end of the swing phase. We report on two main aspects; optimisation of the initial stimulation parameters, and parameter adaption (control). The initial stimulation patterns were experimentally optimised in two paraplegic subjects using a controlled stand device, resulting in an initial satisfactory swinging motion in both subjects. Intersubject differences appeared in the mechanical output (torque joint) per muscle group. During a prolonged open-loop controlled trial with the optimised but unregulated stimulation onsets and burst duration for the three muscle groups, the hip angle range per cycle initially increased above the desired value and subsequently decreased below it. The mechanical performance of the hamstrings and quadriceps remained relatively unaffected. A cycle-to-cycle controller was then designed, operating on the basis of the hip angle ranges obtained in previous swings. This controller successfully adapted the burst duration of the hip flexors to maintain the desired hip angle range

How do treadmill speed and terrain visibility influence neuromuscular control of guinea fowl locomotion?

Locomotor control mechanisms must flexibly adapt to both anticipated and unexpected terrain changes to maintain movement and avoid a fall. Recent studies revealed that ground birds alter movement in advance of overground obstacles, but not treadmill obstacles, suggesting context-dependent shifts in the use of anticipatory control. We hypothesized that differences between overground and treadmill obstacle negotiation relate to differences in visual sensory information, which influence the ability to execute anticipatory manoeuvres. We explored two possible explanations: (1) previous treadmill obstacles may have been visually imperceptible, as they were low contrast to the tread, and (2) treadmill obstacles are visible for a shorter time compared with runway obstacles, limiting time available for visuomotor adjustments. To investigate these factors, we measured electromyographic activity in eight hindlimb muscles of the guinea fowl (Numida meleagris, N=6) during treadmill locomotion at two speeds (0.7 and 1.3 m s−1) and three terrain conditions at each speed: (i) level, (ii) repeated 5 cm low-contrast obstacles (90% contrast, black/white). We hypothesized that anticipatory changes in muscle activity would be higher for (1) high-contrast obstacles and (2) the slower treadmill speed, when obstacle viewing time is longer. We found that treadmill speed significantly influenced obstacle negotiation strategy, but obstacle contrast did not. At the slower speed, we observed earlier and larger anticipatory increases in muscle activity and shifts in kinematic timing. We discuss possible visuomotor explanations for the observed context-dependent use of anticipatory strategies

Leg Coordination during Walking in Insects

Locomotion depends on constant adaptation to different requirements of the environment. An appropriate temporal and spatial coordination of multiple body parts is necessary to achieve a stable and adapted behavior. Until now it is unclear how the neuronal structures can achieve these meaningful adaptations. The exact role of the nervous system, muscles and mechanical constrains are not known. By using preparations in which special forms of adaptations are considered under experimental conditions that selectively exclude external influences, like mechanical interactions through the ground or differences in body mass, one can draw conclusions about the organization of the respective underlying neuronal structures. In the present thesis, four different publications are presented, giving evidence of mechanisms of temporal or spatial coordination of leg movements in the stick insect Carausius morosus and the fruit fly Drosophila melanogaster during different experimental paradigms. At first, state dependent local coordinating mechanisms are analyzed. Electromyographic measurements of the three major antagonistic leg muscle pairs of the forward and backward walking stick insect are evaluated. It becomes evident that only the motor activity of the most proximal leg joint is changed when walking direction is changed from forward to backward, which demonstrates that the neuronal networks driving movement in each individual leg seem to be organized in a modular structure. In the second part mechanisms that influence movement speed of the individual leg and coordination of speed between the different legs of the stick insect come into focus. Electrophysiological and behavioral experiments with the intact and reduced stick insect were used to examine relationships between the velocity of a stepping front leg and neuronal activity in the mesothoracic segment as well as correlations between the stepping velocities of different legs during walks with constant velocity or with distinct accelerations. It was shown that stepping velocity of single legs were not reflected in motoneuron activity or stepping velocity of another leg. Only when an increase in walking speed was induced, clear correlation in the stepping velocities of the individual legs was found. Subsequently, the analysis of changes in temporal leg coordination during different walking speeds in the fruit fly reveals that the locomotor system of Drosophila can cover a broad range of walking speeds and seems to follow the same rules as the locomotor system of the stick insect. Walking speed is increased by modifying stance duration, whereas swing duration and step amplitude remain largely unchanged. Changes in inter-leg coordination are gradually and systematically with walking speed and can adapt to major biomechanical changes in its walking apparatus. In the final part it was the aim to understand the role of neuronal mechanisms for the orientation and spatial coordination of foot placement in the stick insect. Placement of middle and hind legs with respect to the position of their respective rostrally neighboring leg were analyzed under two different conditions. Segment and state dependent differences in the aiming accuracy of the middle and hind legs could be shown, which indicate differences in the underlying neuronal structures in the different segments and the importance of movement in the target leg for the processing of the position information. Taken together, common principles in inter-leg coordination where found, like similarities between different organisms and segment specific or state dependent modifications in the walking system. They can be interpreted as evidence for a highly adaptive and modular design of the underlying neuronal structures

Imprecise dynamic walking with time-projection control

We present a new walking foot-placement controller based on 3LP, a 3D model of bipedal walking that is composed of three pendulums to simulate falling, swing and torso dynamics. Taking advantage of linear equations and closed-form solutions of the 3LP model, our proposed controller projects intermediate states of the biped back to the beginning of the phase for which a discrete LQR controller is designed. After the projection, a proper control policy is generated by this LQR controller and used at the intermediate time. This control paradigm reacts to disturbances immediately and includes rules to account for swing dynamics and leg-retraction. We apply it to a simulated Atlas robot in position-control, always commanded to perform in-place walking. The stance hip joint in our robot keeps the torso upright to let the robot naturally fall, and the swing hip joint tracks the desired footstep location. Combined with simple Center of Pressure (CoP) damping rules in the low-level controller, our foot-placement enables the robot to recover from strong pushes and produce periodic walking gaits when subject to persistent sources of disturbance, externally or internally. These gaits are imprecise, i.e., emergent from asymmetry sources rather than precisely imposing a desired velocity to the robot. Also in extreme conditions, restricting linearity assumptions of the 3LP model are often violated, but the system remains robust in our simulations. An extensive analysis of closed-loop eigenvalues, viable regions and sensitivity to push timings further demonstrate the strengths of our simple controller

Assisting walking balance using a bio-inspired exoskeleton controller

Background: Balance control is important for mobility, yet exoskeleton research has mainly focused on improving metabolic energy efficiency. Here we present a biomimetic exoskeleton controller that supports walking balance and reduces muscle activity. Methods: Humans restore balance after a perturbation by adjusting activity of the muscles actuating the ankle in proportion to deviations from steady-state center of mass kinematics. We designed a controller that mimics the neural control of steady-state walking and the balance recovery responses to perturbations. This controller uses both feedback from ankle kinematics in accordance with an existing model and feedback from the center of mass velocity. Control parameters were estimated by fitting the experimental relation between kinematics and ankle moments observed in humans that were walking while being perturbed by push and pull perturbations. This identified model was implemented on a bilateral ankle exoskeleton. Results: Across twelve subjects, exoskeleton support reduced calf muscle activity in steady-state walking by 19% with respect to a minimal impedance controller (p < 0.001). Proportional feedback of the center of mass velocity improved balance support after perturbation. Muscle activity is reduced in response to push and pull perturbations by 10% (p = 0.006) and 16% (p < 0.001) and center of mass deviations by 9% (p = 0.026) and 18% (p = 0.002) with respect to the same controller without center of mass feedback. Conclusion: Our control approach implemented on bilateral ankle exoskeletons can thus effectively support steady-state walking and balance control and therefore has the potential to improve mobility in balance-impaired individuals.Support Biomechanical Engineerin

Epidural Stimulation Induced Modulation of Spinal Locomotor Networks in Adult Spinal Rats

The importance of the in vivo dynamic nature of the circuitries within the spinal cord that generate locomotion is becoming increasingly evident. We examined the characteristics of hindlimb EMG activity evoked in response to epidural stimulation at the S1 spinal cord segment in complete midthoracic spinal cord-transected rats at different stages of postlesion recovery. A progressive and phase-dependent modulation of monosynaptic (middle) and long-latency (late) stimulation-evoked EMG responses was observed throughout the step cycle. During the first 3 weeks after injury, the amplitude of the middle response was potentiated during the EMG bursts, whereas after 4 weeks, both the middle and late responses were phase-dependently modulated. The middle- and late-response magnitudes were closely linked to the amplitude and duration of the EMG bursts during locomotion facilitated by epidural stimulation. The optimum stimulation frequency that maintained consistent activity of the long-latency responses ranged from 40 to 60 Hz, whereas the short-latency responses were consistent from 5 to 130 Hz. These data demonstrate that both middle and late evoked potentials within a motor pool are strictly gated during in vivo bipedal stepping as a function of the general excitability of the motor pool and, thus, as a function of the phase of the step cycle. These data demonstrate that spinal cord epidural stimulation can facilitate locomotion in a time-dependent manner after lesion. The long-latency responses to epidural stimulation are correlated with the recovery of weight-bearing bipedal locomotion and may reflect activation of interneuronal central pattern-generating circuits

An admittance shaping controller for exoskeleton assistance of the lower extremities

We present a method for lower-limb exoskeleton control that defines assistance as a desired dynamic response for the human leg. Wearing the exoskeleton can be seen as replacing the leg's natural admittance with the equivalent admittance of the coupled system. The control goal is to make the leg obey an admittance model defined by target values of natural frequency, peak magnitude and zero-frequency response. No estimation of muscle torques or motion intent is necessary. Instead, the controller scales up the coupled system's sensitivity transfer function by means of a compensator employing positive feedback. This approach increases the leg's mobility and makes the exoskeleton an active device capable of performing net positive work on the limb. Although positive feedback is usually considered destabilizing, here performance and robust stability are successfully achieved through a constrained optimization that maximizes the system's gain margins while ensuring the desired location of its dominant poles

Detection of Forces and Body Load in Standing and Walking in the American Cockroach

Sense organs in the legs that detect body weight are an important component in the regulation of posture and locomotion. This thesis seeks to gain an understanding of how body weight is detected by sense organs of the legs, and determine how this information influences muscle activities in standing and walking. The first study tested the ability of tibial campaniform sensilla (receptors that monitor forces in the cockroach leg) to encode variations in body load using magnets attached to the thorax. Recordings of sensory activities in freely standing animals showed that proximal tibial sensilla (oriented perpendicular to the leg long axis) encode the level of body load while distal receptors (oriented parallel) fired to decreasing loads. In some postures, sensillum discharges paralleled changes in activity of the trochanteral extensor muscle consistent with a known interjoint reflex. These findings demonstrate that tibial campaniform sensilla can monitor the effects of body weight upon the legs and may aid in generating support of body load. In the second study, sensory activities were compared when animals walked freely in an arena, or upon an oiled glass plate with their body weight supported. Sensilla discharges persisted but were abbreviated when body load was reduced. The results suggest that sensory discharges early in stance result from forces generated by contractions of muscles that press the leg against the substrate. Force feedback later in stance may adjust motor output to changes in loading. In the third study, muscle activities and leg movements were recorded before and after denervation of distal leg segments. Regular bursts occurred in motoneurons to leg extensor muscles following denervation, including ‘fictive’ bursting in a muscle whose tendon (apodeme) was cut in the ablation. Similar motoneuron activities were found in walking on an oiled glass surface, when effects of body weight and mechanical coupling were minimized. When distal segments were completely severed, leg use and muscle bursting were disrupted but could be restored if the stumps were pressed against the substrate. These results support the hypothesis that feedback from receptors in proximal leg segments, that indicate forces, allow for active leg use in walking