Training Physics-based Controllers for Articulated Characters with Deep Reinforcement Learning

In this thesis, two different applications are discussed for using machine learning techniques to train coordinated motion controllers in arbitrary characters in absence of motion capture data. The methods highlight the resourcefulness of physical simulations to generate synthetic and generic motion data that can be used to learn various targeted skills. First, we present an unsupervised method for learning loco-motion skills in virtual characters from a low dimensional latent space which captures the coordination between multiple joints. We use a technique called motor babble, wherein a character interacts with its environment by actuating its joints through uncoordinated, low-level (motor) excitation, resulting in a corpus of motion data from which a manifold latent space can be extracted. Using reinforcement learning, we then train the character to learn locomotion (such as walking or running) in the low-dimensional latent space instead of the full-dimensional joint action space. The thesis also presents an end-to-end automated framework for training physics-based characters to rhythmically dance to user-input songs. A generative adversarial network (GAN) architecture is proposed that learns to generate physically stable dance moves through repeated interactions with the environment. These moves are then used to construct a dance network that can be used for choreography. Using DRL, the character is then trained to perform these moves, without losing balance and rhythm, in the presence of physical forces such as gravity and friction

Octopus-inspired multi-arm robotic swimming

The outstanding locomotor and manipulation characteristics of the octopus have recently inspired the development, by our group, of multi-functional robotic swimmers, featuring both manipulation and locomotion capabilities, which could be of significant engineering interest in underwater applications. During its little-studied arm-swimming behavior, as opposed to the better known jetting via the siphon, the animal appears to generate considerable propulsive thrust and rapid acceleration, predominantly employing movements of its arms. In this work, we capture the fundamental characteristics of the corresponding complex pattern of arm motion by a sculling profile, involving a fast power stroke and a slow recovery stroke. We investigate the propulsive capabilities of a multi-arm robotic system under various swimming gaits, namely patterns of arm coordination, which achieve the generation of forward, as well as backward, propulsion and turning. A lumped-element model of the robotic swimmer, which considers arm compliance and the interaction with the aquatic environment, was used to study the characteristics of these gaits, the effect of various kinematic parameters on propulsion, and the generation of complex trajectories. This investigation focuses on relatively high-stiffness arms. Experiments employing a compliant-body robotic prototype swimmer with eight compliant arms, all made of polyurethane, inside a water tank, successfully demonstrated this novel mode of underwater propulsion. Speeds of up to 0.26 body lengths per second (approximately 100 mm s(-1)), and propulsive forces of up to 3.5 N were achieved, with a non-dimensional cost of transport of 1.42 with all eight arms and of 0.9 with only two active arms. The experiments confirmed the computational results and verified the multi-arm maneuverability and simultaneous object grasping capability of such systems

Legged Robots for Object Manipulation: A Review

Legged robots can have a unique role in manipulating objects in dynamic, human-centric, or otherwise inaccessible environments. Although most legged robotics research to date typically focuses on traversing these challenging environments, many legged platform demonstrations have also included "moving an object" as a way of doing tangible work. Legged robots can be designed to manipulate a particular type of object (e.g., a cardboard box, a soccer ball, or a larger piece of furniture), by themselves or collaboratively. The objective of this review is to collect and learn from these examples, to both organize the work done so far in the community and highlight interesting open avenues for future work. This review categorizes existing works into four main manipulation methods: object interactions without grasping, manipulation with walking legs, dedicated non-locomotive arms, and legged teams. Each method has different design and autonomy features, which are illustrated by available examples in the literature. Based on a few simplifying assumptions, we further provide quantitative comparisons for the range of possible relative sizes of the manipulated object with respect to the robot. Taken together, these examples suggest new directions for research in legged robot manipulation, such as multifunctional limbs, terrain modeling, or learning-based control, to support a number of new deployments in challenging indoor/outdoor scenarios in warehouses/construction sites, preserved natural areas, and especially for home robotics.Comment: Preprint of the paper submitted to Frontiers in Mechanical Engineerin

Structured light enables biomimetic swimming and versatile locomotion of photoresponsive soft microrobots.

Microorganisms move in challenging environments by periodic changes in body shape. In contrast, current artificial microrobots cannot actively deform, exhibiting at best passive bending under external fields. Here, by taking advantage of the wireless, scalable and spatiotemporally selective capabilities that light allows, we show that soft microrobots consisting of photoactive liquid-crystal elastomers can be driven by structured monochromatic light to perform sophisticated biomimetic motions. We realize continuum yet selectively addressable artificial microswimmers that generate travelling-wave motions to self-propel without external forces or torques, as well as microrobots capable of versatile locomotion behaviours on demand. Both theoretical predictions and experimental results confirm that multiple gaits, mimicking either symplectic or antiplectic metachrony of ciliate protozoa, can be achieved with single microswimmers. The principle of using structured light can be extended to other applications that require microscale actuation with sophisticated spatiotemporal coordination for advanced microrobotic technologies.This work was in part supported by the European Research Council under the ERC Grant agreements 278213 and 291349, and the DFG as part of the project SPP 1726 (microswimmers, FI 1966/1-1). SP acknowledges support by the Max Planck ETH Center for Learning Systems.This is the author accepted manuscript. The final version is available from Nature Publishing Group via http://dx.doi.org/10.1038/nmat456

A contact-implicit direct trajectory optimization scheme for the study of legged maneuverability

For legged robots to move safely in unpredictable environments, they need to be manoeuvrable, but transient motions such as acceleration, deceleration and turning have been the subject of little research compared to constant-speed gait. They are difficult to study for two reasons: firstly, the way they are executed is highly sensitive to factors such as morphology and traction, and secondly, they can potentially be dangerous, especially when executed rapidly, or from high speeds. These challenges make it an ideal topic for study by simulation, as this allows all variables to be precisely controlled, and puts no human, animal or robotic subjects at risk. Trajectory optimization is a promising method for simulating these manoeuvres, because it allows complete motion trajectories to be generated when neither the input actuation nor the output motion is known. Furthermore, it produces solutions that optimize a given objective, such as minimizing the distance required to stop, or the effort exerted by the actuators throughout the motion. It has consequently become a popular technique for high-level motion planning in robotics, and for studying locomotion in biomechanics. In this dissertation, we present a novel approach to studying motion with trajectory optimization, by viewing it more as “trajectory generation” – a means of generating large quantities of synthetic data that can illuminate the differences between successful and unsuccessful motion strategies when studied in aggregate. One distinctive feature of this approach is the focus on whole-body models, which capture the specific morphology of the subject, rather than the highly-simplified “template” models that are typically used. Another is the use of “contact-implicit” methods, which allow an appropriate footfall sequence to be discovered, rather than requiring that it be defined upfront. Although contact-implicit methods are not novel, they are not widely-used, as they are computationally demanding, and unnecessary when studying comparatively-predictable constant speed locomotion. The second section of this dissertation describes innovations in the formulation of these trajectory optimization problems as nonlinear programming problems (NLPs). This “direct” approach allows these problems to be solved by general-purpose, open-source algorithms, making it accessible to scientists without the specialized applied mathematics knowledge required to solve NLPs. The design of the NLP has a significant impact on the accuracy of the result, the quality of the solution (with respect to the final value of the objective function), and the time required to solve the proble

The landscape of movement control in locomotion: Cost, strategy, and solution

Features of gait are determined at multiple levels, from the selection of the gait itself (e.g., walk or run) through the specific parameters utilized (stride length, frequency, etc.) to the pattern of muscular excitation. The ultimate choices are determined neurally, but what is involved with deciding on the appropriate strategy? Human locomotion appears stereotyped not so much because the pattern is predetermined, but because these movement patterns are good solutions for providing movement utilizing the machinery available to the individual (the legs and their requisite components). Under different circumstances the appropriate solution may differ broadly (different gait) or subtly (different parameters). Interpretation of the neural decision making process would benefit from understanding the influences that are utilized in the selection of the appropriate solution in any set of circumstances, including normal conditions. In this review we survey an array of studies that point to energetic cost as a key input to the gait coordination system, and not just an outcome of the gait pattern implemented. We then use that information to rigorously define the construct proposed by Sparrow and Newell (1998) where the effects of environment, organism, and task act as constraints determining the solution set available, and the coordination pattern is then implemented under pressure for energetic economy. The fit between the environment and the organism define affordances that can be actualized. We rely on a novel conceptualization of task that recognizes that the task goal needs to be separated from the mechanisms that achieve it so that the selection of a particular implementation strategy can be exposed and understood. This reformulation of the Sparrow and Newell construct is then linked to the proposed pressure for economy by considering it as an optimization problem, where the most readily selected gait strategy will be the one that achieves the task goal at (or near) the energetic minimum