When and Where to Step: Terrain-Aware Real-Time Footstep Location and Timing Optimization for Bipedal Robots

Online footstep planning is essential for bipedal walking robots, allowing them to walk in the presence of disturbances and sensory noise. Most of the literature on the topic has focused on optimizing the footstep placement while keeping the step timing constant. In this work, we introduce a footstep planner capable of optimizing footstep placement and step time online. The proposed planner, consisting of an Interior Point Optimizer (IPOPT) and an optimizer based on Augmented Lagrangian (AL) method with analytical gradient descent, solves the full dynamics of the Linear Inverted Pendulum (LIP) model in real time to optimize for footstep location as well as step timing at the rate of 200~Hz. We show that such asynchronous real-time optimization with the AL method (ARTO-AL) provides the required robustness and speed for successful online footstep planning. Furthermore, ARTO-AL can be extended to plan footsteps in 3D, allowing terrain-aware footstep planning on uneven terrains. Compared to an algorithm with no footstep time adaptation, our proposed ARTO-AL demonstrates increased stability in simulated walking experiments as it can resist pushes on flat ground and on a $10^{\circ}$ ramp up to 120 N and 100 N respectively. For the video, see https://youtu.be/ABdnvPqCUu4. For code, see https://github.com/WangKeAlchemist/ARTO-AL/tree/master.Comment: 32 pages, 15 figures. Submitted to Robotics and Autonomous System

Locomoção de humanoides robusta e versátil baseada em controlo analítico e física residual

Humanoid robots are made to resemble humans but their locomotion abilities are far from ours in terms of agility and versatility. When humans walk on complex terrains or face external disturbances, they combine a set of strategies, unconsciously and efficiently, to regain stability. This thesis tackles the problem of developing a robust omnidirectional walking framework, which is able to generate versatile and agile locomotion on complex terrains. We designed and developed model-based and model-free walk engines and formulated the controllers using different approaches including classical and optimal control schemes and validated their performance through simulations and experiments. These frameworks have hierarchical structures that are composed of several layers. These layers are composed of several modules that are connected together to fade the complexity and increase the flexibility of the proposed frameworks. Additionally, they can be easily and quickly deployed on different platforms. Besides, we believe that using machine learning on top of analytical approaches is a key to open doors for humanoid robots to step out of laboratories. We proposed a tight coupling between analytical control and deep reinforcement learning. We augmented our analytical controller with reinforcement learning modules to learn how to regulate the walk engine parameters (planners and controllers) adaptively and generate residuals to adjust the robot’s target joint positions (residual physics). The effectiveness of the proposed frameworks was demonstrated and evaluated across a set of challenging simulation scenarios. The robot was able to generalize what it learned in one scenario, by displaying human-like locomotion skills in unforeseen circumstances, even in the presence of noise and external pushes.Os robôs humanoides são feitos para se parecerem com humanos, mas suas habilidades de locomoção estão longe das nossas em termos de agilidade e versatilidade. Quando os humanos caminham em terrenos complexos ou enfrentam distúrbios externos combinam diferentes estratégias, de forma inconsciente e eficiente, para recuperar a estabilidade. Esta tese aborda o problema de desenvolver um sistema robusto para andar de forma omnidirecional, capaz de gerar uma locomoção para robôs humanoides versátil e ágil em terrenos complexos. Projetámos e desenvolvemos motores de locomoção sem modelos e baseados em modelos. Formulámos os controladores usando diferentes abordagens, incluindo esquemas de controlo clássicos e ideais, e validámos o seu desempenho por meio de simulações e experiências reais. Estes frameworks têm estruturas hierárquicas compostas por várias camadas. Essas camadas são compostas por vários módulos que são conectados entre si para diminuir a complexidade e aumentar a flexibilidade dos frameworks propostos. Adicionalmente, o sistema pode ser implementado em diferentes plataformas de forma fácil. Acreditamos que o uso de aprendizagem automática sobre abordagens analíticas é a chave para abrir as portas para robôs humanoides saírem dos laboratórios. Propusemos um forte acoplamento entre controlo analítico e aprendizagem profunda por reforço. Expandimos o nosso controlador analítico com módulos de aprendizagem por reforço para aprender como regular os parâmetros do motor de caminhada (planeadores e controladores) de forma adaptativa e gerar resíduos para ajustar as posições das juntas alvo do robô (física residual). A eficácia das estruturas propostas foi demonstrada e avaliada em um conjunto de cenários de simulação desafiadores. O robô foi capaz de generalizar o que aprendeu em um cenário, exibindo habilidades de locomoção humanas em circunstâncias imprevistas, mesmo na presença de ruído e impulsos externos.Programa Doutoral em Informátic

Learning-based methods for planning and control of humanoid robots

Nowadays, humans and robots are more and more likely to coexist as time goes by. The anthropomorphic nature of humanoid robots facilitates physical human-robot interaction, and makes social human-robot interaction more natural. Moreover, it makes humanoids ideal candidates for many applications related to tasks and environments designed for humans. No matter the application, an ubiquitous requirement for the humanoid is to possess proper locomotion skills. Despite long-lasting research, humanoid locomotion is still far from being a trivial task. A common approach to address humanoid locomotion consists in decomposing its complexity by means of a model-based hierarchical control architecture. To cope with computational constraints, simplified models for the humanoid are employed in some of the architectural layers. At the same time, the redundancy of the humanoid with respect to the locomotion task as well as the closeness of such a task to human locomotion suggest a data-driven approach to learn it directly from experience. This thesis investigates the application of learning-based techniques to planning and control of humanoid locomotion. In particular, both deep reinforcement learning and deep supervised learning are considered to address humanoid locomotion tasks in a crescendo of complexity. First, we employ deep reinforcement learning to study the spontaneous emergence of balancing and push recovery strategies for the humanoid, which represent essential prerequisites for more complex locomotion tasks. Then, by making use of motion capture data collected from human subjects, we employ deep supervised learning to shape the robot walking trajectories towards an improved human-likeness. The proposed approaches are validated on real and simulated humanoid robots. Specifically, on two versions of the iCub humanoid: iCub v2.7 and iCub v3

Motion Control of the Hybrid Wheeled-Legged Quadruped Robot Centauro

Emerging applications will demand robots to deal with a complex environment, which lacks the structure and predictability of the industrial workspace. Complex scenarios will require robot complexity to increase as well, as compared to classical topologies such as fixed-base manipulators, wheeled mobile platforms, tracked vehicles, and their combinations. Legged robots, such as humanoids and quadrupeds, promise to provide platforms which are flexible enough to handle real world scenarios; however, the improved flexibility comes at the cost of way higher control complexity. As a trade-off, hybrid wheeled-legged robots have been proposed, resulting in the mitigation of control complexity whenever the ground surface is suitable for driving. Following this idea, a new hybrid robot called Centauro has been developed inside the Humanoid and Human Centered Mechatronics lab at Istituto Italiano di Tecnologia (IIT). Centauro is a wheeled-legged quadruped with a humanoid bi-manual upper-body. Differently from other platform of similar concept, Centauro employs customized actuation units, which provide high torque outputs, moderately fast motions, and the possibility to control the exerted torque. Moreover, with more than forty motors moving its limbs, Centauro is a very redundant platform, with the potential to execute many different tasks at the same time. This thesis deals with the design and development of a software architecture, and a control system, tailored to such a robot; both wheeled and legged locomotion strategies have been studied, as well as prioritized, whole-body and interaction controllers exploiting the robot torque control capabilities, and capable to handle the system redundancy. A novel software architecture, made of (i) a real-time robotic middleware, and (ii) a framework for online, prioritized Cartesian controller, forms the basis of the entire work

Learning Control Policies for Fall Prevention and Safety in Bipedal Locomotion

The ability to recover from an unexpected external perturbation is a fundamental motor skill in bipedal locomotion. An effective response includes the ability to not just recover balance and maintain stability but also to fall in a safe manner when balance recovery is physically infeasible. For robots associated with bipedal locomotion, such as humanoid robots and assistive robotic devices that aid humans in walking, designing controllers which can provide this stability and safety can prevent damage to robots or prevent injury related medical costs. This is a challenging task because it involves generating highly dynamic motion for a high-dimensional, non-linear and under-actuated system with contacts. Despite prior advancements in using model-based and optimization methods, challenges such as requirement of extensive domain knowledge, relatively large computational time and limited robustness to changes in dynamics still make this an open problem. In this thesis, to address these issues we develop learning-based algorithms capable of synthesizing push recovery control policies for two different kinds of robots : Humanoid robots and assistive robotic devices that assist in bipedal locomotion. Our work can be branched into two closely related directions : 1) Learning safe falling and fall prevention strategies for humanoid robots and 2) Learning fall prevention strategies for humans using a robotic assistive devices. To achieve this, we introduce a set of Deep Reinforcement Learning (DRL) algorithms to learn control policies that improve safety while using these robots. To enable efficient learning, we present techniques to incorporate abstract dynamical models, curriculum learning and a novel method of building a graph of policies into the learning framework. We also propose an approach to create virtual human walking agents which exhibit similar gait characteristics to real-world human subjects, using which, we learn an assistive device controller to help virtual human return to steady state walking after an external push is applied. Finally, we extend our work on assistive devices and address the challenge of transferring a push-recovery policy to different individuals. As walking and recovery characteristics differ significantly between individuals, exoskeleton policies have to be fine-tuned for each person which is a tedious, time consuming and potentially unsafe process. We propose to solve this by posing it as a transfer learning problem, where a policy trained for one individual can adapt to another without fine tuning.Ph.D

Motion planning for manipulation and/or navigation tasks with emphasis on humanoid robots

This thesis handles the motion planning problem for various robotic platforms. This is a fundamental problem, especially referring to humanoid robots for which it is particularly challenging for a number of reasons. The first is the high number of degrees of freedom. The second is that a humanoid robot is not a free-flying system in its configuration space: its motions must be generated appropriately. Finally, the implicit requirement that the robot maintains equilibrium, either static or dynamic, typically constrains the trajectory of the robot center of mass. In particular, we are interested in handling problems in which the robot must execute a task, possibly requiring stepping, in environments cluttered by obstacles. In order to solve this problem, we propose to use offline probabilistic motion planning techniques such as Rapidly Exploring Random Trees (RRTs) that consist in finding a solution by means of a graph built in an appropriately defined configuration space. The novelty of the approach is that it does not separate locomotion from task execution. This feature allows to generate whole-body movements while fulfilling the task. The task can be assigned as a trajectory or a single point in the task space or even combining tasks of different nature (e.g., manipulation and navigation tasks). The proposed method is also able to deform the task, if the assigned one is too difficult to be fulfilled. It automatically detects when the task should be deformed and which kind of deformation to apply. However, there are situations, especially when robots and humans have to share the same workspace, in which the robot has to be equipped with reactive capabilities (as avoiding moving obstacles), allowing to reach a basic level of safety. The final part of the thesis handles the rearrangement planning problem. This problem is interesting in view of manipulation tasks, where the robot has to interact with objects in the environment. Roughly speaking, the goal of this problem is to plan the motion for a robot whose assigned a task (e.g., move a target object in a goal region). Doing this, the robot is allowed to move some movable objects that are in the environment. The problem is difficult because we must plan in continuous, high-dimensional state and action spaces. Additionally, the physical constraints induced by the nonprehensile interaction between the robot and the objects in the scene must be respected. Our insight is to embed physics models in the planning stage, allowing robot manipulation and simultaneous objects interaction. Throughout the thesis, we evaluate the proposed planners through experiments on different robotic platforms

Bio-Inspired Robotics

Modern robotic technologies have enabled robots to operate in a variety of unstructured and dynamically-changing environments, in addition to traditional structured environments. Robots have, thus, become an important element in our everyday lives. One key approach to develop such intelligent and autonomous robots is to draw inspiration from biological systems. Biological structure, mechanisms, and underlying principles have the potential to provide new ideas to support the improvement of conventional robotic designs and control. Such biological principles usually originate from animal or even plant models, for robots, which can sense, think, walk, swim, crawl, jump or even fly. Thus, it is believed that these bio-inspired methods are becoming increasingly important in the face of complex applications. Bio-inspired robotics is leading to the study of innovative structures and computing with sensory–motor coordination and learning to achieve intelligence, flexibility, stability, and adaptation for emergent robotic applications, such as manipulation, learning, and control. This Special Issue invites original papers of innovative ideas and concepts, new discoveries and improvements, and novel applications and business models relevant to the selected topics of ``Bio-Inspired Robotics''. Bio-Inspired Robotics is a broad topic and an ongoing expanding field. This Special Issue collates 30 papers that address some of the important challenges and opportunities in this broad and expanding field

Using Model-based Optimal Control for Conceptional Motion Generation for the Humannoid Robot HRP-2 14 and Design Investigations for Exo-Skeletons

The research field of bipedal locomotion has been active since a few decades now. At one hand, the legged locomotion principle comprises highly flexible and robust mobility for technical applications. At the other hand, a thorough technical understanding of bipedalism supports efforts of clinicians and engineers to help people, suffering from reduced locomotion capabilities caused by fatal incidents. Since the technology enabled the construction of numerous robotic devices, among them: various humanoids, researchers started to investigate bipedalism by abstraction and adoption for technical applications. Findings from humanoid robotics are further exploited for the construction of devices for human performance augmentation and mobility support or gait rehabilitation, among them: orthosis and exo-skeletons. Although this research continuously progresses, the motion capacities of humanoid robots still lack far behind those of humans in terms of forward velocity, robustness and appearance of the overall motion. Generally, it is claimed that the difference of performance between humans and robotics is not only due to the limiting characteristics of the employed technology, e.g. constructive lack of specific determinants of gait for bipedalism or dynamic limits of the actuation system, but as well to the adopted methods for motion generation and control. For humanoid robotics, methods for motion generation are classified into optimization-based methods and those that employ heuristics, that are mostly distinguished based on the problem complexity (computation time) and the resulting dynamic error between the generated motion and the dynamics of the real robot. The implementation of the dynamic motion on the robotic platform is usually comprised with an on-line stabilizing control system. This control system must then identify and resolve instantaneously the dynamic error to maintain a continuously stable operation of the device. A large dynamic error and breach of the dynamic limits of the actuation system can quickly lead to a fatal destabilization of the device. This work proposes a contribution to the model computation and the strategy of the problem formulation of direct multiple-shooting based optimal control (Bock et. al.) for dynamically stable optimization-based motion generation. The computation of the whole-body dynamic model inside the optimization relies either on forward or inverse dynamics approach. As the inverse dynamics approach has frequently been perceived as less resource intensive than the forward dynamics approach, a new generic algorithm for insufficiently constrained, under-actuated dynamic systems has been developed and thoroughly tested to comply with all numerical restrictions of the enveloping optimization algorithm. Based on this contribution, various optimal control problems for the humanoid platform HRP-2 14 have been formulated to assess the influence of different biologically inspired optimization criteria on the final motion characteristics of walking motions. From thorough bibliographic researches a dynamically more accurate model was comprised, by taking into account the impact absorbing element in the ankle joint complex. Based on the experiences of the previous study, a problem formulation for the limiting case of, dynamically overstepping an obstacle of 20cm x 11cm (height x width) with only two steps, while maintaining its stable operation was accomplished. This is a new record for this platform. In a further part, this work proposes an iterative comprehensive model-based optimal control approach for the conception of a lower limb exo-skeleton that respects the integrated nature of such a mechatronic device. In this contribution, a human effectively wearing such a lower limb exo-skeleton is modeled. The approach then substantiates all system components in an iterative procedure, based on the complete system model, effectively resolving all complex inter-dependencies between the different components of the system. The study in this work is conducted on an important benchmark motion, walking, of a healthy human being. From this study the limiting characteristics of the system are determined and substantial propositions to the realization of various system components are formulated