Learning-based methods for planning and control of humanoid robots

Nowadays, humans and robots are more and more likely to coexist as time goes by. The anthropomorphic nature of humanoid robots facilitates physical human-robot interaction, and makes social human-robot interaction more natural. Moreover, it makes humanoids ideal candidates for many applications related to tasks and environments designed for humans. No matter the application, an ubiquitous requirement for the humanoid is to possess proper locomotion skills. Despite long-lasting research, humanoid locomotion is still far from being a trivial task. A common approach to address humanoid locomotion consists in decomposing its complexity by means of a model-based hierarchical control architecture. To cope with computational constraints, simplified models for the humanoid are employed in some of the architectural layers. At the same time, the redundancy of the humanoid with respect to the locomotion task as well as the closeness of such a task to human locomotion suggest a data-driven approach to learn it directly from experience. This thesis investigates the application of learning-based techniques to planning and control of humanoid locomotion. In particular, both deep reinforcement learning and deep supervised learning are considered to address humanoid locomotion tasks in a crescendo of complexity. First, we employ deep reinforcement learning to study the spontaneous emergence of balancing and push recovery strategies for the humanoid, which represent essential prerequisites for more complex locomotion tasks. Then, by making use of motion capture data collected from human subjects, we employ deep supervised learning to shape the robot walking trajectories towards an improved human-likeness. The proposed approaches are validated on real and simulated humanoid robots. Specifically, on two versions of the iCub humanoid: iCub v2.7 and iCub v3

Uniform finite time stabilisation of non-smooth and variable structure systems with resets

This thesis studies uniform finite time stabilisation of uncertain variable structure and non-smooth systems with resets. Control of unilaterally constrained systems is a challenging area that requires an understanding of the underlying mechanics that give rise to reset or jumps while synthesizing stabilizing controllers. Discontinuous systems with resets are studied in various disciplines. Resets in states are hard nonlinearities. This thesis bridges non-smooth Lyapunov analysis, the quasi-homogeneity of differential inclusions and uniform finite time stability for a class of impact mechanical systems. Robust control synthesis based on second order sliding mode is undertaken in the presence of both impacts with finite accumulation time and persisting disturbances. Unlike existing work described in the literature, the Lyapunov analysis does not depend on the jumps in the state while also establishing proofs of uniform finite time stability. Orbital stabilization of fully actuated mechanical systems is established in the case of persisting impacts with an a priori guarantee of finite time convergence between t he periodic impacts. The distinguishing features of second order sliding mode controllers are their simplicity and robustness. Increasing research interest in the area has been complemented by recent advances in Lyapullov based frameworks which highlight the finite time Convergence property. This thesis computes the upper bound on the finite settling time of a second order sliding mode controller. Different to the latest advances in the area, a key contribution of this thesis is the theoretical proof of the fact that finite settling time of a second order sliding mode controller tends to zero when gains tend to infinity. This insight of the limiting behaviour forms the basis for solving the converse problem of finding an explicit a priori tuning formula for the gain parameters of the controller when and arbitrary finite settling time is given. These results play a central role ill the analysis of impact mechanical systems. Another key contribution of the thesis is that it extends the above results on variable structure systems with and without resets to non-smooth systems arising from continuous finite time controllers while proving uniform finite time stability. Finally, two applications are presented. The first application applies the above theoretical developments to the problem of orbital stabilization of a fully actuated seven link biped robot which is a nonlinear system with periodic impacts. The tuning of the controller gains leads to finite time convergence of the tracking errors between impacts while being robust to disturbances. The second application reports the outcome of an experiment with a continuous finite time controller

Dynamic Bipedal Locomotion: From Hybrid Zero Dynamics to Control Lyapunov Functions via Experimentally Realizable Methods

Robotic bipedal locomotion has become a rapidly growing field of research as humans increasingly look to augment their natural environments with intelligent machines. In order for these robotic systems to navigate the often unstructured environments of the world and perform tasks, they must first have the capability to dynamically, reliably, and efficiently locomote. Due to the inherently hybrid and underactuated nature of dynamic bipedal walking, the greatest experimental successes in the field have often been achieved by considering all aspects of the problem; with explicit consideration of the interplay between modeling, trajectory planning, and feedback control. The methodology and developments presented in this thesis begin with the modeling and design of dynamic walking gaits on bipedal robots through hybrid zero dynamics (HZD), a mathematical framework that utilizes hybrid system models coupled with nonlinear controllers that results in stable locomotion. This will form the first half of the thesis, and will be used to develop a solid foundation of HZD trajectory optimization tools and algorithms for efficient synthesis of accurate hybrid motion plans for locomotion on two underactuated and compliant 3D bipeds. While HZD and the associated trajectory optimization are an existing framework, the resulting behaviors shown in these preliminary experiments will extend the limits of what HZD has demonstrated is possible thus far in the literature. Specifically, the core results of this thesis demonstrate the first experimental multi-contact humanoid walking with HZD on the DURUS robot and then through the first compliant HZD motion library for walking over a continuum of walking speeds on the Cassie robot. On the theoretical front, a novel formulation of an optimization-based control framework is introduced that couples convergence constraints from control Lyapunov functions (CLF)s with desirable formulations existing in other areas of the bipedal locomotion field that have proven successful in practice, such as inverse dynamics control and quadratic programming approaches. The theoretical analysis and experimental validation of this controller thus forms the second half of this thesis. First, a theoretical analysis is developed which demonstrates several useful properties of the approach for tuning and implementation, and the stability of the controller for HZD locomotion is proven. This is then extended to a relaxed version of the CLF controller, which removes a convergence inequality constraint in lieu of a conservative CLF cost within a quadratic program to achieve tracking. It is then explored how this new CLF formulation can fully leverage the planned HZD walking gaits to achieve the target performance on physical hardware. Towards this goal, an experimental implementation of the CLF controller is derived for the Cassie robot, with the resulting experiments demonstrating the first successful realization of a CLF controller for a 3D biped on hardware in the literature. The accuracy of the robot model and synthesized HZD motion library allow the real-time control implementation to regularize the CLF optimization cost about the nominal walking gait. This drives the controller to choose smooth input torques and anticipated spring torques, as well as regulate an optimal distribution of feasible ground reaction forces on hardware while reliably tracking the planned virtual constraints. These final results demonstrate how each component of this thesis were brought together to form an effective end-to-end implementation of a nonlinear control framework for underactuated locomotion on a bipedal robot through modeling, trajectory optimization, and then ultimately real-time control.</p

Instantaneous Momentum-Based Control of Floating Base Systems

In the last two decades a growing number of robotic applications such as autonomous drones, wheeled robots and industrial manipulators started to be employed in several human environments. However, these machines often possess limited locomotion and/or manipulation capabilities, thus reducing the number of achievable tasks and increasing the complexity of robot-environment interaction. Augmenting robots locomotion and manipulation abilities is a fundamental research topic, with a view to enhance robots participation in complex tasks involving safe interaction and cooperation with humans. To this purpose, humanoid robots, aerial manipulators and the novel design of flying humanoid robots are among the most promising platforms researchers are studying in the attempt to remove the existing technological barriers. These robots are often modeled as floating base systems, and have lost the assumption -- typical of fixed base robots -- of having one link always attached to the ground. From the robot control side, contact forces regulation revealed to be fundamental for the execution of interaction tasks. Contact forces can be influenced by directly controlling the robot's momentum rate of change, and this fact gives rise to several momentum-based control strategies. Nevertheless, effective design of force and torque controllers still remains a complex challenge. The variability of sensor load during interaction, the inaccuracy of the force/torque sensing technology and the inherent nonlinearities of robot models are only a few complexities impairing efficient robot force control. This research project focuses on the design of balancing and flight controllers for floating base robots interacting with the surrounding environment. More specifically, the research is built upon the state-of-the-art of momentum-based controllers and applied to three robotic platforms: the humanoid robot iCub, the aerial manipulator OTHex and the jet-powered humanoid robot iRonCub. The project enforces the existing literature with both theoretical and experimental results, aimed at achieving high robot performances and improved stability and robustness, in presence of different physical robot-environment interactions

Motion planning and reactive control on learnt skill manifolds

We propose a novel framework for motion planning and control that is based on a manifold encoding of the desired solution set. We present an alternate, model-free, approach to path planning, replanning and control. Our approach is founded on the idea of encoding the set of possible trajectories as a skill manifold, which can be learnt from data such as from demonstration. We describe the manifold representation of skills, a technique for learning from data and a method for generating trajectories as geodesics on such manifolds. We extend the trajectory generation method to handle dynamic obstacles and constraints. We show how a state metric naturally arises from the manifold encoding and how this can be used for reactive control in an on-line manner. Our framework tightly integrates learning, planning and control in a computationally efficient representation, suitable for realistic humanoid robotic tasks that are defined by skill specifications involving high-dimensional nonlinear dynamics, kinodynamic constraints and non-trivial cost functions, in an optimal control setting. Although, in principle, such problems can be handled by well understood analytical methods, it is often difficult and expensive to formulate models that enable the analytical approach. We test our framework with various types of robotic systems – ranging from a 3-link arm to a small humanoid robot – and show that the manifold encoding gives significant improvements in performance without loss of accuracy. Furthermore, we evaluate the framework against a state-of-the-art imitation learning method. We show that our approach, by learning manifolds of robotic skills, allows for efficient planning and replanning in changing environments, and for robust and online reactive control

Human-Inspired Balancing and Recovery Stepping for Humanoid Robots

Robustly maintaining balance on two legs is an important challenge for humanoid robots. The work presented in this book represents a contribution to this area. It investigates efficient methods for the decision-making from internal sensors about whether and where to step, several improvements to efficient whole-body postural balancing methods, and proposes and evaluates a novel method for efficient recovery step generation, leveraging human examples and simulation-based reinforcement learning