Superando la brecha de la realidad: Algoritmos de aprendizaje por imitación y por refuerzos para problemas de locomoción robótica bípeda

ilustraciones, diagramas, fotografíasEsta tesis presenta una estrategia de entrenamiento de robots que utiliza técnicas de aprendizaje artificial para optimizar el rendimiento de los robots en tareas complejas. Motivado por los impresionantes logros recientes en el aprendizaje automático, especialmente en juegos y escenarios virtuales, el proyecto tiene como objetivo explorar el potencial de estas técnicas para mejorar las capacidades de los robots más allá de la programación humana tradicional a pesar de las limitaciones impuestas por la brecha de la realidad. El caso de estudio seleccionado para esta investigación es la locomoción bípeda, ya que permite dilucidar los principales desafíos y ventajas de utilizar métodos de aprendizaje artificial para el aprendizaje de robots. La tesis identifica cuatro desafíos principales en este contexto: la variabilidad de los resultados obtenidos de los algoritmos de aprendizaje artificial, el alto costo y riesgo asociado con la realización de experimentos en robots reales, la brecha entre la simulación y el comportamiento del mundo real, y la necesidad de adaptar los patrones de movimiento humanos a los sistemas robóticos. La propuesta consiste en tres módulos principales para abordar estos desafíos: Enfoques de Control No Lineal, Aprendizaje por Imitación y Aprendizaje por Reforzamiento. El módulo de Enfoques de Control No Lineal establece una base al modelar robots y emplear técnicas de control bien establecidas. El módulo de Aprendizaje por Imitación utiliza la imitación para generar políticas iniciales basadas en datos de captura de movimiento de referencia o resultados preliminares de políticas para crear patrones de marcha similares a los humanos y factibles. El módulo de Aprendizaje por Refuerzos complementa el proceso mejorando de manera iterativa las políticas paramétricas, principalmente a través de la simulación pero con el rendimiento en el mundo real como objetivo final. Esta tesis enfatiza la modularidad del enfoque, permitiendo la implementación de los módulos individuales por separado o su combinación para determinar la estrategia más efectiva para diferentes escenarios de entrenamiento de robots. Al utilizar una combinación de técnicas de control establecidas, aprendizaje por imitación y aprendizaje por refuerzos, la estrategia de entrenamiento propuesta busca desbloquear el potencial para que los robots alcancen un rendimiento optimizado en tareas complejas, contribuyendo al avance de la inteligencia artificial en la robótica no solo en sistemas virtuales sino en sistemas reales.The thesis introduces a comprehensive robot training framework that utilizes artificial learning techniques to optimize robot performance in complex tasks. Motivated by recent impressive achievements in machine learning, particularly in games and virtual scenarios, the project aims to explore the potential of these techniques for improving robot capabilities beyond traditional human programming. The case study selected for this investigation is bipedal locomotion, as it allows for elucidating key challenges and advantages of using artificial learning methods for robot learning. The thesis identifies four primary challenges in this context: the variability of results obtained from artificial learning algorithms, the high cost and risk associated with conducting experiments on real robots, the reality gap between simulation and real-world behavior, and the need to adapt human motion patterns to robotic systems. The proposed approach consists of three main modules to address these challenges: Non-linear Control Approaches, Imitation Learning, and Reinforcement Learning. The Non-linear Control module establishes a foundation by modeling robots and employing well-established control techniques. The Imitation Learning module utilizes imitation to generate initial policies based on reference motion capture data or preliminary policy results to create feasible human-like gait patterns. The Reinforcement Learning module complements the process by iteratively improving parametric policies, primarily through simulation but ultimately with real-world performance as the ultimate goal. The thesis emphasizes the modularity of the approach, allowing for the implementation of individual modules separately or their combination to determine the most effective strategy for different robot training scenarios. By employing a combination of established control techniques, imitation learning, and reinforcement learning, the framework seeks to unlock the potential for robots to achieve optimized performances in complex tasks, contributing to the advancement of artificial intelligence in robotics.DoctoradoDoctor en ingeniería mecánica y mecatrónic

Finite-time disturbance reconstruction and robust fractional-order controller design for hybrid port-Hamiltonian dynamics of biped robots

In this paper, disturbance reconstruction and robust trajectory tracking control of biped robots with hybrid dynamics in the port-Hamiltonian form is investigated. A new type of Hamiltonian function is introduced, which ensures the finite-time stability of the closed-loop system. The proposed control system consists of two loops: an inner and an outer loop. A fractional proportional-integral-derivative filter is used to achieve finite-time convergence for position tracking errors at the outer loop. A fractional-order sliding mode controller acts as a centralized controller at the inner-loop, ensuring the finite-time stability of the velocity tracking error. In this loop, the undesired effects of unknown external disturbance and parameter uncertainties are compensated using estimators. Two disturbance estimators are envisioned. The former is designed using fractional calculus. The latter is an adaptive estimator, and it is constructed using the general dynamic of biped robots. Stability analysis shows that the closed-loop system is finite-time stable in both contact-less and impact phases. Simulation studies on two types of biped robots (i.e., two-link walker and RABBIT biped robot) demonstrate the proposed controller's tracking performance and disturbance rejection capability

Discrete Mechanics and Optimal Control Applied to the Compass Gait Biped

This paper presents a methodology for generating locally optimal control policies for simple hybrid mechanical systems, and illustrates the method on the compass gait biped. Principles from discrete mechanics are utilized to generate optimal control policies as solutions of constrained nonlinear optimization problems. In the context of bipedal walking, this procedure provides a comparative measure of the suboptimality of existing control policies. Furthermore, our methodology can be used as a control design tool; to demonstrate this, we minimize the specific cost of transport of periodic orbits for the compass gait biped, both in the fully actuated and underactuated case

Optimal elastic coupling in form of one mechanical spring to improve energy efficiency of walking bipedal robots

This paper presents a method to optimize the energy efficiency of walking bipedal robots by more than 80% in a speed range from 0.3 to 2.3 m/s using elastic couplings – mechanical springs with movement speed independent parameters. The considered planar robot consists of a trunk, two two-segmented legs, two actuators in the hip joints, two actuators in the knee joints and an elastic coupling between the shanks. It is modeled as underactuated system to make use of its natural dynamics and feedback controlled via input-output linearization. A numerical optimization of the joint angle trajectories as well as the elastic couplings is performed to minimize the average energy expenditure over the whole speed range. The elastic couplings increase the swing leg motion’s natural frequency thus making smaller steps more efficient which reduce the impact loss at the touchdown of the swing leg. The process of energy turnover is investigated in detail for the robot with and without elastic coupling between the shanks. Furthermore, the influences of the elastic couplings’ topology and of joint friction are analyzed. It is shown that the optimization of the robot’s motion and elastic coupling towards energy efficiency leads to a slightly slower convergence rate of the controller, yet no loss of stability but a lower sensitivity with respect to disturbances. The optimal elastic coupling discovered via numerical optimization is a linear torsion spring with transmissions between the shanks. A design proposal for this elastic coupling – which does not affect the robot’s trunk and parallel shank motion and can be used to enhance an existing robot – is given for planar as well as spatial robots

Optimization of energy efficiency of walking bipedal robots by use of elastic couplings in the form of mechanical springs

This paper presents a method to optimize the en- ergy efficiency of walking bipedal robots by more than 50 % in a speed range from 0.3 to 2.3 m/s using elastic couplings – mechanical springs with movement speed independent pa- rameters. The considered robot consists of a trunk, two stiff legs and two actuators in the hip joints. It is modeled as un- deractuated system to make use of its natural dynamics and feedback controlled with input-output linearization. A nu- merical optimization of the joint angle trajectories as well as the elastic couplings is performed to minimize the average energy expenditure over the whole speed range. The elastic couplings increase the swing leg motion’s natural frequency thus making smaller steps more efficient which reduce the impact loss at the touchdown of the swing leg. The pro- cess of energy turnover is investigated for the robot with and without elastic couplings. Furthermore, the influence of the elastic couplings’ topology, its degree of nonlinearity, the mass distribution, the joint friction, the coefficient of static friction and the selected actuator is analyzed. It is shown that the optimization of the robot’s motion and elastic coupling towards energy efficiency leads to a slightly slower conver- gence rate of the controller, yet no loss of stability and a lower sensitivity with respect to disturbances. The optimal elastic coupling discovered by the numerical optimization is a linear torsion spring between the legs

Energy Conservative Limit Cycle Oscillations

This paper shows how globally attractive limit cycle oscillations can be induced in a system with a nonlinear feedback element. Based on the same principle as the Van der Pol oscillator, the feedback behaves as a negative damping for low velocities but as an ordinary damper for high velocities. This nonlinear damper can be physically implemented with a continuous variable transmission and a spring, storing energy in the spring when the damping is positive and reusing it when the damping is negative. The resulting mechanism has a natural limit cycle oscillation that is energy conservative and can be used for the development of robust, dynamic walking robots

Walking Stabilization Using Step Timing and Location Adjustment on the Humanoid Robot, Atlas

While humans are highly capable of recovering from external disturbances and uncertainties that result in large tracking errors, humanoid robots have yet to reliably mimic this level of robustness. Essential to this is the ability to combine traditional "ankle strategy" balancing with step timing and location adjustment techniques. In doing so, the robot is able to step quickly to the necessary location to continue walking. In this work, we present both a new swing speed up algorithm to adjust the step timing, allowing the robot to set the foot down more quickly to recover from errors in the direction of the current capture point dynamics, and a new algorithm to adjust the desired footstep, expanding the base of support to utilize the center of pressure (CoP)-based ankle strategy for balance. We then utilize the desired centroidal moment pivot (CMP) to calculate the momentum rate of change for our inverse-dynamics based whole-body controller. We present simulation and experimental results using this work, and discuss performance limitations and potential improvements