112 research outputs found
Emergence of Human-comparable Balancing Behaviors by Deep Reinforcement Learning
This paper presents a hierarchical framework based on deep reinforcement
learning that learns a diversity of policies for humanoid balance control.
Conventional zero moment point based controllers perform limited actions during
under-actuation, whereas the proposed framework can perform human-like
balancing behaviors such as active push-off of ankles. The learning is done
through the design of an explainable reward based on physical constraints. The
simulated results are presented and analyzed. The successful emergence of
human-like behaviors through deep reinforcement learning proves the feasibility
of using an AI-based approach for learning humanoid balancing control in a
unified framework
Momentum Control with Hierarchical Inverse Dynamics on a Torque-Controlled Humanoid
Hierarchical inverse dynamics based on cascades of quadratic programs have
been proposed for the control of legged robots. They have important benefits
but to the best of our knowledge have never been implemented on a torque
controlled humanoid where model inaccuracies, sensor noise and real-time
computation requirements can be problematic. Using a reformulation of existing
algorithms, we propose a simplification of the problem that allows to achieve
real-time control. Momentum-based control is integrated in the task hierarchy
and a LQR design approach is used to compute the desired associated closed-loop
behavior and improve performance. Extensive experiments on various balancing
and tracking tasks show very robust performance in the face of unknown
disturbances, even when the humanoid is standing on one foot. Our results
demonstrate that hierarchical inverse dynamics together with momentum control
can be efficiently used for feedback control under real robot conditions.Comment: 21 pages, 11 figures, 4 tables in Autonomous Robots (2015
Humanoid Balancing Behavior Featured by Underactuated Foot Motion
A novel control synthesis is proposed for humanoids to demonstrate unique foot-tilting behaviors that are comparable to humans in balance recovery. Our study of model-based behaviors explains the underlying mechanism and the significance of foot tilting well. Our main algorithms are composed of impedance control at the center of mass, virtual stoppers that prevent overtilting of the feet, and postural control for the torso. The proof of concept focuses on the sagittal scenario and the proposed control is effective to produce human-like balancing behaviors characterized by active foot tilting. The successful replication of this behavior on a real humanoid proves the feasibility of deliberately controlled underactuation. The experimental validation was rigorously performed, and the data from the submodules and the entire control were presented and analyzed
Impact-Aware Multi-Contact Balance Criteria
Intentionally applying impacts while maintaining balance is challenging for
legged robots. This study originated from observing experimental data of the
humanoid robot HRP-4 intentionally hitting a wall with its right arm while
standing on two feet. Strangely, violating the usual zero moment point balance
criteria did not systematically result in a fall. To investigate this
phenomenon, we propose the zero-step capture region for non-coplanar contacts,
defined as the center of mass (CoM) velocity area, and validated it with
push-recovery experiments employing the HRP-4 balancing on two non-coplanar
contacts. To further enable on-purpose impacts, we compute the set of candidate
post-impact CoM velocities accounting for frictional-impact dynamics in three
dimensions, and restrict the entire set within the CoM velocity area to
maintain balance with the sustained contacts during and after impacts. We
illustrate the maximum contact velocity for various HRP-4 stances in
simulation, indicating potential for integration into other task-space
whole-body controllers or planners. This study is the first to address the
challenging problem of applying an intentional impact with a
kinematic-controlled humanoid robot on non-coplanar contacts
Learning dynamic motor skills for terrestrial locomotion
The use of Deep Reinforcement Learning (DRL) has received significantly increased attention
from researchers within the robotics field following the success of AlphaGo, which demonstrated
the superhuman capabilities of deep reinforcement algorithms in terms of solving complex
tasks by beating professional GO players. Since then, an increasing number of researchers
have investigated the potential of using DRL to solve complex high-dimensional robotic tasks,
such as legged locomotion, arm manipulation, and grasping, which are difficult tasks to solve
using conventional optimization approaches.
Understanding and recreating various modes of terrestrial locomotion has been of long-standing interest to roboticists. A large variety of applications, such as rescue missions,
disaster responses and science expeditions, strongly demand mobility and versatility in legged
locomotion to enable task completion. In order to create useful physical robots, it is necessary
to design controllers to synthesize the complex locomotion behaviours observed in humans
and other animals.
In the past, legged locomotion was mainly achieved via analytical engineering approaches.
However, conventional analytical approaches have their limitations, as they require relatively
large amounts of human effort and knowledge. Machine learning approaches, such as DRL,
require less human effort compared to analytical approaches. The project conducted for this
thesis explores the feasibility of using DRL to acquire control policies comparable to, or better
than, those acquired through analytical approaches while requiring less human effort.
In this doctoral thesis, we developed a Multi-Expert Learning Architecture (MELA) that
uses DRL to learn multi-skill control policies capable of synthesizing a diverse set of dynamic
locomotion behaviours for legged robots. We first proposed a novel DRL framework for the
locomotion of humanoid robots. The proposed learning framework is capable of acquiring
robust and dynamic motor skills for humanoids, including balancing, walking, standing-up
fall recovery. We subsequently improved upon the learning framework and design a novel
multi-expert learning architecture that is capable of fusing multiple motor skills together in
a seamless fashion and ultimately deploy this framework on a real quadrupedal robot. The
successful deployment of learned control policies on a real quadrupedal robot demonstrates
the feasibility of using an Artificial Intelligence (AI) based approach for real robot motion control
Fall Prediction and Controlled Fall for Humanoid Robots
Humanoids which resemble humans in their body structure and degrees of freedom are anticipated to work like them within infrastructures and environments constructed for
humans. In such scenarios, even humans who have exceptional manipulation, balancing, and locomotion skills are vulnerable to fall, humanoids being their approximate imitators
are no exception to this. Furthermore, their high center of gravity position in relation to their small support polygon makes them more prone to fall, unlike other robots such
as quadrupeds. The consequences of these falls are so devastating that it can instantly annihilate both the robot and its surroundings. This has become one of the major stumbling
blocks which humanoids have to overcome to operate in real environments. As a result, in this thesis, we have strived to address the imminent fall over of humanoids by developing
different control techniques. The fall over problem as such can be divided into three subissues: fall prediction, controlled fall, and its recovery. In the presented work, the first two
issues have been addressed, and they are presented in three parts.
First, we define what is fall over for humanoids, different sources for it to happen, the effect fall over has both on the robot and to its surroundings, and how to deal with them.
Following which, we give a brief introduction to the overall system which includes both the hardware and software components which have been used throughout the work for varied
purposes.
Second, the first sub-issue is addressed by proposing a generic method to predict the falling over of humanoid robots in a reliable, robust, and agile manner across various
terrains, and also amidst arbitrary disturbances. The aforementioned characteristics are strived to attain by proposing a prediction principle inspired by the human balance sensory
systems. Accordingly, the fusion of multiple sensors such as inertial measurement unit and gyroscope (IMU), foot pressure sensor (FPS), joint encoders, and stereo vision sensor,
which are equivalent to the human\u2019s vestibular, proprioception, and vision systems are considered. We first define a set of feature-based fall indicator variables (FIVs) from the
different sensors, and the thresholds for those FIVs are extracted analytically for four major disturbance scenarios. Further, an online threshold interpolation technique and an
impulse adaptive counter limit are proposed to manage more generic disturbances. For the generalized prediction process, both the instantaneous and cumulative sum of each FIVs
are normalized, and a suitable value is set as the critical limit to predict the fall over.
To determine the best combination and the usefulness of multiple sensors, the prediction performance is evaluated on four different types of terrains, in three unique combinations:
first, each feature individually with their respective FIVs; second, an intuitive performance based (PF); and finally, Kalman filter based (KF) techniques, which involve the usage
of multiple features. For PF and KF techniques, prediction performance evaluations are carried out with and without adding noise. Overall, it is reported that KF performs better
than PF and individual sensor features under different conditions. Also, the method\u2019s ability to predict fall overs during the robot\u2019s simple dynamic motion is also tested and
verified through simulations. Experimental verification of the proposed prediction method on flat and uneven terrains was carried out with the WALK-MAN humanoid robot.
Finally, in reference to the second sub-issue, i.e., the controlled fall, we propose two novel fall control techniques based on energy concepts, which can be applied online to mitigate
the impact forces incurred during the falling over of humanoids. Both the techniques are inspired by the break-fall motions, in particular, Ukemi motion practiced by martial arts
people. The first technique reduces the total energy using a nonlinear control tool, called energy shaping (ES) and further distributes the reduced energy over multiple contacts by
means of energy distribution polygons (EDP). We also include an effective orientation control to safeguard the end-effectors in the event of ground impacts. The performance of
the proposed method is numerically evaluated by dynamic simulations under the sudden falling over scenario of the humanoid robot for both lateral and sagittal falls. The effectiveness of the proposed ES and EDP concepts are verified by diverse comparative simulations regarding total energy, distribution, and impact forces.
Following the first technique, we proposed another controller to generate an online rolling over motion based on the hypothesis that multi-contact motions can reduce the impact
forces even further. To generate efficient rolling motion, critical parameters are defined by the insights drawn from a study on rolling, which are contact positions and attack
angles. In addition, energy-injection velocity is proposed as an auxiliary rolling parameter to ensure sequential multiple contacts in rolling. An online rolling controller is synthesized
to compute the optimal values of the rolling parameters. The first two parameters are to construct a polyhedron, by selecting suitable contacts around the humanoid\u2019s body. This
polyhedron distributes the energy gradually across multiple contacts, thus called energy distribution polyhedron. The last parameter is to inject some additional energy into the
system during the fall, to overcome energy drought and tip over successive contacts. The proposed controller, incorporated with energy injection, minimization, and distribution
techniques result in a rolling like motion and significantly reduces the impact forces, and it is verified in numerical experiments with a segmented planar robot and a full humanoid
model
Motion Planning and Control of Dynamic Humanoid Locomotion
Inspired by human, humanoid robots has the potential to become a general-purpose platform that lives along with human. Due to the technological advances in many field, such as actuation, sensing, control and intelligence, it finally enables humanoid robots to possess human comparable capabilities. However, humanoid locomotion is still a challenging research field. The large number of degree of freedom structure makes the system difficult to coordinate online. The presence of various contact constraints and the hybrid nature of locomotion tasks make the planning a harder problem to solve. Template model anchoring approach has been adopted to bridge the gap between simple model behavior and the whole-body motion of humanoid robot.
Control policies are first developed for simple template models like Linear Inverted Pendulum Model (LIPM) or Spring Loaded Inverted Pendulum(SLIP), the result controlled behaviors are then been mapped to the whole-body motion of humanoid robot through optimization-based task-space control strategies. Whole-body humanoid control framework has been verified on various contact situations such as unknown uneven terrain, multi-contact scenarios and moving platform and shows its generality and versatility. For walking motion, existing Model Predictive Control approach based on LIPM has been extended to enable the robot to walk without any reference foot placement anchoring. It is kind of discrete version of \u201cwalking without thinking\u201d.
As a result, the robot could achieve versatile locomotion modes such as automatic foot placement with single reference velocity command, reactive stepping under large external disturbances, guided walking with small constant external pushing forces, robust walking on unknown uneven terrain, reactive stepping in place when blocked by external barrier. As an extension of this proposed framework, also to increase the push recovery capability of the humanoid robot, two new configurations have been proposed to enable the robot to perform cross-step motions. For more dynamic hopping and running motion, SLIP model has been chosen as the template model. Different from traditional model-based analytical approach, a data-driven approach has been proposed to encode the dynamics of the this model. A deep neural network is trained offline with a large amount of simulation data based on the SLIP model to learn its dynamics.
The trained network is applied online to generate reference foot placements for the humanoid robot. Simulations have been performed to evaluate the effectiveness of the proposed approach in generating bio-inspired and robust running motions. The method proposed based on 2D SLIP model can be generalized to 3D SLIP model and the extension has been briefly mentioned at the end
Compliance control for stabilizing the humanoid on the changing slope based on terrain inclination estimation
This paper presents a stabilization framework integrated with the estimation of the terrain inclination to balance a humanoid on the changing slope as an extension to our previous study. In this paper, the estimation of the terrain inclination is improved for walking in place on an inclination-varying slope. A passivity based admittance control utilizes the force/torque sensing in feet to actively regulate the impedance at the center of mass to stabilize the robot. The logic-based inclination estimation algorithm uses the feet to probe the terrain and deals with the under-actuation. The equilibrium set-point in the admittance control is regulated based on the detected inclination. The effectiveness of the control framework is validated on the humanoid robot COMAN and demonstrated by estimating the terrain inclination, coping with the under-actuation phase, adapting to the slope with changing inclination during both standing and walking. Experimental data are analyzed and discussed, and the future work is suggested
- …