Development of a Locomotion and Balancing Strategy for Humanoid Robots

The locomotion ability and high mobility are the most distinguished features of humanoid robots. Due to the non-linear dynamics of walking, developing and controlling the locomotion of humanoid robots is a challenging task. In this thesis, we study and develop a walking engine for the humanoid robot, NAO, which is the official robotic platform used in the RoboCup Spl. Aldebaran Robotics, the manufacturing company of NAO provides a walking module that has disadvantages, such as being a black box that does not provide control of the gait as well as the robot walk with a bent knee. The latter disadvantage, makes the gait unnatural, energy inefficient and exert large amounts of torque to the knee joint. Thus creating a walking engine that produces a quality and natural gait is essential for humanoid robots in general and is a factor for succeeding in RoboCup competition. Humanoids robots are required to walk fast to be practical for various life tasks. However, its complex structure makes it prone to falling during fast locomotion. On the same hand, the robots are expected to work in constantly changing environments alongside humans and robots, which increase the chance of collisions. Several human-inspired recovery strategies have been studied and adopted to humanoid robots in order to face unexpected and avoidable perturbations. These strategies include hip, ankle, and stepping, however, the use of the arms as a recovery strategy did not enjoy as much attention. The arms can be employed in different motions for fall prevention. The arm rotation strategy can be employed to control the angular momentum of the body and help to regain balance. In this master\u27s thesis, I developed a detailed study of different ways in which the arms can be used to enhance the balance recovery of the NAO humanoid robot while stationary and during locomotion. I model the robot as a linear inverted pendulum plus a flywheel to account for the angular momentum change at the CoM. I considered the role of the arms in changing the body\u27s moment of inertia which help to prevent the robot from falling or to decrease the falling impact. I propose a control algorithm that integrates the arm rotation strategy with the on-board sensors of the NAO. Additionally, I present a simple method to control the amount of recovery from rotating the arms. I also discuss the limitation of the strategy and how it can have a negative impact if it was misused. I present simulations to evaluate the approach in keeping the robot stable against various disturbance sources. The results show the success of the approach in keeping the NAO stable against various perturbations. Finally,I adopt the arm rotation to stabilize the ball kick, which is a common reason for falling in the soccer humanoid RoboCup competitions

Project Pele: Humanoid Robotic Programming A Study in Artificial Intelligence

In the ever changing world of technology, the humanoid robot has been a constant member of science fiction culture. Our project goal was to develop a humanoid robot capable of independently displaying effective soccer skills. We divided the tasks into two teams; one designed a ball kicking robot program while the other designed a path tracking robot program. After each group completed their four major objectives, we had created a superior program than its predecessors. Using our optimized code as a foundation, another group can further develop these robot programs to demonstrate even more humanlike soccer skills

Project Pele: Humanoid Robotic Programming -A Study in Artificial Intelligence

In the ever changing world of technology, the humanoid robot has been a constant member of science fiction culture. Our project goal was to develop a humanoid robot capable of independently displaying effective soccer skills. We divided the tasks into two teams; one designed a ball kicking robot program while the other designed a path tracking robot program. After each group completed their four major objectives, we had created a superior program than its predecessors. Using our optimized code as a foundation, another group can further develop these robot programs to demonstrate even more humanlike soccer skills

Project Pele: Humanoid Robotic Programming - A Study in Artificial Intelligence

In the ever changing world of technology, the humanoid robot has been a constant member of science fiction culture. Our project goal was to develop a humanoid robot capable of independently displaying effective soccer skills. We divided the tasks into two teams; one designed a ball kicking robot program while the other designed a path tracking robot program. After each group completed their four major objectives, we had created a superior program than its predecessors. Using our optimized code as a foundation, another group can further develop these robot programs to demonstrate even more humanlike soccer skills

Humanoid Robot Soccer Locomotion and Kick Dynamics: Open Loop Walking, Kicking and Morphing into Special Motions on the Nao Robot

Striker speed and accuracy in the RoboCup (SPL) international robot soccer league is becoming increasingly important as the level of play rises. Competition around the ball is now decided in a matter of seconds. Therefore, eliminating any wasted actions or motions is crucial when attempting to kick the ball. It is common to see a discontinuity between walking and kicking where a robot will return to an initial pose in preparation for the kick action. In this thesis we explore the removal of this behaviour by developing a transition gait that morphs the walk directly into the kick back swing pose. The solution presented here is targeted towards the use of the Aldebaran walk for the Nao robot. The solution we develop involves the design of a central pattern generator to allow for controlled steps with realtime accuracy, and a phase locked loop method to synchronise with the Aldebaran walk so that precise step length control can be activated when required. An open loop trajectory mapping approach is taken to the walk that is stabilized statically through the use of a phase varying joint holding torque technique. We also examine the basic princples of open loop walking, focussing on the commonly overlooked frontal plane motion. The act of kicking itself is explored both analytically and empirically, and solutions are provided that are versatile and powerful. Included as an appendix, the broader matter of striker behaviour (process of goal scoring) is reviewed and we present a velocity control algorithm that is very accurate and efficient in terms of speed of execution

Individual and coordinated decision for the CAMBADA team

Mestrado em Engenharia de Computadores e TelemáticaA coordenação em sistemas multi-robô é um aspecto crucial no futebol robótico. A maneira como cada equipa coordena cada um dos seus robôs em acções cooperativas define a base da sua estratégia. Este trabalho tem como foco o desenvolvimento da coordenação e estratégia da equipa CAMBADA. CAMBADA é a equipa de futebol robótico da modalidade RoboCup Middle Size League da Universidade de Aveiro. Foi desenvolvida pelo grupo ATRI, pertencente µa unidade de investigação IEETA. O presente trabalho baseia-se em trabalho desenvolvido anteriormente, tentando melhorar o desempenho da equipa. Cada robô da equipa CAMBADA é um agente independente e autónomo capaz de coordenar as suas acções com os colegas de equipa através da comunicação e da partilha de informação. O comportamento de cada robô deverá ser integrado na estratégia global da equipa, resultando assim em acções cooperativas de todos os robôs. Isto é conseguido através do uso de papeis(roles) e comportamentos(behaviours) que definem a atitude de cada robô e as acções que daí resultam. Novos papeis foram desenvolvidos para complementar a estratégia de equipa, e alguns dos papeis existentes foram melhorados. Também foram efectuadas melhorias em alguns dos comportamentos existentes. É efectu- ada a descrição de cada um destes papeis e comportamentos, assim como as alterações efectuadas. O trabalho desenvolvido foi testado nas competições do Robótica 2008 (o desenvolvimento não estava ainda concluído) e por fim nas competições do RoboCup'2008. A participação da equipa no RoboCup'2008 é analisada e discutida. A equipa consagrou-se campeã mundial, vencendo a competição da Middle Size League do RoboCup'2008 em Suzhou, China. ABSTRACT: Multi-robot coordination is one crucial aspect in robotic soccer. The way each team coordinates its individual robots into cooperative global actions define the foundation of its strategy. CAMBADA is the RoboCup Middle Size League robotic soccer team of the University of Aveiro. It was created by the ATRI group, part of the IEETA research unit. This work is focused on coordination and strategy development for the CAMBADA team. It is built upon previous work and tries to improve the team performance further. In CAMBADA each robot is an independent agent, it coordinates its actions with its teammates through communication and information exchange. The resulting behaviour of the individual robot should be integrated into the global team strategy, thus resulting in cooperative actions by all the robots. This is done by the use of roles and behaviours that define each robot attitude in the field and resulting individual actions. In this work, new roles were created to add to the team strategy and some of the previous existing roles were improved. Some of the existing behaviours were also improved to better fit the desired goals. Each role and behaviour is described as well as the changes made. The resulting work was put to test in the portuguese Robotica 2008 competition (while still in progress) and finally in the RoboCup'2008 world competitions. The performance of the team in the latter is analysed and discussed. The team achieved the 1st place in the RoboCup'2008 MSL world competitions

Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning

We investigate whether Deep Reinforcement Learning (Deep RL) is able to synthesize sophisticated and safe movement skills for a low-cost, miniature humanoid robot that can be composed into complex behavioral strategies in dynamic environments. We used Deep RL to train a humanoid robot with 20 actuated joints to play a simplified one-versus-one (1v1) soccer game. We first trained individual skills in isolation and then composed those skills end-to-end in a self-play setting. The resulting policy exhibits robust and dynamic movement skills such as rapid fall recovery, walking, turning, kicking and more; and transitions between them in a smooth, stable, and efficient manner - well beyond what is intuitively expected from the robot. The agents also developed a basic strategic understanding of the game, and learned, for instance, to anticipate ball movements and to block opponent shots. The full range of behaviors emerged from a small set of simple rewards. Our agents were trained in simulation and transferred to real robots zero-shot. We found that a combination of sufficiently high-frequency control, targeted dynamics randomization, and perturbations during training in simulation enabled good-quality transfer, despite significant unmodeled effects and variations across robot instances. Although the robots are inherently fragile, minor hardware modifications together with basic regularization of the behavior during training led the robots to learn safe and effective movements while still performing in a dynamic and agile way. Indeed, even though the agents were optimized for scoring, in experiments they walked 156% faster, took 63% less time to get up, and kicked 24% faster than a scripted baseline, while efficiently combining the skills to achieve the longer term objectives. Examples of the emergent behaviors and full 1v1 matches are available on the supplementary website.Comment: Project website: https://sites.google.com/view/op3-socce

FC Portugal - High-Level Skills Within A Multi-Agent Environment

Ao longo dos anos a RoboCup, uma competição internacional de robótica e da inteligência artificia, foi palco de muitos desenvolvimentos e melhorias nestes duas áreas científicas. Esta competição tem diferentes desafios, incluindo uma liga de simulação 3D (Simulation 3D League). Anualmente, ocorre um torneio de jogos de futebol simulados entre as várias equipas participantes na Simulation 3D League, todas estas equipas deveram ser compostas por 11 robôs humanoides. Esta simulação obedece às leis da física de modo a se aproximar das circunstâncias dos jogos reais. Além disso, as regras da competição são semelhantes às regras originais do futebol com algumas alterações e adaptações. A equipa portuguesa, o FC Portugal 3D é um participante assíduo nos torneios desta liga e chegou até a ser vitoriosa várias vezes nos últimos anos, no entanto, para participar nesta competição é necessário que as equipas tenham os seus agentes capazes de executar skills (ou habilidades) de baixo nível como andar, chutar e levantar-se. O bom registo da equipa FC Portugal 3D advém do facto de os métodos utilizados para treinar os seus jogadores serem continuamente melhorados resultando em melhores habilidades. De facto, considera-se que estes comportamentos de baixo nível estão num ponto em que é possível mudar o foco das implementações para competências de alto nível que deveram ser baseadas nestas competências fundamentais de baixo nível. O futebol pode ser visto como um jogo cooperativo onde jogadores da mesma equipa têm de trabalhar em conjunto para vencer os seus adversários, consequentemente, este jogo é considerado como um bom ambiente para desenvolver, testar e aplicar implementações relativas a cooperações multi-agente. Com isto em mente, o objetivo desta dissertação é construir uma setplay multi-agente baseada nas skills de baixo nível previamente implementadas pela FC Portugal para serem usadas em situações de jogo específicas em que a intenção principal é marcar um golo. Recentemente, muitos participantes da 3D League (incluindo a equipa portuguesa) têm desenvolvido competências utilizando métodos de Deep Reinforcement Learning obtendo resultados satisfatórios num tempo razoável. A abordagem adotada neste projeto foi a de utilizar o algoritmo de Reinforcement Learning, PPO, para treinar todos os ambientes criados com o intuito de desenvolver a setplay pretendida, os resultados dos treinos estão presentes no penúltimo capítulo deste documento seguidos de sugestões para implementações futuras.Throughout the years the RoboCup, an international competition of robotics and artificial intelligence, saw many developments and improvements in these scientific fields. This competition has different types of challenges including a 3D Simulation League that has an annual tournament of simulated soccer games played between several teams each composed of 11 simulated humanoid robots. The simulation obeys the laws of physics in order to approximate the games as much as possible to real circumstances, in addition, the rules are similar to the original soccer rules with a few alterations and adaptations. The Portuguese team, FC Portugal 3D has been an assiduous participant in this league tournaments and was even victorious several times in the past years, nonetheless, to participate in this competition is necessary for teams to have their agents able to execute low-level skills such as walk, kick and get up. The good record of the FC Portugal 3D team comes from the fact that the methods used to train the robots keep being improved, resulting in better skills. As a manner of fact, it is considered that these low-level behaviors are at a point that is possible to shift the implementations' focus to high-level skills based on these fundamental low-level skills. Soccer can be seen as a cooperative game where players from the same team have to work together to beat their opponents, consequently, this game is considered to be a good environment to develop, test, and apply cooperative multi-agent implementations. With this in mind, the objective of this dissertation is to construct a multi-agent setplay based on FC Portugal's low-level skills to be used in certain game situations where the main intent is to score a goal. Recently, many 3D League participants (including the Portuguese team) have been developing skills using Deep Learning methods and obtaining successful results in a reasonable time. The approach taken on this project was to use the Reinforcement Learning algorithm PPO to train all the environments that were created to develop the intended setplay, the results of the training are present in the second-to-last chapter of this document followed by suggestions for future implementations