Robotic soccer is a complex domain where, rather than hand-coding computer programs to control
the players, it is possible to create them through evolutionary methods. This has been successfully
done before by using genetic programming with high-level genes. Such an approach is, however,
limiting. This work attempts to reduce that limit by evolving control programs using genetic
programming with low-level nodes

Walker, Matthew

English

Massey Research Online

Res. Lett. Inf. Math. Sci., (2002) 3, 15–23Available online at http://www.massey.ac.nz/∼wwiims/research/letters/Evolution of a Robotic Soccer PlayerMatthew WalkerI.I.M.S., Massey University Albany Campus, Auckland, New Zealandm.g.walker@massey.ac.nzAbstractRobotic soccer is a complex domain where, rather than hand-coding computer programs to controlthe players, it is possible to create them through evolutionary methods. This has been successfullydone before by using genetic programming with high-level genes. Such an approach is, however,limiting. This work attempts to reduce that limit by evolving control programs using geneticprogramming with low-level nodes.1 IntroductionThere’s nothing like the thrill of watching expensive bits of machinery almost ﬂying around aconﬁned area, no longer under your direct control, but nonetheless your responsibility. Robotsoccer provides such a research platform.The robots move around the ﬁeld with wheels rather than legs, and the ﬁeld is not a standardgrass paddock but rather a scaled-down version about the size of a ping pong table and with awooden surface. Controlled by a master computer with a bird’s eye view of the ﬁeld, for theduration of the game, the robots may not be assisted by humans. Figure 1 shows a simulation ofthe game.“Without human assistance” is what makes this task diﬃcult, for programmers must developalgorithms to play every aspect of soccer. Programming soccer-playing is complex because thereare so many levels. Firstly, the robots need to be told how to chase, kick, and stop the ball. Theythen need to be given jobs such as “be the goalie” or “kick for goal”, and these jobs can change asthe game progresses. Then there are individual strategies such as “take the ball up the ﬁeld andkick for goal”. And ﬁnally, the robots need to act as a team; “support him!” is not easy to code.There have been many successful human-coded robotic soccer teams and each year there are anumber of competitions where these robots demonstrate successful high-level and low-level control.That these teams exist shows that we can divide the problem into parts small enough to solve [3].This does not say that the way we have chosen to divide the problem is the best way; indeed, itmay not even be near optimal as there are many ways of coding a soccer team.Rather than hand-coding the control program, it should be possible for the computer to learnhow to play. How could the computer learn? One answer is to use genetic programming.2 What is Genetic Programming?Selective breeding of farm animals has occurred for thousands of years. Dairy cattle, pigs, beefcattle, sheep, poultry, goats, and horses have all been carefully bred by farmers to produce desirabletraits. Modern day dairy cattle produce more milk, chickens grow faster and horses gallop quickerthan any of their long-dead ancestors.This principle, selective breeding, can also be harnessed to generate computer programs. If aprogram can somehow be bred and also classiﬁed as “better” than another, then a population ofprograms can be evolved. This is genetic programming.Rather than the DNA found in biological life, programs are generally stored as trees. The treestructure of a program to calculate 3(x + 6) is shown in ﬁgure 2.16 R.L.I.M.S. Vol. 3, April, 2002Figure 1: A simulation of a MiroSot three-a-side robotic soccer game.6x3 plustimesFigure 2: A typical, albeit simple, tree-structured program (individual) that returns 3(x + 6).M. Walker, Evolution of a Robotic Soccer Player 17653 plustimes1.  The selection function picks two individuals2.  From each individual a randomly selected subtree is cutdividetimes add2 8 4 1times2 865plus3.  The subtrees are then swapped thus generating two offspring (new individuals)divideadd4 13timesFigure 3: The crossover operation applied to two individuals.18 R.L.I.M.S. Vol. 3, April, 2002Figure 4: The RoboCup SoccerServer simulator.Tree structures make it possible to breed programs. A number of genetic operators are available,the most common are reproduction, mutation and crossover. Reproduction makes an exact copyfrom one generation to the next. Mutation takes an individual’s tree and makes an exact copyexcept that it alters one randomly selected node. Crossover takes two individuals and copies themto the new generation; it then cuts one subtree from each and swaps them around. Figure 3 is anexample of crossover in action.3 Other WorkLuke, et al. [6] worked on creating a program to control a team of robots for the RoboCup simulatorleague [2] (see ﬁgure 4). This league is eﬀectively virtual soccer with eleven players on each side.Although it is never played with real hardware, it is a very challenging domain.The nodes of their trees were primitive soccer-based notions such as: ball, a vector to the ball;opp-closer, a check on whether an opponent is closer to the ball than the robot and dribble, ahuman-coded function to dribble the ball. An example program tree, taken from their paper [6],can be seen in ﬁgure 5.Luke, et al. attempted to evolve an entire team of virtual players but due to the massivecomputation requirements they were forced to make a number of simpliﬁcations. The most im-portant simpliﬁcation was to use the same program for each of the eleven robots. This way, onlyone program needed to be evolved, which would decrease the time required to ﬁnd an acceptablesolution, but specialisation of players, such as strikers and defenders, could not occur.Their paper discuses the history of their evolution: from random movers, to kiddie-soccerplayers, to kiddie-soccer with defenders, and ﬁnally to an acceptable solution.That they were able to evolve an acceptable team using genetic programming was the basis forthis work.M. Walker, Evolution of a Robotic Soccer Player 19if−vopp−closer block−goal if−vmate−closer weight−+ ball7 home away−mateFigure 5: An example program tree from Luke, et al.4 ObjectiveUnlike the virtual robots in the simulator league, this work aims to produce robots for real com-petitions. These real robots also need control strategies which, so far, have been hand-coded.The problem with hand-coding is that it is time consuming and requires a programmer. Theuse of genetic programming allows this task to be given to the computer and it is possible thatevolution may produce a solution that would not have been considered by humans. It is alsopossible that given suﬃcient time, the programs evolved may even be better than those handcoded. The objective of this work is to evolve such control and strategy algorithms.The main problem with evolving control algorithms is that the robots must play literally thou-sands of soccer games. Given that their batteries last not more than a number of hours beforeneeding a time-consuming recharge, and that after almost every game a robot will need to berepaired, and that during a game human interference is often required, thousands of games are notfeasible in the time constraints of a few months.Because the evolutionary process, genetic programming, needs all these thousands of games,a solution is required. The solution is to simulate the environment so that evolution can takeplace in a virtual world. This environment must accurately model the real world because, withthe evolutionary process complete, the ﬁnal result will be exported from this virtual world into therobots.The robots have two wheels that are controlled by setting a value, from -127 to 127, for eachwheel: 127 is full speed forward; -127 is full speed backward and zero is stopped. This structuredictates what the ﬁnal result of the evolved program must be: two integer values to control thewheels.It was decided not to copy the high-level nodes of Luke et al. (such as dribble and opp-closer) asthis would limit the evolutionary process. Low level nodes allow the evolution of motion control aswell as strategy algorithms. High level nodes, on the other hand, do not allow such optimisationof motion control.To give more focus to general robot control it was decided to give only low-level abilities suchas arithmetic (e.g. add and subtract), important game constants (e.g. the ball’s position with ball xand ball y) and randomly selected constants. This means that as well as developing strategy, low-level knowledge, such as how to dribble the ball, will also be optimised by the genetic programmingprocess.20 R.L.I.M.S. Vol. 3, April, 2002Figure 6: First attempt at ball following. Top left: best individual from generation zero (BOG0). Top right: BOG 1. Bottom left: BOG 8; this movement pattern was used almost exclusivelyfrom BOG 8 to BOG 50. Bottom right: BOG 50; diﬀerent initial positions but notice the samemovement pattern as BOG 8.5 Transferring to Real SystemsMoving the evolved code from a simulated environment to the real world has been successfullyexecuted by Lee and Zhang [5] with Khepera robots.Using a commonly available simulator which accurately simulates the Khepera robot light-ﬁnding problem domain, they were able to evolve a program to solve this problem. The geneticprogramming system produced a solution that, in the simulator, very quickly travelled towardsthe light source. They were able to take this program and use it to control a real Khepera robotwhich was equally successful.Of major concern when moving programs from the simulator to the real world is that they willbe optimised for a slightly diﬀerent task. For example, the simulator for the Khepera problemassumes that there is no ambient lighting; in the real world however, removing ambient light isvery impractical. To get around this problem Lee and Zhang used perceptrons to separate thesensors from the program. This way the perceptrons were trained to output expected values evenin diﬀerent environments.Fortunately there are no sensors on the robots used in this study. Instead, all the gameinformation comes from the camera looking down from above. The robots have access to variableswhose values are derived from the camera’s images (i.e. ball position, robot positions and robotorientations) and these variables can be accurately simulated. Thus, Lee and Zhang’s perceptroninterface technique is unnecessary in this domain.6 ResultsBefore evolving a soccer team, a simple ball-following problem was attempted. It was expected thatthis would be a problem that was very quick to solve and that would test the simulator and geneticprogramming kernel. However, the problem proved so diﬃcult that more advanced problems werenot considered.M. Walker, Evolution of a Robotic Soccer Player 21Figure 7: A further attempt at ball following: BOG 50 starting from an arbitrary position andrunning for 2.5 seconds.The ﬁrst scenario given to the kernel was that the robot’s initial position was in the lower leftand the ball was stationary in the centre. For every frame of the simulation (there were 133 secondsper frame for 0.75 seconds) the distance between the robot and the ball was squared and this wasadded to the individual’s raw ﬁtness score. Thus, low raw ﬁtness scores imply the robot movedquickly towards the ball. This problem was allowed 50 generations with a population of 500.After only eight generations a very eﬃcient solution was found (see ﬁgure 6). It travelled almostdirectly towards the ball at high speed. As the robot got closer it would slow down to avoid hittingthe ball.However when this program was applied to an alternative initial starting scenario, it was almostcompletely useless. Rather than follow the ball, as intended, the robot had learned to traveldiagonally upwards from its initial position (see the bottom left image in ﬁgure 6). From thisexperience it is clear that it is important that the learning environment represents the actualproblem.A further attempt at learning to follow the ball was made. Rather than just one startingscenario, eight scenarios were used: the robot was initially placed in the centre with the ball inpositions around the robot. The available terminals were also expanded. This was notably moresuccessful for the general task of ball following, but it was also considerably more time consumingrequiring more than 50 hours of computation. Figure 7 shows the ﬁnal evolved program in action.Although the program is not the most eﬃcient of all ball-followers, it does display some inter-esting behaviour. It is important to realise that following the ball so closely as to actually hit it,makes the problem signiﬁcantly more diﬃcult. As soon as the robot contacts the ball, the ballmoves oﬀ in a diﬀerent direction: the robot now has to follow a moving ball. To counteract thisproblem, the evolved program executes very tight spins as it nears the ball.A more detailed description of the simulator, GP kernel and the problems faced with this work,can be found in Walker [7].7 ConclusionsFrom the experience of evolving a ball following program, problem speciﬁcation is seen as veryimportant. Certainly a solution for the ﬁrst problem (with just one starting scenario) was evolved22 R.L.I.M.S. Vol. 3, April, 2002quickly, but because it was not general, it was not useful. The later attempt, with eight startingscenarios, produced better results.The later attempt was signiﬁcantly assisted by the addition of extra terminals. Although theseterminals provided no more functionality than what was initially available, their addition producedsigniﬁcantly better results. Were the GP kernel equipped with Koza’s ADFs [4], Angeline’s moduleacquisition [1] or some similar feature, the addition of extra terminals may not have been necessary.The tight spinning behaviour observed in the program evolved in the later attempt is the type ofbehaviour that is both unexpected and useful: an element of strategy is already evolving. Withinthe larger scope of a full soccer game there is signiﬁcantly more room to, and more advantagein, developing such unexpected strategies. It is for this reason that genetically evolved programsmay become skilled adversaries: evolutionary search algorithms scan the nooks and crannies of theproblem domain, ﬁnding solutions that humans may never consider.The relative failure of this work compared to that produced by Luke, et al. was disappointing.Successful evolution of a soccer team was anticipated, yet poor quality ball-following was whateventuated.The ﬁrst issue is that of the available genes. Luke, et al. started with high-level genes (suchas dribble and opp-closer) whereas I started with low-level genes (such as genes for arithmetic).The two approaches are signiﬁcantly diﬀerent. Luke, et al. do not allow the genetic process toalter fundamental abilities, such as how to dribble, instead the process focuses on soccer-playingstrategies. With a low-level approach, fundamental abilities are just as likely to be optimised asgeneral strategy. Thus the low-level approach gives more ﬂexibility to the evolutionary process atthe expense of computing resources.The second issue is the available computing resources. Although 50 hours seemed a largeamount of time for such a simple task as ball following, to produce their ﬁnal team, Luke, etal. waited several months’ time. Further to the amount of time, the super-computer they usedoutperformed the desktop I used by about a factor of 50. Eﬀectively they used several years of mycomputer to produce their results. 50 hours pales into insigniﬁcance.To slightly improve this issue, the environment could have been simpliﬁed. Rather than a ballthat moved when the robot came too close, a static goal point might have been an easier initialproblem.A further improvement would have been to optimise the calculations performed by the simula-tor; they were very ineﬃcient since the simulator was graphical.References[1] P. J. Angeline and J. B. Pollack. Evolutionary module acquisition. In D. Fogel and W. Atmar,editors, Proceedings of the Second Annual Conference on Evolutionary Programming, pages154–163, La Jolla, CA, USA, 25-26 1993.[2] Mao Chen, Ehsan Foroughi, Fredrik Heintz, Zhan Xiang Huang, Spiros Kapetanakis, KostasKostiadis, Johan Kummeneje, Itsuki Noda, Oliver Obst, Pat Riley, Timo Steﬀens, Yi Wang,and Xiang Yin. Robocup soccer server: Users manual. This manual (for Soccer Server version7.07) is available from the Web at sserver.sourceforge.net., June 2001.[3] G. Sen Gupta, C. H. Messom, and H. L Sng. State transition based supervisory controlfor a robot soccer system. In M. Renovell, S. Kajihara, I. Al-Bahadly, and S. Demidenko,editors, Proceedings of The First IEEE International Workshop on Electronic Design, Test andApplications 2002, pages 338–342. IEEE, 2002.[4] John R. Koza. Genetic Programming II: Automatic Discovery of Reusable Programs. MITPress, Cambridge Massachusetts, May 1994.M. Walker, Evolution of a Robotic Soccer Player 23[5] K. J. Lee and B. T. Zhang. Learning robot behaviors by evolving genetic programs. Proceedingsof the 26th International Conference on Industrial Electronics, Control and Instrumentation(IECON-2000), pages 2867–2872, 2000.[6] S. Luke, C. Hohn, J. Farris, G. Jackson, and J. Hendler. Co-evolving soccer softbot teamcoordination with genetic programming. In H. Kitano, editor, RoboCup-97: Robot SoccerWorld Cup, pages 398–411. Springer, 1997.[7] Matthew Walker. Development of robot soccer strategies by genetic programming util-ising a simulated environment. Massey Univeristy Honours project report. Available atwww.massey.ac.nz/∼mgwalker/gp., 2000.24 R.L.I.M.S. Vol. 3, April, 2002

Evolution of a robotic soccer player

https://mro.massey.ac.nz/bitstream/10179/4362/1/Evolution_of_a_Robotic_Soccer_Player.pdf

Evolution of a robotic soccer player

Abstract

Similar works

Full text

Available Versions

Massey Research Online