van Heusden, R.

Visser, A.

English

This research will focus on is the question of how to train a humanoid robot to stop a penalty using Q Learning combined with Transfer Learning on a neural network. The Transfer Learning technique proved allow the robot to learn stop the robot without domain knowledge in the form of an intermediate reward.</p

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Making a robot stop a penaltyvan Heusden, R.; Visser, A.Publication date2018Document VersionFinal published versionPublished in30th Benelux Conference on Artificial IntelligenceLink to publicationCitation for published version (APA):van Heusden, R., & Visser, A. (2018). Making a robot stop a penalty. In M. Atzmueller, & W.Duivesteijn (Eds.), 30th Benelux Conference on Artificial Intelligence: BNAIC 2018Preproceedings : November 8-9, 2018, Jheronimus Academy of Data Science (JADS), 's-Hertogenbosch, The Netherlands (pp. 89-90). (BNAIC; Vol. 30). Jheronimus Academy of DataScience.General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 202330th Benelux Conferenceon Artificial IntelligenceBNAIC 2018 PreproceedingsNovember 8-9, 2018Jheronimus Academy of Data Science (JADS),’s-Hertogenbosch, The NetherlandsEditors:Martin AtzmuellerCSAI/JADS, Tilburg UniversityWouter DuivesteijnEindhoven University of TechnologyMaking a Robot Stop a PenaltyRuben van Heusden Supervisor : dhr. dr. A. VisserIntelligent Robotics Lab, Faculty of Science, University of AmsterdamAbstract. This research will focus on is the question of how to traina humanoid robot to stop a penalty using Q Learning combined withTransfer Learning on a neural network. The Transfer Learning techniqueproved allow the robot to learn stop the robot without domain knowledgein the form of an intermediate reward.Keywords: Reinforcement Learning · Transfer Learning · RoboCupContext This is a short extract of my thesis [4], which addresses the problemof decision making for an autonomous goalkeeping robot in a penalty shootout.A solution would contribute to the goal of having robot soccer players outper-form human soccer players by 2050, which is still an open challenge in artificialintelligence [1]. The problem is complex as the amount of information providedto the goalkeeper is limited and the goalkeeper only has a limited time framewithin it has to make a decision. Previous approaches often involved a humancoding complex behavioral rules for the robots, which is very inflexible and theprocess of constructing behavioral rules and discovering the right parameterscan be very time consuming [3].Method To allow a simulated robot to efficiently learn an optimal strategy byitself, a sensible reward function should be used, which requires domain knowl-edge. Instead, the domain knowledge is used to present the robot a succession oftasks, increasing in difficulty and using the experience gained in previous tasksto guide the robot in the following tasks [5]. In this research, 4 tasks increasingin difficulty (possible shooting angles for the penalty kicker) were used as shownin Figure 1, where the agent progressed to the next task when it was able to stop30 penalties in a row. By using this approach, the agent would not necessarilyneed an intermediate reward, as it might be able to learn the basic concepts ofthe game in a simple environment and build on that knowledge to have successin more difficult tasks.Results The approach presented in this thesis uses Q learning combined with adeep neural network [2] applied to discrete state and actions spaces and transferlearning to learn a behavioral policy. With intermediate rewards the algorithmis able to successfully stop a penalty in ∼96% of the trials, with an averagetraining length of 813 trials. Without intermediate rewards and using transferlearning, the agent was able to stop a penalty with an accuracy of ∼65% afteran average of 7476 trials.892 R.J. van HeusdenFig. 1. Increasing difficulty in the tasks; allowing a wider shooting range to the penaltykicker.Conclusion In the experiments, two different agent were trained, one that usedtransfer learning, and one that only uses the neural network function approxi-mator. The results showed that the agent that was trained using intermediaterewards was able to reliably catch the ball with the penalty kicker at full ca-pacity. The intermediate reward was based on the distance of the goal keeper tothe expected trajectory of the ball, which is a feature containing a lot of domainknowledge. In the case of the agent that was trained using transfer learning, theagent was able to progress through all the training stages, and learn to implic-itly estimate the most likely trajectory from the position of the penalty kickerbehind the ball. The implicit estimate was not perfect: the goalkeeper developeda ’weak’ side. Video’s of the learned behavior are publicly available1. For futurework, experiments can be conducted on the size and shape of the neural net-work used, as this seemed to have quite a big influence on the performance ofthe algorithm.References1. Gerndt, R., Seifert, D., Baltes, J.H., Sadeghnejad, S., Behnke, S.: Humanoid robotsin soccer: Robots versus humans in robocup 2050. IEEE Robot. Automat. Mag.22(3), 147–154 (2015)2. Hausknecht, M., Stone, P.: Deep reinforcement learning in parameterized actionspace. arXiv preprint arXiv:1511.04143 (2015)3. Lagrand, C.G.: Learning a robot to score a penalty: Minimal reward reinforcementlearning. Bachelor’s thesis, Universiteit van Amsterdam (July 2017)4. van Heusden, R.J.: Making a robot stop a penalty using q learning and transferlearning. Bachelor’s thesis, Universiteit van Amsterdam (June 2018)5. Watkinson, W., Camp, T.: Training a robocup striker agent via transferred rein-forcement learning. In: RoboCup 2018: Robot Soccer World Cup XXII. LectureNotes in Artificial Intelligence, Springer (2019)1 https://www.youtube.com/watch?v=MlHNmP- R3g90

Making a robot stop a penalty

https://pure.uva.nl/ws/files/48957001/Making_a_robot_stop_a_penalty.pdf

Making a robot stop a penalty

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications