Gerbscheid, J.

Groot, T.

Le Roy, N.

Terwijn, B.

Testroote, M.

Visser, A.

English

This team description paper describes the approach that will be taken by the UvA@Home team to compete in Social Standard Platform League. The research challenges concern person recognition, object recognition, natural language processing and navigation. Modules implemented so far include people detection, face recognition, speech recognition and natural language processing. The remaining challenges will be solved using the previous research and achievements of the UvA teams in the RoboCup

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)UvA@Home Team Description paper 2018Gerbscheid, J.; Groot, T.; Testroote, M.; Le Roy, N.; Terwijn, B.; Visser, A.Publication date2017Document VersionSubmitted manuscriptLink to publicationCitation for published version (APA):Gerbscheid, J., Groot, T., Testroote, M., Le Roy, N., Terwijn, B., & Visser, A. (2017).UvA@Home Team Description paper 2018. Intelligent Robotics Lab, University ofAmsterdam.General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 2023UvA@Home Team Description Paper 2018Jonathan Gerbscheid, Thomas Groot, Merijn Testroote, Noah Le Roy,Bas Terwijn, and Arnoud VisserIntelligent Robotics LabFaculty of ScienceUniversity of AmsterdamThe Netherlandshttp://www.uvahome.nl/Abstract. This team description paper describes the approach that willbe taken by the UvA@Home team to compete in Social Standard Plat-form League. The research challenges concern person recognition, objectrecognition, natural language processing and navigation. Modules imple-mented so far include people detection, face recognition, speech recogni-tion and natural language processing. The remaining challenges will besolved using the previous research and achievements of the UvA teamsin the RoboCup.1 IntroductionThe UvA@Home team consists currently of four bachelor Artificial Intelligencestudents, assisted by two senior university staff members. Yet, the most impor-tant team member is our Pepper robot, called Ginger. The team is part of theIntelligent Robotics Lab1 (IRL) of the University of Amsterdam. The IRL func-tions as the governing body for all university robotics teams, which include theDutch NAO Team (SPL) and the UvA@Home team (SSPL).The focus of the university and the IRL lies in perception, world modeling,machine learning and decision making. Previous efforts done by other teamsat the university also include efforts done for the simulation of the @Homecompetition [1, 2].Last year the UvA@Home team competed in the RoboCup in Nagoya [3] andeven though the final ranking was not very high, the team was satisfied with thedemonstration of the conversational mode [4]. A lack of working localizationand mapping modules were the primary reasons for not being able to continueto the final round. In preparation for RoboCup Montréal the team is planning toparticipate in the @Home competition at the German Open from April 25 to 29in Magdenburg. Here the team wants to test the improved interaction modulesand integration with a working localization and mapping component. The timeafter the German Open will be used to further train and test the neural networksfor object recognition relevant to the challenges.1 http://www.intelligentroboticslab.nl/2 Jonathan Gerbscheid, Thomas Groot, Merijn Testroote, Noah Le Roy et al.2 Current ModulesFor the previous RoboCup in Nagoya the UvA@Home team built a rudimentaryPython framework on top of NAOqi. For the 2018 RoboCup in Montréal, theteam decided to stick with the Python language due to the teams familiarity withit and due to the language being accessible to new team members. Additionally,the choice was made to rework the original framework using the qi frameworkinstead of the NAOqi API. The qi framework is able use the Peppers processingpower more efficiently than the NAOqi API for the reasons stated in the qidocumentation2.2.1 Natural Language ProcessingA rule based Natural Language Processing module was developed for 2017 Cock-tail Party challenge [5] where the robot needed to be able to extract names anddrinks embedded in commands. The extracting of drinks and names was doneby parsing the lowest laying noun and recursively searching the tree of hyper-nyms for a specific word definition [6]. The hypernym tree is constructed usingWordNet which is an English lexical database. In figure 1 the tree of hypernymsfor the word water is illustrated. Because the hypernyms Liquid and Fluid oc-cur in the tree it is likely that the word water is a drink. However, if BinaryCompound and Substance were the only hypernyms in the tree the word wouldnot be classified as a drink. A similar approach is used for extracting names.This module made for the Cocktail Party challenge can be extended byadding different definitions that the system can look for. Adding rules uponrules is not a final solution for robots to work in an @Home environment butwhen challenges have specified objects and commands, it will be easy to add thenew definitions to the database.WaterBinary Compound LiquidFluidSubstanceFig. 1. Tree of hypernyms for the word: Water2.2 Face Detection & RecognitionFor the 2017 RoboCup, both the OpenCV Local Binary Patterns Histogram(LBPH) [7] and the OpenFace classifier [8] were used for face recognition. How-2 http://doc.aldebaran.com/libqi/guide/py-tonaoqi2.html#qi-framework-main-featuresUvA@Home Team Description Paper 2018 3ever, neither were used during the competition itself as it was not needed forthe challenges that the team participated in. Face detection, which was usedfor the speech and person recognition challenge, was done using the OpenCVHaar Feature-based Cascade Classifier. The LBPH method is still used in thecurrent implementation due to its ease of use and low computational cost. Aserver-based implementation of the OpenFace Classifier can also be used butonly if the LBPH method underperforms in the challenge environments.2.3 Speech RecognitionThe UvA@Home team currently relies on the Google Speech API for speechprocessing as it performed well in the competition, but will also implement abackup using NAOqi’s built-in speech recognition in case of network connectionproblems. The Google Speech API works by converting an audio file containingspeech to text. The main task here being able to know when to start and stoprecording audio so that the Google Speech API can accurately determine thecontent. This is done listening to peaks in sound energy levels in the environment.Recording will start when registering a peak in sound energy levels and will stopwhen sound energy levels settle again. The sound energy levels are calculatedusing the built-in front microphone energy function in the NAOqi API 3. A peakin sound energy levels is determined to be relevant when the difference of soundenergy level passes some threshold. This threshold is increased by measuring theamount of times it is passed without any speech being properly detected and isdecreased when not being passed for a predetermined amount of time.3 Future ModulesSeveral modules are currently not implemented in the new framework but areplanned to be implemented in the future. The team hopes to complete thesemodules for the 2018 competition.3.1 Image RecognitionA full body people detection method was developed for the 2017 competition [9],however, integration with the full system was not completed for the competitionitself, as the team decided to focus on other modules that were more importantfor the challenges. The detection works by combining data from Peppers 3D sen-sor with a CNN made in Tensorflow [10]. A color image dataset was made to testand train this method on. This dataset will be expanded with images of com-mon objects from the @Home competition in order to train a general detectionnetwork, this network will be integrated into the new framework. This methodrelies on offloading the computations of the network to an external computer3 http://doc.aldebaran.com/2-1/naoqi/audio/alaudiodevice-api.html#ALAudioDeviceProxy::getFrontMicEnergy4 Jonathan Gerbscheid, Thomas Groot, Merijn Testroote, Noah Le Roy et al.with a dedicated GPU, making the system dependant on internet connectivityand thus more likely to have difficulties in the competition. Detection on therobot itself, however, is very limited due to its relatively low processing power.It was therefore decided that offloading detection was the most ideal solution.3.2 Navigation & LocalizationNavigation & localization are currently done using NAOqi’s built-in navigationmodule. The performance of this module does not seem to be at the desiredlevel required for the competition therefore the team will conduct additionaltests on the navigation module in a variety of indoor environments to assesits effectiveness. If it is found to consistently underperform, the team will haveto consider switching to the ROS framework in order to use the ROS SLAMgmapping implementation.4 ContributionThe team plans to contribute to the competition by making all code public anddocumented4. Modules and classes in general are made to be straight forwardand intuitive as to help other teams integrate the source code. Because the codeis built upon the qi framework which is already present on all Peppers, installingthe code on a new robot is relatively easy compared to ROS based frameworks,allowing inexperienced teams to quickly have a working basis for competition.The team also believes that because the competition is based around a standardplatform, development for the robot should strive to make it as autonomousas possible. Where this is not possible external computing should only be usedsparingly and the robot should still be able to function without it.A module has also been developed allowing integration with the Slack5 chat-ting program. Updating the robot with the latest code from the GitHub repos-itory and running a specific program can be done through Slack to make inter-action with the robot from a software perspective easier.5 ConclusionThe UvA@Home team has learned a lot from the previous competition and islooking forward to demonstrating the progress made for the 2018 RoboCup. Theteam believes a better understanding of the challenges and the @Home leaguein general will allow it to improve upon previous performances and competein more challenges. Yet, note that this team description paper is written on amoment that the rules for the 2018 competition are not finalized [11]6. If newchallenges are designed for the Social Standard Platform League, then it would4 https://github.com/SpinazieSin/UvA-Home5 https://slack.com/6 Rules Last Build Date: September 5, 2017UvA@Home Team Description Paper 2018 5be preferable if they are announced well in advance to allow the teams to initiatethe new research projects to solve these challenges.In summary, our aim for the 2018 RoboCup is to offload image processingto neural networks on external computing devices and to improve existing mod-ules such as speech recognition, face detection, natural language processing andnavigation & localization.Software ListMain software– Operating System/Robot Control:NAOqi qi API http://doc.aldebaran.com/libqi/guide/py-tonaoqi2.html#qi-framework-main-features.– Face recognition:LBPH (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/),OpenFace (https://cmusatyalab.github.io/openface/),Cascade classifier (https://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html)– Navigation:Naoqi (http://doc.aldebaran.com/2-5/naoqi/motion/alnavigation.html)ROS gmapping (http://wiki.ros.org/gmapping) .– Natural Language Processing:Wordnet (https://wordnet.princeton.edu/),NLTK (http://www.nltk.org/)Used Cloud service:– Speech recognition: Google speech to text API https://pypi.python.org/pypi/SpeechRecognition/– Object & People Detection: Tensorflow https://www.tensorflow.org/Bibliography[1] Sander van Noort and Arnoud Visser. Extending Virtual Robots towardsRoboCup Soccer Simulation and @Home, pages 332–343. Springer BerlinHeidelberg, Berlin, Heidelberg, 2013.[2] Victor I.C. Hofstede. The importance and purpose of simulation in robotics.Bachelor thesis, Universiteit van Amsterdam, June 2015.[3] Jonathan Gerbscheid, Thomas Groot, and A. Visser. Uva@home team de-scription paper 2017. RoboCup Social Standard Platform League, July2017.[4] Jonathan Gerbscheid, Thomas Groot, and Arnoud Visser. Intelligent newsconversation with the pepper robot. In Proceedings of the 29th Belgian-Netherlands Conference on Artificial Intelligence (BNAIC 2017), Gronin-gen, The Netherlands, November 2017.[5] Arnoud Visser. A new robocup@ home challenge. Benelux A.I. Newsletter,31(1):3–6, 2017.[6] Tirza F.E. Soute. Discovering available drinks through natural, robot-led,human-robot interaction between a waiter and a bartender. Bachelor thesis,Universiteit van Amsterdam, July 2017.[7] Timo Ahonen, Abdenour Hadid, and Matti Pietikäinen. Face recognitionwith local binary patterns. In Computer Vision - ECCV 2004: 8th EuropeanConference on Computer Vision, Prague, Czech Republic, May 11-14, 2004.Proceedings, Part I, pages 469–481, Berlin, Heidelberg, 2004. Springer BerlinHeidelberg.[8] Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. Open-face: A general-purpose face recognition library with mobile applications.Technical report, Technical report, CMU-CS-16-118, CMU School of Com-puter Science, 2016.[9] Jonathan R. Gerbscheid. People detection on the pepper robot using convo-lutional neural networks and 3d blob detection. Bachelor thesis, Universiteitvan Amsterdam, July 2017.[10] Mart́ın Abadi et al. TensorFlow: Large-scale machine learning on heteroge-neous systems, 2015. Software available from tensorflow.org.[11] Loy van Beek, Dirk Holz, Mauricio Matamoros, Caleb Rascon, and SvenWachsmuth. Robocup@home 2018: Rules and regulations. http://www.robocupathome.org/rules/2018_rulebook.pdf, 2018.

UvA@Home Team Description paper 2018

https://pure.uva.nl/ws/files/18350635/TDP_Home_2018.pdf

UvA@Home Team Description paper 2018

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications