This article discusses a method that performs gesture recognition, with the objective of extracting characteristics of the segmented hand, from dynamic images captured from a webcam and identifying signal patterns. With this method it is possible to manipulate simulated multirobots that perform specific movements. The method consists of the Continuously Adaptive Mean-SHIFT algorithm, followed by the Threshold segmentation algorithm and Deep Learning through Boltzmann restricted machines. As a result, an accuracy of 82.2%

FONSECA FERREIRA, N. M.

MARQUES JUNIOR, F.C.F

SANTOS, D. B. S

Saraiva, Arata A.

SOUSA, JOSE VIGNO M.

Valente, Antonio

Repositorio Institucional de la Universidad Tecnológica de Panamá

Navigation of quadruped multirobots by gesture recognition using restricted boltzmann machines

Santos, D. B. S

Marques Junior, F.C.F

Sousa, Jose Vigno M.

Fonseca Ferreira, N. M.

Portal de Revistas Académicas UTP (Universidad Tecnológica de Panamá)

August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6NAVIGATION OF QUADRUPED MULTIROBOTS BYGESTURE RECOGNITION USING RESTRICTEDBOLTZMANN MACHINESARATA A. SARAIVA1UTAD University,Coimbra, PortugalORCID: 0000-0002-3960-697X 1E-mail: aratasaraiva@gmail.comSANTOS, D. B. S2 and MARQUES JUNIOR, F.C.F 3State University of Piaui,Piripiri, Piaui, Bazil2ORCID: 0000-0003-4018-242X 2E-mail: domingosbruno@prp.uespi.brORCID: 0000-0002-2751-0137 3E-mail: dev.jrmax@prp.uespi.brJOSE VIGNO M SOUSA3State University of Piaui, Piripiri,BrazilUniversity Brazil, Sao Paulo, BrazilORCID: 0000-0002-5164-360X 3E-mail: vigno@prp.uespi.brN. M. FONSECA FERREIRA5Department of Electrical Engineering, Institute of Engineering of Coimbra,Coimbra, Polytechnic InstituteRua Pedro Nunes, 3031-601 Coimbra, PortugalKnowledge Engineering and Decision-Support Research Center (GECAD) of theInstitute of Engineering, Polytechnic Institute of PortoINESC-TEC Technology and Science, Campus da FEUP, Rua Dr. Roberto Frias, 378,4200 - 465 Porto, PortugalORCID: 0000-0002-2204-6339 5E-mail: nunomig@isec.ptANTONIO VALENTE6INESC-TEC Technology and ScienceCampus da FEUP, Rua Dr. Roberto Frias, 378, 4200 - 465 Porto, PortugalSchool of Science and Technology, University of Trás-os-Montes and Alto Douro,Vila Real, PortugalORCID: 0000-0002-5798-1298 6E-mail: avalente@utad.ptThis article discusses a method that performs gesture recognition, with theobjective of extracting characteristics of the segmented hand, from dynamicH. Montes et al. (eds.), CLAWAR 2018: 21st International Conference on Climbing andWalking Robots and the Support Technologies for Mobile Machines, Robotics Transforming the Future© ELSEVIERAugust 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6432images captured from a webcam and identifying signal patterns. With thismethod it is possible to manipulate simulated multirobots that perform specificmovements. The method consists of the Continuously Adaptive Mean-SHIFTalgorithm, followed by the Threshold segmentation algorithm and Deep Learn-ing through Boltzmann restricted machines. As a result, an accuracy of 82.2%.Keywords: Manipulates, multirobots, restricted Boltzmann Machines.1. IntroductionA line of research and development in robotics that has received atten-tion in recent years is the development of bio-inspired walking robots1.2Systems with artificial intelligence are applied in pattern recognition andcan be coupled to manipulable robots, being used in industrial automation,patrols and territorial surveillance,3 have resulted in systems that can op-erate on structured and unstructured media,4 through the use of advancedsensorial feedback mechanisms, making decisions based on learning algo-rithms and reasoning.Based on this has inspired the design of manipulation of 4 legs spidermultirobots Figure 4, quadruped simulated, by means of manual gestures.The Objects of interest that need to be detected and reviewed are coordi-nated as being processed in real time. The method chosen and implementedis the capture and detection of regions of interest in the scene. With a pre-processing for noises reduction in the image acquisition process, the Thresh-old segmentation algorithm was used. And in the last step is performed thereal-time classification by an artificial intelligence, which consists of a deeplearning known as Restricted Boltzmann Machines (RBM).The method covered ensures robust coverage in image recognition,5 un-der certain assumptions that will be clarified throughout the text in roboticsimulation. The developed structure allows the robots to maintain or alterthe formation of the specified trajectories.The document is divided into 6 sections, in which section 2 is character-ized by the model of simulated robots used, section 3 consists of kinematics,follows the formulation of the central algorithm applied in section 4 and thestatistical method to verify the reliability of the used system . The resultsafter application of the proposal are presented in section 5 and the conclu-sion in section 6.2. Quadruped RobotLocomotion of legs on natural terrain presents complex problems suchas foot placement, avoiding obstacles, load distribution, general stability6.7August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6433One way to deal with these problems is to use models that describe mathe-matically the different situations.8 The model adopted has 4 identical legs,as can be seen in the Figure1.Figure 1. Quadruped Model3. KinematicKinematics is the branch of mechanics that studies the movement ofa body or a system of bodies without considering its mass or the forcesacting on it.9Kinematics has two types, direct and inverse kinematics. Inthe direct kinematics it is desired to obtain the position and orientation ofthe actuator in the Cartesian space of the base, from the positions of thejoints10 .The inverse kinematics consists of the process of determining the anglesin terms of the coordinates for the desired position of the leg in the Cartesiansystem.9The procedure for obtaining the equations is strongly dependent on theconfiguration of the robot.11 The kinematics chosen by the present work isthe inverse, in this case, is obtained through geometric definitions based onthe shape of the leg Figure2, the equations obtained are 1, 2, 3.Figure 2. Kinematic model of the legs.August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6434θ1 = arccos(y1x1)(1)θ2 = arccos(L22 + x23 + y23 − L232.L2.√x32 + y23)+ arctan(y3y3)(2)θ3 = pi − arccos(L22 + L23 −(x23 + y23)2.L2.L3)(3)Each leg of the Quadruped Robots is composed of a chain of three linksof freedom Figure 2.The assignment model of the link frame of the links isbased on Denavit-Hartenberg direct geometric modeling algorithm.The Figure 2 illustrates the leg used for the Robots, in which it hasthree joints R1, R2 and R3 respectively. R1 is the point at which the leg isattached to the body of the robot, L1 represents the "thigh", L2 is calledthe "femur" and L3 is the "tibia".The Table 1 represents the parameters obtained through the applicationof the Danavit-Hartenberg method.Table 1. Denavit-Hartenberg pa-rameters.Link ai αi di θi1 L1 pi/2 0 θ12 L1 0 0 θ23 L1 0 0 θ34. MethodologyIn this section we will discuss the structure of the adopted system,the manual gestures chosen, the structure of the algorithm and finally themethod of the classifier analysis.The structure of the system corresponds to a sequence of steps that willallow the tracking of gestures in real time. Each step of recognition beginswith algorithms to establish a region of interest, segmenter to reduce noiseand improve classifier processing and lastly the use of the classifier. Fromthe recognized gesture the robots will execute a certain movement. Theoperation of the used algorithm can be analyzed in the Figure 3.August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6435Figure 3. Flowchart of the proposed methodFor the real-time tracking of the object of interest was used the continu-ously adaptive algorithm CamShift which is an adaptation of the algorithmof MeanShift. For each captured frame, the original image is converted toa probabilistic color distribution of the image, which using a histogrammodel of the color to be traced.12 In this case, the characteristic color tobe tracked is that of the hand, after having recognized this pattern, thedesired location is made in the scene, and then the segmentation of this asshown in Figure 4.b.Figure 4. a) Gestures used. b) Tracking and segmentation the object of interest. c)Usedrobots.Four gestures were used to control the robots Figure 4.a, each gesturetransmits an action to the robots being able to be of two categories, col-lective or individual. The gesture "G1" conveys a collective action for therobots which consists of the same ones executing a forward movement move-ment, "G2" just like the previous gesture transmits a collective action caus-ing the robots to return to the initial position, "G3" by its Instead it consistsof an individual action that causes the robot from the lower right cornerto move to the center, "G4" transmits an individual action that causes therobot from the upper left corner to move up to the scepter.The next step is segmentation. The algorithm chosen to perform thisAugust 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6436stage of the system is Theshold algorithm, in order to reduce the noisesin the image to obtain a greater precision in the classifier. This type ofalgorithm is based on the principle that an object of interest in the scenehas distinct quantifiable characteristics, such characteristics as intensity ormagnitude of the gradient.13This segmentation process is based on the search for pixels whose valuesare within the ranges defined as thresholds. From a threshold establishedwith the characteristics of the objects you want to isolate, the image canbe segmented into two groups: the group of pixels with gray levels belowthe threshold and the group of pixels with gray levels above the threshold.If the value of the pixel is greater than a value stipulated as a limit, it isassigned a value, in this case it is white, however otherwise, another valueis assigned that is black, the relevance of this method can be seen in Figure4.b.The last step is to start the classifier training that will be responsiblefor performing the signal recognition. The RBM algorithm was used. Inaddition, RBMs can be used as non-linear classifiers and can achieve betterperformance compared to conventional neural networks and SVMs.14A RBM is a stochastic neural network composed of two layers,14 visi-ble and hidden. The visible units layer represents the observed data and isconnected to the hidden layer, which in turn must learn to extract charac-teristics of this data. In a RBM neurons of the same layer are not connectedbetween themselves. Therefore, there is only a connection between neuronsof different layers, so the machine is restricted.For the RBM classifier, 5685 images were used, which are an adaptationof the datasets provided by,15 3411 used for training consisting of 60 % ofthe total and 2274 images for testing consisting of 40 %. As a statisticaltool we have the confusion matrix,16 which provides the basis for describingthe accuracy of the classification and characterizing the errors, helping torefine the classification.175. ResultsSatisfactory results were obtained for each gesture evaluated, worst case,"G1", with 61% as shown in Table 2. The overall accuracy of the approachis 82.2%August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6437Table 2. Confusion matrixKnown G1 G2 G3 G4G1 61% - - -G2 - 95% - -G3 - - 76% -G4 - - - 92%Note: Predicted %Figure 5. Manipulating the robots.In the Figure 5 is displayed to the real-time evaluation of the methodused in the project, it can be observed that the gesture used is the "G4",that means that the command is transmitting a collective action, causingthe robots to return to their initial position.6. ConclusionThis article presents a real-time multirobot control system from robustand reliable hand signals. In addition, a good accuracy was obtained in theapplication of the algorithm chosen to classify the gestures Table 2.The system presented satisfactory results with regard to real-time videoprocessing, obtaining analysis and classification of the gestures, in order totransmit information to the robots, and with this to be performed actionsinstantly.As future work it is suggested to increase the number of gestures, totest with other classifiers and to implement in real robots.AcknowledgmentsThis work is financed by the ERDF - European Regional DevelopmentFund through the Operational Programme for Competitiveness and Inter-August 23, 2018 19:4 WSPC - Proceedings Trim Size: 9in x 6in ws-procs9x6438nationalization - COMPETE 2020 Program within project POCI-01-0145-FEDER-006961,and by National Funds through the FCT - Fundação para aCiência e a Tecnologia (Portuguese Foundation for Science and Technology)as part of project UID/EEA/50014/2013.Bibliography1. M. F. Silva, J. A. T. Machado and A. M. Lopes, Robotica 23, p. 595–606(2005).2. N. M. Fonseca Ferreira, F. Moita, V. Santos, J. Ferreira, J. Santos, F. Santosand M. Silva, Education with robots inspired in biological systems(04 2018).3. Y. Girdhar and G. Dudek, Autonomous Robots 40, 1267 (2016).4. M. Li, M. Su, R. Xie, Y. Zhang, H. Zhu, T. Zhang and Y. Guan, Develop-ment of a bio-inspired soft gripper with claws, in Robotics and Biomimetics(ROBIO), 2017 IEEE International Conference on, 2017.5. transações IEEE em imagens médicas 35, 1262.6. M. Luneckas, T. Luneckas, D. Udris and N. M. Fonseca Ferreira, Elektronikair Elektrotechnika 20, 7(09 2014).7. X. Xiong, F. Wörgötter and P. Manoonpong, IEEE transactions on cyber-netics 46, 2521 (2016).8. H. Yu, H. Gao, L. Ding, M. Li, Z. Deng and G. Liu, IEEE Transactions onIndustrial Electronics 63, 5488 (2016).9. P. Corke, Robotics, Vision and Control: Fundamental Algorithms InMATLABR© Second, Completely Revised (Springer, 2017).10. A. Zeiaee, R. Soltani-Zarrin, R. Langari and R. Tafreshi, Design and kine-matic analysis of a novel upper limb exoskeleton for rehabilitation of strokepatients, in Rehabilitation Robotics (ICORR), 2017 International Conferenceon, 2017.11. M. S. Couceiro, N. M. Fonseca Ferreira and J. A. T. M. Dainius, Journal ofCommunications in Nonlinear Science and Numerical Simulation 17, 893(022012).12. C.-H. Hsia, Y.-J. Liou and J.-S. Chiang, Journal of Real-Time Image Pro-cessing 12, 183 (2016).13. M. Yazdi and M. Fani, Shot boundary detection with effective prediction oftransitions’ positions and spans by use of classifiers and adaptive thresholds,in Electrical Engineering (ICEE), 2016 24th Iranian Conference on, 2016.14. C. Gou, K. Wang, Y. Yao and Z. Li, IEEE Transactions on Intelligent Trans-portation Systems 17, 1096 (2016).15. P. K. Pisharady, P. Vadakkepat and A. P. Loh, International Journal ofComputer Vision 101, 403 (2013).16. X. Deng, Q. Liu, Y. Deng and S. Mahadevan, Information Sciences 340, 250(2016).17. A. Saraiva, R. T. Melo, V. Filipe, J. V. M. Sousa, N. M. Fonseca Ferreiraand A. Valente, international journal of systems applications, engineeringand development 12, 63 (2018).

https://revistas.utp.ac.pa/index.php/memoutp/article/download/2013/2955/

Navigation of quadruped multirobots by gesture recognition using restricted boltzmann machines

Abstract

Similar works

Full text

Available Versions

Repositorio Institucional de la Universidad Tecnológica de Panamá

Portal de Revistas Académicas UTP (Universidad Tecnológica de Panamá)