This paper presents an interactive computer environment to be implemented on a mobile robot for human-robot interface (HRI) by incorporating skin detection technique. Human face detection yielded by visual sensor is the core towards an interaction between human and robots. Hence, efficient skin detection can serve as the basis towards producing fast face feature extraction for robotics application. The proposed skin detection method is based on color space analysis. The goal is to demonstrate that a common pattern exists among human skin color. In order to support the human-robot interaction application, a hardware and software structure is proposed for the HRI

H. M. Amin, Shamsudin

Wong, Hwee Ling

English

Universiti Teknologi Malaysia Institutional Repository

Student Conference on Research and Development (SCOReD) 2003 Proceedings, Putrajaya, Malaysia An Interactive Environment for a Mobile Robot Using Skin Detection Wong Hwee Ling and Shamsudin H. M. Amin Center for Artificial Intelligence and Robotics (CAIRO) Faculty of Electrical Engineering Universiti Teknologi Malaysia 8 13 10 UTM Skudai, Johor, Malaysia. TeI: 07-5535793 e-mail wongl~weelinn~~~yahoa.com Abmacr- This paper presents an interactive computer environment to be implemented on a mobile robot for human-robot interface (HlU) by incorporating skin detection technique. Human face detection yielded by visual sensor is the core towards an interaction between human and robots. Hence, efficient skin detection can serve as the basis towards producing fast face feature extraction for robotics application. The proposed skin detection method is based on color space analysis. The goal is to demonstrate that a common pattern exists among human skin color. In order to support the human- robot interaction application, a hardware and software structure is proposed for the HRI. Keywords Human-robot interaction, skin color detection. I. INTRODUCTION Study in human-robot interaction (HRI) is important to develop robots that can collaborate and support humans in a save and fiiendIy environment. With assumption that the machine will perfom human bidding at the end, the methods for communicating the goal and the methods for promoting user understanding in a mobile robot must take a more central role [IJ. The key problems in human-robot interaction stated by Simsarian are the visualization of the remote environment, multi-modal specification of spatial- temporal tasks and framework for human robot relationship. Examples of BRI are home robot, ofice robots, intelligent toy, etc Detection of a human presence is the initial step towards HRI. Hence, face detection is a suitable approach in building a strong representation of a human in a real- environment. In recent years, 'there are a number of studies in face detection. Processing time for the algorithm is time-constraint in real-time implementation, mainly robotics application. Face detection can be done in many methods. The choice of method depends on the application required. An overview of face detection techniques is found in [2, 31. Challenges in face detection are pose, presence or absence of face structural components, facial expression, occlusion, image orientation and imaging condition. Skin color based face detection is a type of technique that can provide fast and robust processing. . Skin color based 0-7803-8 173-4/03/$17.00 02003 EEE. face extraction and segmentation can be seen in [&IO]. In HSI and YUV color space, pixels can be separated from its luminosity and chrotnaticity. Studies have shown that skin color is consistent in its chromaticity values. Hence, the skin color detection approach will be utilized in our system in order to detect human presence. Upon successful detection of a human presence, the robot must be able to understand a human and interact with hidher. In this study, human and robot will interact with each other in English language through manipulation of keyboard and buttons. The LCD display on a notebook is to show the commands entered, instructions and human skin detection results. By utilizing human language, that can provide a natural interaction to create comfort and comprehensibility for a user in HRI. A graphical user interface (GUI) is written to accommodate the dialog-based interaction process. The organization of this paper is as follows. In section 11, we present the hardware of a mobile robot to support the HRI. Section I11 will be discussed on software architecture that includes skin color detection techniques for face detection and an interaction workflow for HRI. Section IV shows current results on the skin color-based face detection and discussions. Finally, conclusion will be prescnted in section V. II. HARDWARE DESIGN Prototype of a robot was built in order to test the skin based face detection in a real-world environment. I t  can be generalized as the robot itself and tools for HRI. A. TheRobot The prototype has a composition of a basic robot system, which includes the mechanical and electronic components. The mechanical structure includes the design of robot frame and locomotion. The robot has a height of approximately 1.46 meters tall to accommodate RM. It is divided into 4 layers to house different electronic units. The robot has two wheels for locomotion and two castor wheels mounted on the robot's base for balancing. The electronic system comprises the vision unit, display/ processing/ control unit, interfacing circuits, power unit and motor unit. Based on Fig. 1, each unit is positioned on the layers of the robot. 64 Camera 111. SOFTWARE DESIGN Display, processor and controller Interfacing cards and circuits Power supply and motom Fig. 1. The robot for HlU The camera used in this study is a FireWire camera by Unibriii that comes along with supporting software to capture images. A Toshiba Satellite notebook with .Pentium I11 lGHz processor is utilized for image processing and robot control. The camera is connected to a FireWire PCMCM card for notebook interfacing. The wheels are attached to brushless direct current (BLDC) motors that can be conk01 through supported motor drivers. Theoretically,' a BLDC motor has linearity in speed increment, high torque and high precision. Later, the motor driver i s  interfaced with the notebook through various interfacing circuit boards and ,an input-output interfacing card supplied by Ines. Power supplies for the robot are two 12 volts batteries to support the motors, a camera and an input-output card. B. Hardware Tools for HRI The hardware tool for BRI is located on the notebook itself. T h e  notebook mounted on the robot will provide sufficient interfacing for a human and a robot. The images acquired from the digital camera will be transferred. onto the notebook for processing. The notebook will present the skin detection results on its LCD screen. Besides that, the screen also displays choices, instructions and commands of the human and robot. Fig. 2 shows the basic input-output structure for the HRK. The code for the project is written in Visual C++ 6.0. This platform is selected because of its ffexibility for' programmer to develop the state of the art Windows application h o u g h  the tools provided. The software section can be generalized into skin color detection algorithm to detect human face presence and workflow ofthe inteiaction process on a GVI. A. Skin Color Detection Human skin color detection is employed to detect possible human face. In this study, only a face of a person facing directly towards the robot is detected. If more than one face exists in an image, the face located nearest to the robot is detected with assumption that the human situated nearest to the robot intends to interact with it. At this stage, static images were captured by the camera software and processed using our algorithm. Our algorithm is designed to detect human skin color in YUV color space. Y is the fuminosity white UV is the chromaticity. Initially, images captured are saved into a designated folder. Pixels of every image are in RGB format. Hence, we use the formula from [ 111 to convert RGB into W V  color. For each image, we evaluate the pixels by plotting histograms for each individual component (Y,  U and V). Fig. 3(a) below shows histograms of an image for its Y, U and V values. Fig. 3(b) shows histograms of a 100% skin region cropped from the original image. The y-axis o f  histogram is the number of occurrence while x-axis is the pixel value. Both images were evaluated at a resolution of 55x70. Based on the histogram, it can be observed that U histogram and V histograms show more concentration of pixels in values compare to Y histograms. If the image is 100% skin region, the pixel range for UV histogram is smaller. In this study, other face images are also evaluated for their histograms. Likewise, the shape and location of the UV histograms are almost similar. After the study, new images are processed in a resolution of 160x120. Through those experiments, we threshold the image into binary image by: The U image and V image are ANDed to increase possibility of skin-like colors. However, 'salt and pepper' noise still exist in the image. Thus, the binary image undergoes another median filtering process to eliminate the existing noise. From the filtered image, the longest white column was detected. Then, the longest white row corresponding to the column was computed. The assumption taken is that the largest white color pixels clustered is the probable face region. Then, a box is plotted on the RGB image based on the Iongest Fig. 2. HRI tools on the mobile robot 65 column and row detected. The overall process to detect the largest cluster in an image is displayed Fig. 4. !@@ Cropped skin section Fig. 3 (a). Y, U and V histograms for original image (b). Y, U and V histogram for skin region Later, the .width and height of the detected region (grey box) is evaluated. Several crisp rules were applied to eliminate non-face skin color region. Many non-face skin color and background can be eliminated by implementing these rules, Firstly, width of a human face must never be longer than the height of the face. Next, the width and the height of a human face must not be equal. Thirdly, a ratio is set for heightlwidth as height of face must not be too long compare to face. For example, if height is  100 pixels long while width is 25 pixels long will result in a ratio of4 ,  which is absurd to be a face image. Finally, OUT algorithm a130 eliminates pixel area smaller than a threshold as small image tends to lose its structure information. Besides that, the robot will not tend to interact with a human located far away. Each image will undergo four iterations of face verification process. If no face is detected in all four, then assume that the image is faceless. The final resutt is display in a white box as shown in Fig. 5 bellow, U V  i n q p  arc ilircslrotd into binary images and I’erfornl mcdian threshold on the binary rmagc Obtain longest column then longest row for (he Display largest white blob Fig. 4. Process overview to detect possible face area Fig. 5. Final detection is displayed in yellow box B. Workflow for HRI Process The robot must be able to sense a human presence before starting the interaction process, Through section 3.1, possible face region is detected and the robot will begin the HRI process. The code in this study is written on a Visual C++ 6.0 platform as the software provides sufficient tools to build a GUI for HRI application. As oppose to monologs, dialog is a conversation between two or more parties. Dialog is commonIy used in our HRI to enable a human and a robot to interact with one another. Many diagrammatic notations are used in dialogs design [ 121. Diagrammatic notations are important to allow a programmer to see at a glance the structure of the dialog. In our case, flow chart is applied to show the overall interaction process of a human and the robot. In the workflow, the dialogs can be divided into four phases; starting phase; introduction phase, permission phase and interruption phase. In Fig. 6, several dialogs examples are inserted into the flow chart to visualize interaction between human and robot. English language is utilized in the dialogs to exchange information. Human language should be able to provide interaction in a normal and understandable manner for a novice. user interface. Later, human communicates with the robot through another edit box and control buttons. If he intends to continue, the robot will perform frame-by- frame processing to continuoudy detect a face region by coordinating it with the robot’s mechanism in order to piace the human face around the centre of the image frame. The robot will continue the detection unless the human intempts the program. If  an error was encounter, an error message will be displayed. The program runs in continuous loops until the designer switch off the robot. I Robot: “Welcome lo HRL Please enreryotrr nnme in the edi! box to proceed.” (H~ininn enters name lo conlinue) introduction phase Robot: “Hi (human nnme),I am Robol X. This is a face defection npplicntion ... ... .... ” : I I Permission phase I Robot: “(humnn name). do you wan! to join this rerl? ’’ (Huuninri presses Retecl/No1 To Detect burton) Interruption phase (Ijuser presses Stop button) RODOI: “Thanks far pflrlicipaling in this tcst npplicnliun. Nice meeringyou. Good bye. ” Fig. 6. A workflow to provide an interactive environment for the user interface. .. . Fig. 7. GUI for HRI upon detecting face region ’ To accommodate this study, we design a GUI as shown in Fig. 7. The GUI provides an interactive environment for the H€U. Upon detecting a face, the robot displays message in human language through the edit box in the IV. RESULTS & DISCUSSIONS The hardware system is completed and currently under testing. The skin color detection is tested on static images acquired from the digital camera. The world environment taken for testing is at Robotics Lab, Faculty of Electrical Engineering, Universiti Teknologi Malaysia, Skudai, Johor. The face detection and robot control algorithm will be integrated into the workflow in Fig. 6 once each component is successfully completed and tested. Results in Fig. 8 are images taken from our database. True face detection (grey dot) refers to image in the white box that belongs to a face or the region is mostly to a face region. It also means that image in grey box is not a face, False face detection (black dot) occurs when inmge in white box is not a face, more than one face or a partially hidden face. It applies, too, if image in grey box i s  a face. Through the results shown, one can 67 observe that the algorithm is robust to different skin tone and size. Skin color detection method provides fast processing time for face localization. However, the result can be affected by-background and absence of facial components due existence of other human parts (moustache, beard, spectacles, etc). Overall, , the detection aIgorithm has to be improved to reduce false face detection. Furthermore, it must be employed in real-time application. Fume  work for face detection includes face structure anaksis of the detected region. Fig. 8. Results for face detection based on skin color detection in W images. The color indications .shows detection result; grey dot for true detection and black dot for false detection V. CONCLUSIONS Face detection can provide a natural and transparent representation of human presence in the robot’s surrounding. The aim of this study is to develop a face detection aIgorithm for human-robot interface. Detection of skin color provides robust and fast computation for face detection. Processing time to evaluate the face region is reduced as the program does not need to scan though the whole image to verify a face. This can be achieved by reprocessing only one detected face region (white box). Design of the physical robot and interfacing tool plays an important role to create an interactive environment for HRI. The graphical user interface (GUI) incorporated with the suggested workflow should be able to create a simple interaction between a human and the robot. This study hopes to contribute towards the development of human and robot interaction through face detection, With a human friendly robotic environment, robot can assist and cooperate with men to accomplish various tasks. VI. ACKNOWLEDGEMENT This work i s  supported by the Malaysian Ministry of Science, Technology and Environment under the Intensification of Research Priority *Areas grants, to which we gratehlly acknowledge. VII. REFERENCES K. T. Simsarian, “Towards Human Robot Collaboration”, Swedish Institute of Computer Science, Stockholm, 2002. Yang, M-H., Kriegman, D. J , ,  Ahuja, N., “Detecting Faces In Images: A Survey”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 1, pp.34-58, January, 2002. E. Hjelmas, “Face Detection: A Survey”, Computer Vision and Image Understanding”, pp. 236-274, 2001. Braun, M., Betty, P., Yoon, S., “Face Detection: Integrating the maximal Rejection Classifier with Color Based Skin Detection”, Report on project for E8368 Class, Stanford University. Chen , Q., Wu, H., Yachida, M., “Face Detection by Fuzzy Pattern Matching”, Proceedings of Fifth International Conference on Computer Vision, pp. Fitsch, J. ,  Lang, Kleinehagenbrock, S. M., Frank, G. A., Sagerer, G., “Improving Adaptive Skin Calor Segmentation Incorporating Results From Face Detection”, Proceedings of 1 lth IEEE International Workshop on Robot and Human Jnteractive Communication, pp. 337 -343, 2002. Hsu, R. L., Abdel-Mottaled, M., Jain, A. K., “Face Detection Color Images”, Proceedings of the 2001 International Conference on Image Processing, V01.2, pp. 681 -684,2001. Kim. C. L., Lee, S. O., Lee, B., Park, G., “An Effective Face Region Detection Using Fuzzy- Neural Networks”, Proceedings of the International Conference on Control, Automation and Systems, Pei, S. C., Tseng, C. L., “Robust Face Detection for Different Chromatic Illumination”, Proceedings of 2002 Intemational Conference on Image Processing, pp. 765 -768, 2002. 591-596, 1995. pp. 1512-1515,2001. [lo] Phung, S.L,Bourzerdoum, A., Chai, D., “A Novel Skin Color Model In YCbCr Color Space and Its Application To Human Face Detection”, Proceedings of 2002 International Conference on Image Processing, pp. 289 -292,2002. [ 1 11 Iillp://cedar.intel.corn [12]A. Dix, I, Finly, G. Abowd, R. Beale, Human- Computer Interaction, Prentice Hall International Limited, Great Britain, 1993. 68 

An interactive environment for a mobile robot using skin detection

http://eprints.utm.my/1950/1/ShamsudinHMAmin2003_AnInteractiveEnvironmentForAMobileRobot.pdf

An interactive environment for a mobile robot using skin detection

Abstract

Similar works

Full text

Available Versions

Universiti Teknologi Malaysia Institutional Repository