Vision-based human-computer interaction is becoming important nowadays. It offers natural interaction with computers and frees users from mechanical interaction devices, which is favourable especially for wearable computers. This paper presents a human-computer interaction system based on a conventional webcam and hand gesture recognition. This interaction system works in real time and enables users to control a computer cursor with hand motions and gestures instead of a mouse. Five hand gestures are designed on behalf of five mouse operations: moving, left click, left-double click, right click and no-action. An algorithm based on Particle Filter is used for tracking the hand position. PCA-based feature selection is used for recognizing the hand gestures. A transition network is also employed for improving the accuracy and reliability of the interaction system. This interaction system shows good performance in the recognition and interaction test

Sun, Y

Wang, F

Wang, Z

Zhang, Z

Journal of Information and Computational Science

English

Brunel University Research Archive

Journal of Information & Computational Science 11:4 (2014) 1037{1045 March 1, 2014Available at http://www.joics.comVision-based Hand Gesture Interaction Using ParticleFilter, Principle Component Analysis andTransition Network ?Zijian Wang a, Zuo Zhang a, Fang Wang b, Yaoru Sun a;aDepartment of Computer Science and Technology, Tongji University, Shanghai 200092, ChinabDepartment of Information Systems and Computing, Brunel University, Uxbridge, United KingdomAbstractVision-based human-computer interaction is becoming important nowadays. It offers natural interactionwith computers and frees users from mechanical interaction devices, which is favourable especially forwearable computers. This paper presents a human-computer interaction system based on a conventionalwebcam and hand gesture recognition. This interaction system works in real time and enables usersto control a computer cursor with hand motions and gestures instead of a mouse. Five hand gesturesare designed on behalf of ve mouse operations: moving, left click, left-double click, right click andno-action. An algorithm based on Particle Filter is used for tracking the hand position. PCA-basedfeature selection is used for recognizing the hand gestures. A transition network is also employed forimproving the accuracy and reliability of the interaction system. This interaction system shows goodperformance in the recognition and interaction test.Keywords: Human Computer Interaction; Hand Gesture Recognition; Hand Tracking; PrincipleComponent Analysis; Particle Filter1 IntroductionCompared with using traditional mechanical interaction devices, vision-based Human-computerInteraction (HCI) offers a more natural and friendly way to interact with computers. In recentyears, there has been a growing interest in vision-based HCI because it's an inexpensive andnon-obstructive solution for HCI [1, 2].Many studies of human-computer interaction were published in past years. Jakub Segen [3]used a predened background that could facilitate segmentation of hand region. Hand gesturesare recognized according to the hand region. Using markers or auxiliary gloves can make thedesign of hand interaction systems easier and improve the eciency of the system [4, 5]. We?Project was supported by the grants from National Natural Science Foundation of China (61173116 and60970062) and Specialized Research Fund for the Doctoral Program of Higher Education (20110072110014).Corresponding author.Email address: yaoru@tongji.edu.cn (Yaoru Sun).1548{7741 / Copyright© 2014 Binary Information PressDOI: 10.12733/jics201028901038 Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045adopted an approach without restrictions of auxiliary tools. Our system can work in some usualbackgrounds, and a user can interact with a computer using a bare hand in real time.With a low-cost webcam, users can use pre-designed hand gestures to control the cursor andperform conventional operations: moving the cursor, left click, left-double click and right click.The system can work in real time on a conventional processor. We propose a color-based segmen-tation method based on the segmentation approach in [6]. Particle Filter algorithm [7] is employedto locate hand position accurately and eciently. Principle Component Analysis (PCA) [8] isused to reduce the dimensions of hand images, and a minimum distance classier is used forrecognition.The remainder of this paper is organized as follows:(1) The hand region extraction method is reported in Section 2.1.(2) The hand tracking algorithm based on Particle Filter algorithm is described in Section 2.2.(3) The recognition method and ve dierent hand gestures on behalf of ve operations as moving,left click, left-double click, right click and no action (shown in Fig. 1) are detailed in Section2.3.(4) A transition network with anti-shake measure and usage of an auxiliary no-action hand gestureare proposed in Section 2.4.(5) Recognition and interaction tests are detailed in Section 3.(6) Finally, we oer some conclusions in Section 4.The schematic diagram of the HCI system is shown in Fig. 1.SegmentTrackRecognizemoveControl the mouseFig. 1: The schematic diagram of the method. Five hand gestures are designed on behalf of ve mouseoperations (images in the left column). The cursor is controlled according to the tracking and recognitionresults. The tracking and recognition processes are based on the result of segmentationZ. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045 10392 Hand Gesture Interaction2.1 Hand SegmentationThe hand region is segmented from the background before tracking and recognition. Hand seg-mentation methods based on skin color have been used as ecient approaches in the previousresearch [6, 9, 10] on the premise that human skin color is distinct from the background color. Anecient skin-color segmentation method based on HSV (Hue, Saturation and Value) color spaceis implemented in our system because dierent skin color has a similar range in the HSV colorspace [10].An image is binarized according to the HSV color range of hand skin: the pixels whose color isin the skin color range are set as \1" while the others are set as \0". But some \1" pixels outsidethe hand region could be mistaken as a part of the hand region by the tracking algorithm. Toresolve this problem, the \1" region enclosed by the longest \1" contour is lled with \1" andthe others are lled with \0". Finally, a new binary image whose \1" pixels stand for the handregion is obtained. The hand segmentation algorithm and the results are shown in Fig. 2.Binarize the frameSelect thelongest contourSet the contourregion and resetorhter pixelsFig. 2: The hand segmentation algorithm and the results. Five hand gestures are shown. The left imagesare video-input images and the rights are binarized images2.2 Hand TrackingIn order to obtain the accurate hand position, an accurate and eective algorithm based onParticle Filter algorithm is used. This algorithm has three steps: initial sampling, resamplingand hand position calculating.Initial sampling runs in processing the rst frame. N sample points are put randomly in thebinary image (the result acquired in Section 2.1). The size of the binary image is W H pixels.Coordinates of the N sample points are < xi; yi > (1  i  N).Resampling runs in every frame. The sample points in the current frame are resampled ac-cording to the condence values and posterior probabilities of the sample points in the previous1040 Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045frame. The condence is calculated by the following equation:confidencei =(1 if < xi; yj >= 10 if < xi; yj >= 0(1)This equation assigns 1 to the condence values of the sample points in the hand region and 0to the others. The posterior probability (PP ) is given by:PPi =confidenceiPNi=1 confidencei(2)It's obvious that PP of a sample point in the background is 0, but PP of one in the handregion is: 1PNi=1 confidencei. The sample points in the hand region are selected for resampling. Toprevent the deterioration of Particle Filter Algorithm, each of them is resampled for RN times(RN = N\1" pixel number). The new position of each resample point is selected randomly in therectangle range from < xi  W=5; yi  H=5 > to < xi +W=5; yi +H=5 >.In the last step, hand position is obtained by:handposition =Xi< xi; yi > PPi where i refers to all the remaining points (3)The hand position is obtained by calculating sample points in this algorithm, but the number ofthe sample points may be reduced to 0 when the hand moves outside the camera window, whichcan lead the tracking to failure. To prevent it, the initial step will restart when the number ofthe sample points is reduced to 0. The sample points and the hand positions are shown in Fig. 3.Fig. 3: Three hand images and the tracking results. Blue circles represent the sample points and redcircles represent the hand positions2.3 Hand RecognitionFive hand gestures stand for mouse operations: move, left click, left-double click, right click andno action. Every hand gesture has ve training images shown in Fig. 4. It will cost a lot of timeto process a W0 H0-pixel-size hand gestures image. The PCA algorithm is used to reduce thedimensions of hand image data. This algorithm is composed of a training phase and a recognitionphase.Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045 1041Move Left click Left-doubleclickRight click No actionFig. 4: The 5 hand gestures and 5 training images for each gesture2.3.1 Training PhaseAn eigenspace is built in the training phase. The eigen hands converted from the prepared traininghands are located in the eigenspace.First, the twenty-ve W0  H0-pixel-size training images are converted into column vectorsxi(1  i  25) by:0BB@a11 : : : a1M0.... . ....aH01 : : : aH0M01CCA = (a11 a21 : : : aH01 a12 : : : aH02 : : : a1M0 : : : aH0M0) (4)A dierence image di can be calculated for each hand image by di = xi 	. The average image	 is dened by:	 =12525Xi=1xi (5)Through the dierence image, a covariance matrix C is built according to:C =125AAT where A = (d1; d2; : : : ; d25) (6)The eigenvalues and eigenvectors of C are used to build the eigenspace. Singular Value Decom-position (SVD) is used to calculate the eigenvalues i and eigenvectors i.The rst ten eigenvalues and eigenvectors are selected in descending order. The bigger theeigenvalue is, the more important the corresponding eigenvector is. The ten eigenvectors makeup the eigenspace ! by:! = (1; 2; : : : ; 10) (7)Every projection vector in the eigenspace represents a hand image. The training images areprojected onto the eigenspace by the following equation:j = !Tdj where j = 1; 2; : : : ; 25 (8)2.3.2 Recognition PhaseRecognition phase runs in every frame. In this phase, the binarized hand image is recognizedin real time. A rectangle is set to cover the hand region as a Region of Interest (ROI). It is1042 Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045also scaled into an image of W0  H0 pixels that has the same size as training images beforerecognition.Recognition is similar to the training phase. First, the normalized image is converted into acolumn vector x0 and the dierence image is calculated by d0 = x0  	. The hand image can beprojected onto the eigenspace, and the projection vector 0 is dened by:0 = !Td0 (9)Finally it is necessary to match the hand image with one of the twenty-ve training images.The Euclidean distance between the hand image and training image is calculated by Eq. (10).The hand image matches with the training image that has the lowest i.i = ki   0k where i = 1; 2; : : : ; 25 (10)2.4 Cursor ControlHand tracking provides the hand position, and hand recognition provides the category of thehand gesture. They are both parameters used in the cursor controlling function. A cursor actiontransition network based on the transition network proposed in [11] is implemented to improvethe accuracy.In the transition network, the cursor responses to the video input in a delay of one frame.It is operated based on parameters in the current, previous and the second previous frames.When the recognition results of the current, previous and the second previous frames are all\move", the cursor starts moving. When the recognition result of the second previous frame is\move" and the ones of current and previous frames are click-kind-action, the cursor will proceedthe corresponding click-kind-action. But the no-action operation is unique. When the currentgesture means no-action, the cursor will do nothing. The transition network is shown in Fig. 5.currentprevioussec prev1 21 21Move Click Donothing131=moving2=click3=no actionFig. 5: Transition network. In this transition network, recognition results in 3 sequential frames todetermine one cursor action (except no-action)The new position is determined by the original cursor location and the scaling hand speed whenthe cursor moves. A measure for reducing the cursor shake is implemented. The hand positionin the current, previous and second previous frame is denoted as C, P and S, respectively. Theangle made by line SP and line PC is . An abrupt change in the moving direction (small ) isprobably caused by an unintentional hand shake, which could be blocked by setting a thresholdof . The moving command is valid only if  is bigger than 100 (Fig. 6).To facilitate moving, a no-action gesture is proposed to help when the hand is moving tothe edge of the camera window. When it's necessary to move the hand far away, even out ofthe camera window, users can move the hand to an appropriate position holding the no-actionZ. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045 1043CααS<100°>100°P Sshake not shakeCPFig. 6: The method to reduce cursor shake. The cursor moves only if  is bigger than 100gesture and then continue the unnished movement. Besides, a deviation will occur when thehand moves into the camera window after the hand moves out. The system reinitializes trackingalgorithm when the hand moves out of camera window to remove the deviation.3 PerformanceIn this section, we test the accuracy and eciency of our system. All participants in our tests wearlong sleeves covering the arms. The application is implemented in Visual C++ using OpenCVlibraries [12] on the laptop platform with Intel Core 2 CPU (2.0 GHz), and 3GB RAM. And theoperating system is Windows 7. The average execution speed of the application is 21 frames persecond. The processing rate of the webcam is 30 frames per second. The average execution speedis faster than the speed in [9] which is 10 frames per second.3.1 Recognition TestTen participants took part in the test and each of them tested 500 hand images, 100 for eachhand gesture. The test images included 50% normal hand images and 50% images of hand witha little bend or rotation. The results are shown in the Table 1. Five thousand hand images weretested in total and the global mean accuracy is 98:3%.Table 1: Accuracy of hand recognition in the testParticipants Move Left Click Left-double Right Click No Action1 100% 97% 97% 99% 99%2 100% 100% 96% 95% 96%3 96% 100% 99% 100% 95%4 100% 98% 97% 96% 100%5 100% 100% 100% 95% 96%6 100% 96% 100% 98% 100%7 100% 100% 96% 99% 98%8 99% 99% 96% 100% 96%9 100% 97% 100% 99% 99%10 100% 99% 95% 98% 100%Average accuracy 99:5% 98:6% 97:6% 97:9% 97:9%1044 Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{10453.2 Interaction TestWe designed three interaction tasks to test the performance of our interaction system. Participantsperformed the tasks using our interaction system and a touch tablet on a laptop respectively.In the rst task, participants created a text le by a right click on the desktop, opened it by aleft-double click and then close it by clicking the left button.In the second task, participants entered disk D through the windows explorer and viewed theattribute of the rst le.In the third task, participants calculated 1 plus 2 using the calculator.The average completion time with our interaction system is 16.5 sec, 14.2 sec and 9.5 sec whilethe time with a touch tablet is 12.4 sec, 9.8 sec, and 5.8 sec, respectively. Considering the maturityof tablets interface design, the performance of our vision-based hand gesture interaction systemis acceptable and has potential for improvement in the future.4 ConclusionIn this paper, we proposed a detailed framework of a vision-based real-time hand gesture interac-tion system. We used a tracking algorithm based on particle algorithm for tracking the hand andused the PCA algorithm for recognition. A cursor action transition network was implemented toimprove the accuracy of the application. This application could work in real time of 21 frames persecond. The experimental results showed a high recognition accuracy of 98.3%. The interactiontests showed a good performance.References[1] Vladimir I. Pavlovic, Rajeev Sharma, Thomas S. Huang, Visual interpretation of hand gesturesfor human-computer interaction: A review, IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol. 19, No. 7, 1997, 677-695[2] Ronald Poppe, Vision-based human motion analysis: An overview, Computer Vision and ImageUnderstanding, Vol. 108, No. 1, 2007, 4-18[3] Jakub Segen, Senthil Kumar, Shadow gestures: 3D hand pose estimation using a single camera,IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, 1999[4] Pattie Maes, SixthSense: Integrating information and the real world, 8th IEEE InternationalSymposium on Mixed and Augmented Reality, ISMAR 2009, IEEE, 2009[5] James M. Rehg, Takeo Kanade, Digiteyes: Vision-based hand tracking for human-computer inter-action, Proceedings of the 1994 IEEE Workshop on Motion of Non-Rigid and Articulated Objects,IEEE, 1994[6] Cristina Manresa et al., Hand tracking and gesture recognition for human-computer interaction,Electronic Letters on Computer Vision and Image Analysis, Vol. 5, No. 3, 2005, 96-104[7] Michael Isard, Andrew Blake, Condensation-conditional density propagation for visual tracking,International Journal of Computer Vision, Vol. 29, No. 1, 1998, 5-28[8] Svante Wold, Kim Esbensen, Paul Geladi, Principal component analysis, Chemometrics and In-telligent Laboratory Systems, Vol. 2, No. 1, 1987, 37-52Z. Wang et al. / Journal of Information & Computational Science 11:4 (2014) 1037{1045 1045[9] Thomas Coogan et al., Real time hand gesture recognition including hand segmentation andtracking, Advances in Visual Computing, 2006, 495-504[10] Y. Shen, S. K. Ong, A. Y. C. Nee, Vision-based hand interaction in augmented reality environment,Intl. Journal of Human-Computer Interaction, Vol. 27, No. 6, 2011, 523-544[11] Yasushi Hamada, Nobutaka Shimada, Yoshiaki Shirai, Hand shape estimation using sequence ofmulti-ocular images based on transition network, Proceedings of the International Conference onVision Interface, 2002[12] Gary R. Bradski, Vadim Pisarevsky, Intel's Computer Vision Library: Applications in calibra-tion, stereo segmentation, tracking, gesture, face and object recognition, Proceedings of IEEEConference on Computer Vision and Pattern Recognition, Vol. 2, 2000

Vision-based hand gesture interaction using particle filter, principle component analysis and transition network

Zijian Wang

Crossref

Vision-based Hand Gesture Interaction Using Particle Filter, Principle Component Analysis and Transition Network

http://bura.brunel.ac.uk/bitstream/2438/11261/1/Fulltext.pdf

Vision-based hand gesture interaction using particle filter, principle component analysis and transition network

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive

Crossref