The problem of hand tracking in the presence of occlusion is addressed. In bimanual movements the hands tend to be synchronised effortlessly. Different aspects of this synchronisation are the basis of our research to track the hands. The spatial synchronisation in bimanual movements is modelled by the position and the temporal synchronisation by the velocity and acceleration of each hand. Based on a dynamic model, we introduce algorithms for occlusion detection and hand tracking

Shamaie, Atid

Sutherland, Alistair

English

Name not available

A Dynamic Model for Real-Time Tracking of Hands inBimanual MovementsAtid Shamaie and Alistair SutherlandCentre for Digital Video Processing, School of Computer Applications,Dublin City University, Dublin 9, Ireland{ashamaie, alistair}@computing.dcu.ieAbstract. The problem of hand tracking in the presence of occlusion is ad-dressed. In bimanual movements the hands tend to be synchronised effortlessly.Different aspects of this synchronisation are the basis of our research to trackthe hands. We propose a Kalman filtering-based dynamic model to catch themovement of the hands. The spatial synchronisation in bimanual movements ismodelled by the position and the temporal synchronisation by the velocity andacceleration of each hand. Based on this model, we introduce algorithms for oc-clusion detection and hand tracking during occlusion.1 IntroductionMany tasks are done everyday by our two hands together. In order to achieve ourgoal, our hands’ movement have to be finely coordinated. Activities where bimanualcoordination is important include clapping, opening a wine bottle, typing on a key-board, eating with knife and fork, drumming, guiding a pilot driving an aircraft to theparking, showing the size of something (e.g. a fish), etc. Tracking the hands in a bi-manual movement is important in order to recognise the meaning of the movement.Hand tracking and gesture recognition have been widely addressed in the literature[1]. Spatio-temporal hand gesture recognition [2], hidden Markov models for gesturerecognition [3], parametric hidden Markov models [4], HMM-based threshold modelsfor gesture recognition [5], position based gesture recognition [6], tracking interactinghands using Bayesian networks [7], tracking of articulated structures in disparitymaps derived from stereo image sequences [8], tracking of multiple articulated ob-jects in the presence of occlusion in moderately complex scenes [9], representing andrecognising human body motion [10], hand tracking for behavior understanding [11]and many other techniques have been used to deal with the problems of tracking andrecognition of hand and body gestures.In this paper we introduce a new tracking algorithm based on a dynamic model andKalman filtering for bimanual movements. In the next section the main problem isaddressed. In Section 3, we will discuss bimanual movements in more detail. In Sec-tion 4, the process of segmentation and extraction of hands from the background isgiven briefly. Section 5 is dedicated to the dynamic model and the algorithms for oc-clusion detection and tracking. Some experimental results are presented in Section 6.Comparisons to the other works and conclusion are presented at the end of paper.2 Tracking Hands in Bimanual MovementsIn a bimanual movement, when one hand, completely or partially, covers the otherhand resuming tracking accurately at the end of occlusion is crucial. Since we don’tknow what happens during occlusion, after the end of occlusion, we have to knowwhich hand in the image is the right hand and which one is the left. The hands maypass each other, movement types a, c, d, and h shown in Fig. 1(a), 1(c), 1(d), and 1(h),or may collide and return in opposite direction, types b and g presented in Fig. 1(b)and 1(g). In some cases they may not to collide but return in opposite direction, typese and f shown in Fig.  1(e) and 1(f).3 Bimanual CoordinationIn bimanual movements, naturally, there is some sort of synchronisation between thehands [12]. This synchronisation appears in temporal and spatial forms [13]. Tempo-rally, when the two hands reach for different goals, they start and end their move-ments simultaneously [13]. Spatially, we are almost not able to draw a circle and arectangle by the two hands at the same time [13]. Researchers have presented differ-ent anatomical and perceptual explanations for this phenomenon [14][15].  The tem-poral synchronisation in bimanual movements enables us to detect the concurrenthand pauses. These pauses help us to track the hands during occlusion. In order todetect the hand pauses we monitor the hand velocities. A very well known experimentcalled Circle Drawing shows that the two hand velocities are highly synchronisedwith no phase difference in bimanual movements [16]. This synchronisation is the ba-sis of the algorithm proposed here.4 Hand Extraction and Pre-processingIn order to extract the hand from the background we use colour segmentation and analgorithm called Grassfire [17]. The Grassfire algorithm scans the image from left toright, top to bottom to find connected areas with the colour belonging to the hands.The first connected area is labeled number 1, the second is labeled number 2, and soon. However, because of the search manner of Grassfire, in two consequent imageframes the hands may be labelled differently. The other problem is occlusion. As soonas the hands reach each other the Grassfire algorithm finds a big blob and labels it asone object. Detecting occlusion and tracking the hands are the problems to be ad-dressed in the next section.5 A Dynamic Model for Occlusion Detection and TrackingIn this section, first we deal with the problem of detecting occlusion and then trackingthe hands.Fig. 1. The path of the hands in the 8 types of the bimanual movements. The thick ellipses rep-resent the occlusion areas (a, c, d, e, f, and h), and the small lines represent collision (b and g)5.1 Occlusion DetectionIn order to detect occlusion, for every hand in an image a rectangle is constructedaround it. The sides of each rectangle represent the left most, right most, top and bot-tom of the hand. By tracking the rectangles if there is any intersection between them itcan be recognised as an alarm for occlusion. However, it is possible that without anyprior intersection occlusion happens, Fig 2, and the Grassfire algorithm detects onlyone area, which is the same as the case that one hand hides behind a part of body. Weuse a model to predict future movement of each hand and catch occlusion.5.2 A Dynamic ModelWe propose a Kalman filtering-based algorithm to track the movement and predict thefuture position of the rectangles a few steps in advance. Every side of a rectangle ismodelled by a dynamic model. This gives us model independence from the handshapes. The position, velocity and acceleration of each side are considered in thismodel. These parameters are related together [18] based on the Equation 1 for i=1, 2for the two hands and j=1, 2 for the left and right or top and bottom sides.Fig. 2. A rectangle is formed around the hands or the big blob in occlusionH1 H2H2 H1aH1 H1H2 H2dH1H1H2H2cH1H1H2H2 gH1H1H2H2bH1H1H2H2 eH1H1H2H2 fH1H2H1H2 ht t+1+=++=++ikjikjikjikjikjikjikjxhxxxhxhxx,,1,,2,,1, 21 .(1)where h>0 is the sampling time [18], k is the time index, x is the position, x  the ve-locity and x  the acceleration [18] of every side of a rectangle. In the normal Kalmanfiltering loop [19] the estimation of a state vector representing a rectangle side is up-dated with the current measurement of the position of the side and its prior estimationcalculated in the last iteration of the loop. We use this prior estimation to predict thenext position of each side of a rectangle one step in advance. If, by prediction, there isany intersection between the rectangles of the two hands, an occlusion alarm is set.Therefore, we are able to forecast a hand-occlusion before it actually happen.5.3 Hand TrackingBy defining the centre of each hand and comparing them in consecutive images theproblem of mislabelling in the consecutive frames can be resolved. By using thistechnique we are able to track the hands correctly even when something else like apart of body occludes them. It can be done by keeping records of the last position ofthe hand before occlusion and the position of the other hand. This is expected becausewhen a hand moves behind something like the body or moves out of the image frameit most probably appears in an area close to the last position before the occlusion.In order to track the hands when they occlude each other we introduce a techniquebased on the Kalman filtering model of section 5.2. As in Section 5.1 a rectanglearound each hand is constructed. As soon as the occlusion is detected by our system abig rectangle around the big blob is formed. We use the Kalman filtering model of thelast section to model the big rectangle. During occlusion, if the vertical sides of therectangle stop, the velocities of these sides reach zero (or a small number with respectto their normal velocity). This is recognised as a horizontal pause of the hands. There-fore, they would go back horizontally. However, if the hands pass each other we ob-serve a sign change in the velocities of the vertical sides without detecting any pause.These are similarly observable in the vertical or diagonal movements of the hands. Bycatching the hands pause during occlusion and comparing the current position of thehands at the end of occlusion, with respect to each other, with their position prior toocclusion we can decide the correct position of each hand. We label the sides of thebig rectangle during occlusion with a and b for the horizontal and c and d for the ver-tical sides.  The following measurements are defined,+=+=2222dchbavvvVvvV .(2)where av  and  bv  stand for the velocity of horizontal sides a and b, cv  and dv  standfor the velocity of vertical sides c and d. Therefore, if any of these measurements getclose to zero, say less than a small value of ε , a vertical or horizontal pause is de-tected. For example, if the hands have a movement of type d, Fig. 1(d), in which thehands just pass each other with no pause and no collision we detect no horizontalpause and conclude that the position of the hands is the opposite of their position priorto occlusion. In Figure 1(g), however, the hands collide and the model detects it.Therefore, the correct positions of the hands at the end of occlusion are the same asbefore the occlusion. In some of the movements like types c and d, Fig. 1(c) and 1(d),a horizontal or vertical pause may be detected during occlusion. This may cause awrong decision by the algorithm. In order to deal with this we take the relative veloc-ity of the rectangle sides into account. The standard deviation of the velocity differ-ences of the horizontal sides, vS , and vertical sides, hS , are used. This labelling isused because the horizontal sides represent the vertical movement of the hands andvice versa. If a small standard deviation in one of these measurements is observed wedo tracking by detecting the pauses in the movement of the other sides of the rectan-gle. The following is an abstract of the decision-making algorithm,1. If  vS  is less than a particular value   1.A. If ε<hV  then: the hands are horizontally back to their original position   1.B. Else: the hands horizontally passed each other2. Else: if hS  is less than a particular value   2.A. If ε<vV  then: the hands are vertically back to their original position   2.B. Else: the hands vertically passed each other3. Else: if ε<hV  then: the hands are horizontally back to their original position4. Else: if ε<vV  then: the hands are vertically back to their original position5. Else: the hands passed each other6 Experimental ResultsFor the sake of brevity, we just look at the results of the dynamic model in trackingand predicting one of the sides of a rectangle. The results for the others are similar.Fig. 3 shows a part of an experiment. The predicted next position, the triangles, ateach time is pretty close to the actual position of the side of the rectangle at the nextstep, the solid small circles.The velocities of the rectangle sides during occlusion show a kind of uniformity insome of the movements like types a, b, d, and e, Fig. 1(a), 1(b), 1(d), and 1(e). The re-sult of an experiment is shown in Fig. 4(a). In this experiment the difference of thevelocities at each time slice is less than a small value represented by the vertical linesin the graph. This uniformity causes small values of standard deviations of the relativevelocities. This is true also for the vertical sides in movements of type c, Fig. 1(c).However, the velocities of the sides of the rectangle in cases where the hands passeach other in opposite directions (e.g. movement type h) show opposite movements,Fig. 4(b).  Large standard deviation in this case enables us to detect the non-uniformmovements of the parallel sides of the rectangle. A change in the sign of the velocitiesin this figure is observable. When one hand passes the other they push the sides in theopposite directions. A few images of a movement of type h are shown in Fig. 5. For amovement of type g, in which the hand collision is detected by the model , the graphsFig. 3. The result of predicting by the dynamic model in a hand movement. The solid small cir-cles represent the actual position and the triangles are the predictions one step in advanceFig. 4.  Relative velocities of the horizontal sides of rectangles during occlusion in differentmovements. The small circles represent the top side and the triangles represent the bottom sideof the rectangle sides velocities during occlusion are presented in Fig. 6. The handcollision is detected in the 9th to 11th time slices in which all the velocities get veryclose to zero.In order to measure the performance of the algorithm 3500 experiments were donewith many different hand shapes. The results of the experiments are presented in Ta-ble 1. The other important parameter is the processing speed. With a fast cameraworking in 120 frames per second on a PII-1 GHz the algorithm is able to process37.5 frames per second in average.Although some other groups have done some work on hand tracking, none of themhas considered the problem of tracking the hands during occlusion so widely.  TheCONDENSATION algorithm [20] is implemented by Gong et al. [7] works at a lowspeed able to process an image in 24 seconds on a PII-330 MHz. Gong et al. have alsoa Bayesian Network based technique for modelling the semantics of interactive be-haviors able to process 5 frames per second on a PII-330 MHz [7]. Their result of13% error are based on the number of images in the database and not on the numberof events.  In our experiments of Table 1 the tracking results are event-based in whicheach event (movement) may consist dozens of images.1 0 01 2 01 4 01 6 01 8 02 0 02 2 02 4 02 6 02 8 03 0 01 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46T i m ePosition-8-6-4-20246810121 2 3 4 5 6 7 8 9 10 11 12TimeVelocity-8-6-4-20246810121 2 3 4 5 6 7 8 9 10 11 12TimeVelocitya bFig. 5. A hand movement of type h. The hands are tracked correctly after occlusionFig. 6. Graphs of the velocities of the rectangle sides during occlusion. The velocities of the (a)vertical sides, (b) horizontal sides have drop near zero from 9th to 11th time sliceTable 1. Performance of the tracking algorithm using the dynamic modelMovement Type # Events(movements)# Errors Error Rate (%)a 400 25 6.25b 600 106 17.67c 400 65 16.25d 500 29 5.8e 400 41 10.25f 400 44 11g 400 17 4.25h 400 24 6Total: 3500 Weighted Average: 10.03McAllister et al. [11] have, recently, introduced a model for hand tracking in smartdesks and driving.  In this model the camera should be in a top-view position in whichthe both hands and forearms are visible. Using optimization they fit a circle to thepalm and a line to the forearm. For some of the hand shapes the algorithm cannot fitthe circle.However, the model presented in this paper doesn’t have those restrictions and isindependent of hand shape and camera position. Therefore, for different angle ofviews and the changing hand shapes during movements it works effectively.-15-10-50510151 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16TimeVelocity-15-10-50510151 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16TimeVelocitya bReferences1. Cipolla, R., Pentland, A.: Computer Vision for Human-Machine Interaction. CambridgeUniversity Press (1998)2. Lin, D.: Spatio-Temporal Hand Gesture Recognition Using Neural Networks. Proc. IEEEWorld Congress on Computational Intelligence (1998)3. Starner, T., Pentland, A.: Visual Recognition of American Sign Language Using HiddenMarkov Models. Proc. Int’l Workshop on Automatic Face and Gesture Recognition, Zurich,Switzerland (1995)4. Wilson, A.D., Bobick, A.: Parametric Hidden Markov Models for Gesture Recognition.IEEE Trans. Patt. Anal. Mach. Intell., Vol. 21, No. 9  (1999)5. Lee, H., Kim, J.: An HMM-Based Threshold Model Approach for Gesture Recognition.IEEE Trans. Patt. Anal. Mach. Intell., Vol. 21, No. 10  (1999)6. Ng, C.W., Ranganath, S.: Gesture Recognition via Pose Classification. Proc. Int’l Conf. Pat-tern Recognition ICPR’00, Barcelona, Spain (2000)7. Gong, S., Ng, J., Sherrah, J.: On the Semantics of Visual Behaviour, Structured Events andTrajectories of Human Action. Image and Vision Computing, Vol. 20  (2002)8. Jojic, N., Turk, M., Huang, T.: Tracking Self-occluding Articulated Objects in Dense Dis-parity Maps. Proc. IEEE International Conf. Computer Vision ICCV’99, Kerkyra Greece(1999)9. Dockstader, S., Tekalp, A.M.: Tracking Multiple Objects in the Presence of Articulated andOccluded Motion. Workshop on Human Motion HUMO’00, Austin Texas (2000)10. Campbell, L., Bobick, A.: Recognition of Human Body Motion Using Phase Space Con-straints. 5th International Conf. Computer Vision, Cambridge Massachusetts (1995)11. McAllister, G., McKenna, S. J., Ricketts, I. W.: Hand Tracking for Behaviour Understand-ing. Image and Vision Computing, Vol. 20  (2002)12. Jackson, G.M., Jackson, S.R., Husain, M., Harvey, M., Kramer, T., Dow, L.: The Coordi-nation of Bimanual Prehension Movements in a Centrally Deafferented Patient. Brain, Vol.123  (2000)13. Diedrichsen, J., Hazeltine, E., Kennerley, S., Ivry, R.B.: Moving to Directly Cued Loca-tions Abolishes Spatial Interference During Bimanual Actions. Psychological Science, Vol.12, No. 6  (2001)14. Mechsner, F.: Why Are We Particularly Good at Performing Symmetrical Movements.Max-Planck Research  (2002)15. Mechsner, F., Kerzel, D., Knoblich, G., Prinz, W.: Perceptual Basis of Bimanual Coordina-tion. Nature, Vol. 414  (2001)16. Kennerley, S., Diedrichsen, J., Hazeltine, E., Semjen, A., Ivry, R.B.: Callosotomy PatientsExhibit Temporal Uncoupling During Continuous Bimanual Movements. Nature Neurosci-ence. Online Publication  (2002)17. Pitas, I.: Digital Image Processing Algorithms, Prentice Hall (1993)18. Chui, C.K., Chen, G.: Kalman Filtering With Real Time Applications. Springer-Verlag(1999)19. Brown, R.G., Hwang, P.Y.C.: Introduction to Random Signals and Applied Kalman Filter-ing. John Wiley and Sons (1997)20. Isard, M., Blake, A.: CONDENSATION - Conditional Density Propagation for VisualTracking. Intl. J. Computer Vision, Vol. 29  (1998)

A dynamic model for real-time tracking of hands in bimanual movements

DCU Online Research Access Service

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.12.5475

A dynamic model for real-time tracking of hands in bimanual movements

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service