Skip to main content
Article thumbnail
Location of Repository

Hand gesture recognition for multimedia applications

By Moaath Al-Rajab


Hand gesture is potentially a very natural and useful modality for human-machine interaction. It is considered to be one of the most complicated and interesting challenges\ud in computer vision due to its articulated structure and environmental variations. Solving such challenges requires robust hand detection, feature description, and viewpoint invariant classification.\ud \ud This thesis introduces several steps to tackle these challenges and applies them in a hand-gesture-based application (a game) to demonstrate the proposed approach.\ud Techniques on new feature description, hand gesture detection and viewpoint invariant methods are explored and evaluated. A normal webcam is used in the research as input\ud device. Hands are segmented using pre-trained skin colour models and tracked using the CAMShift tracker. Moment invariants are used as a shape descriptor.\ud \ud A new approach utilising the Zernike Velocity Moments (ZVMs, first introduced by Shutler and Nixon [1,2]), is examined on hand gestures. Results obtained using the\ud ZVMs as spatial-temporal descriptor are compared to an HMM with Zemike moments (ZMs). Manually isolated hand gestures are used as input to the ZVM descriptor which generates vectors of features that are classified using a regression classifier. The performance of ZVM is evaluated using isolated, user-independent and user-dependent data.\ud \ud Isolating (segmenting) the gesture manually from a video stream for gesture recognition is a research proposition only and real life scenarios require an automatic hand\ud gesture detection mechanism. Two methods for detecting gestures are examined. Firstly, hand gesture detection is performed using a sliding window which segments sequences of frames and then evaluates them against pre-trained HMMs. Secondly, the set of class-specific HMMs is combined into a single HMM and the Viterbi algorithm is then used to find the optimal sequence of gestures.\ud \ud Finally, the thesis proposes a flexible application that provides the user with options to perform the gesture from different viewpoints. A usable hand gesture recognition\ud system should be able to cope with such viewpoint variations. To solve this problem, a new approach is introduced which makes use of 3D models of hand gestures (not postures) for generating projections. A virtual arm with 3D models of real hands is created. After that, virtual movements of the hand are simulated using animation\ud software and projected from different viewpoints. Using a multi-Gaussian HMM, the system is trained on the projected sequences. Each set of hand gesture projections is\ud marked with its specific class and used to train the single multi-class HMNI with gestures across different viewpoints

Publisher: School of Computing (Leeds)
Year: 2008
OAI identifier:

Suggested articles


  1. 3D-Tracking of flead and flands for Pointing Gesture Recognition in a Human-Robot Interaction Scenario, " doi
  2. (2008). A AI-Rajab, "Leeds Hand Gesture Dataset, ý. "
  3. (2003). A Brief Overview of Hand Gestures used in Wearable Human Computer Interfaces, "
  4. (2008). A Comparative Study on Using Zcmlkc Velocity Moments and Hidden Markov Models for Hand Gesture Recognition " doi
  5. (1995). A Decision-Theoretic Generalization of OnLine Learning and an Application to Boosting, "
  6. A Fast Subpl\ci Edge Detection Method Using Sobel-Zernike Moments Operator, " doi
  7. (2006). A Framework for Research and Design of Gesture-based Human Computer Interactions, "
  8. A Generic Approach for on-line Handwriting Recognition, " doi
  9. (2004). A Linguistic Feature Vector for the Visual Interpretation of Sign Languagc. " doi
  10. (2005). A Novel Direction Chain Code-bascd Image Retrieval, " doi
  11. A Scalable Model-based Hand Posture Analysis System. " doi
  12. (2007). A Survey of Affect Recognition Methods: Audio, Visual and Spontaneous Expressions, " doi
  13. (1993). A Survey of Gesture Recognition Techniques, "
  14. A Survey of Glove-based Input, " doi
  15. (1999). A Survey of Hand Posture and Gesture Recognition Techniquc,,
  16. (1998). Action Recogmition using Probab ilistic Parsing, "
  17. Adaptive Visual Gesture Recognition for Hunian-Robot Interaction using A Knowledge-Based Software Platforrn, "
  18. Ambiguities in Visual Tracking of Articulated Objects Using Two- and Three-Dimensional Models. " doi
  19. An Appearance-based Framework for 3D Hand Shape Classification and Camera Viewpoint Estimation, " doi
  20. An Experimental Comparison of Trajector-vBased and History-Based Representation for Gesture Recognition, " in GestureBased Communication in Human-Computer Interaction:
  21. An HMM Based Gesture Recognition for Perceptual User Interface, "Advances in Multimedia InIbi-ination Nocc., oýing,,.
  22. (2002). Analyzing Video Sequences ol'Ahiltiple Humans : Tracking, Posture Estimation and Behavior Recognition. doi
  23. (2005). Appearance-Based Real-Time Understandim-, of Gesture Using Projected Euler Angles, " doi
  24. (2005). Automatic 2D Hand Tracking in Video Sequences, " in IEEE Workshop on Applications of Coinputer Vision PP. doi
  25. (2008). Cambridge-Hand-Gesture Database. "
  26. (2004). CAMSHIFT Tracker Design Experiments wid,
  27. Capturing Natural Hand Articulation, " doi
  28. (2006). Cognitive Vision Si'stems. - Sampling the Spectrum ofApproaches.
  29. (1998). Computer Vision Face Tracking For Use in a PercCptLial User Interface, " Intel Technology Journal,
  30. (1998). Condensation Conditional Density Propagation t-or Visual Tracking, "
  31. Data Glove with a Force Sensor. " doi
  32. (1994). Digiteyes: Vision-based Hand Tracking for HunlanComputer Interaction, " doi
  33. Distinctive Image Features firom Scale-fnvariant Keypoints. "
  34. (2007). Elderly 'addicted' to Nintendo Wii at care home, "
  35. Embedded Profile Hidden %Iarko\, Models for Shape Analysis, " doi
  36. Error-Tolerant Sign Retrieval Using Visual Features and Maximum a Posteriori Estimation, " doi
  37. (2005). Explaining visible behaviour, "
  38. (2008). FGnet datasets, " INRIA
  39. (2003). Finger Instead of Mouse: Touch Screens as a . 11cans oj Enhancing Universal Access: SPnnger doi
  40. (2002). Gait Analysis for Recognition and Classification, " in doi
  41. Gestural Intcrface to a Visual Computing Environment for Molecular Biologists, " doi
  42. (2005). Gesture and Thought: published by the Uni% ersity of Cliicago Press,
  43. (2004). Gesture Components for Natural Interaction with In-Car Devices, " doi
  44. (1999). Gesture Recognition for Virtual Reality Applications Using Data Gloves and Neural Networks, " doi
  45. Gesture Recognition Under Small Sample Size, " doi
  46. (2003). Gesture Recognition, " in The human-computer interaction handbook: Lawrence Erlbaurn Associates,
  47. Gesture Recognition: A Survey, " doi
  48. Gesture, " in Cambi-idge Enc-i'dopaedia of the Languq! ze Scienccý,
  49. Good Features to Track, " doi
  50. (2003). Grammar, Gesture, and Meaning in American Sign Languagc: doi
  51. GWindows: Towards Robust Perception- Bascd U'U' doi
  52. (1998). Hand Gesture Estimation and Model Refinement using Monocular Camera - Ambiguitý Limitation bý, -135- 4ppendi-VA Inequality Constraints, " doi
  53. Hand Gesture Rccounition Using Combined Features of Location, Angle and Velocity, "
  54. (2000). Hand Tracking for Vision-Based Drawing'. " Tcch Report,
  55. (2008). Hidden Markov Model (HMM) Toolbox for Matlab. "
  56. (1994). Hidden Markov Model for Gesture Recognition, " Roboticý, Institue,
  57. Hidden Markov Models with Kernel Densit\ Estimation of Emission Probabilities and their Use in Activity Recognition, " doi
  58. (2008). http: //www. loizitech. com, " Last accessed
  59. Human Action Recognition
  60. (1999). Human Hand Modeling, Analysis and Animation in the Context of HC1, " doi
  61. iGesture: A General Gcsture Recognition Framework, " doi
  62. (1996). Image Processing, Anaývsis, atul. lfaclfinc Vision, vol. Third Edition: PWS, Brooks and Thornsoii Publishingo.
  63. (2005). Improving drag-and-drop on wall-size displays, "
  64. Interacting "'ith Large Displays from a Distance with Vision-Tracked Multi-Finger Gestural Input, " doi
  65. (2007). Interacting with the Computer Using Gaze Gestures, doi
  66. Kemel-Based Object Tracking. " doi
  67. Large Display Interactimi Using Video Avatar and Hand Gesture Recognition, " in ICIAR'04: SpringerVerlag doi
  68. (2007). Launches New Product Category: Surface Computing Comes to Life in Restaurants, Hotels, Retail Locations and Casino Resorts, " Microsoft PrcssPass,
  69. (1998). Leaming Deformable Shapes Models for Object Tracking, "
  70. (2008). Leaming Realistic Human Actions from Movies, " doi
  71. (2006). Local Descriptors for Spatio-temporal Recognition, " in Spatial Coherence for Visual Motion Analvsis. doi
  72. (2004). Machine Vision: doi
  73. (2008). Massey Hand Gesture Database, "
  74. (2008). Matlab Simulink Examples, http : //www. mathworks. com/p roducts/simuIink/ " , Last accessed
  75. Mean shift: a Robust Approach Toward Feature Space Analysis, " doi
  76. Model-Based Hand Tracking L"Slfh! an Unscented Kalman Filter, " doi
  77. Model-Based Hand Tracking Using a Hierarchical Bayesian Filter, " doi
  78. (2006). Multi-view Appearance-based 3D Hand Pose Estimation, " doi
  79. Multimodal Human Computer Interaction: A Survey. " doi
  80. (2004). Object Tracking Using CarriShift Algorithm and Multiple Quantized Feature Spaces, "
  81. (2005). Okapi-Chamfer Matching For Articulate Object Recognition, " doi
  82. On the Independence of Rotation Moment Invariants, " doi
  83. (2001). Open Source Computer Vision Library Reference Manual, " Intel Corporation,
  84. (2008). or ", Last accessed
  85. (2004). Orthogonal Moment Features for Use with Parametric and Non-Parametric Classifiers, " doi
  86. (2006). Parametric Hand Tracking for Recognition of Virtual Drawings, " doi
  87. (2000). Partitioned Sampling. Aniculated Objects. and interface-quality hand tracking, " doi
  88. (1999). Pattern Recognition in Grey Lc\-cl Iniaocs usim-1 Moment Based Invariant Features, "
  89. Pengyu Hong, Thomas Huang, " Gesture Modeling and Recognition Using Finite State Machines, " doi
  90. (2003). Performance evaluation metrics and statistics for positional tracker evaluation" doi
  91. (2004). Pointing Gesture Visual Recognition for Large Display, " doi
  92. (2008). Polhemus Laser handheld scanner, " http: //ww\ý, ý. polhemLis. com/, Last accessed
  93. (2001). Real-time 3D hand posture estimation based on 2D appearance retrieval using monocular camera, " doi
  94. (2004). Real-Time Detection and Understanding ofisolated Protruded Fingers, " doi
  95. (2002). Real-Time Fingertip Tracking and Gesture Recognition. " doi
  96. Real-Time Gesture Recognition by Learning and Selective Control of Visual Interest Points, " doi
  97. (2002). Real-time Gesture Recognition wsing Deterministic Boosting, " doi
  98. Real-Time Posture Analysis in a Crowd using Thermal Imaging, " doi
  99. Recognition of gestures in Arabic sign langua, -, c using neuro-fuzzy systems, " doi
  100. (2001). Recognition of Shapechanging Hand Gestures based on Switching Linear Model, " doi
  101. Recognition of' 1-solated Fingerspelling Gestures Using Depth Edges, " doi
  102. Representation and Synthesis of Beha-vlour us,,, (, Gaussian Mixtures "
  103. Retrieving Actions in Movies, " doi
  104. Review of Shape Representation and Description Techniques, " doi
  105. Robust Analysis of Feature Spaces: Color Iniage Segmentation, " doi
  106. (2006). Robust Computer Vision-Based Detection of Pinchitig, for Otle and Two-Handed Gesture Input, "
  107. Robust Spotting of Kcv Gestures from Whole Body Motion Sequence, " doi
  108. (2001). Shape Analysis and Classification. Theoiý'
  109. (2006). Sign Recognition using Depth Imacy i ,e Streams. " doi
  110. (2008). SignWriting The Sutton Fonts, littp: //www. siiYnwi-itiiiý-,. or! -,,, " Last accessed
  111. Simultaneous Localization and Recognition of Dynamic Hand Gestures, " doi
  112. (2008). Smartcanvas: a Gesture-Driven Intelligent Drawing Desk System, " doi
  113. (2003). Speech-Gesture Driven Multimodal Intafaces for Crisis Management, " doi
  114. (1998). Spotting Dynamic Hand Gestures in Video Image Sequences Using Hidden Markov Models, " doi
  115. Static Hand Posture Recognition Based on Okapi-Chamfer Matching, " in Real-Time Vision for Human-ComInitcr Interaction, doi
  116. (2008). Statistical Color Models with Application to Skin Detection, " doi
  117. (2007). Tangible friterfaces for Real-time 3D Virtual Environments, " doi
  118. Techniques for Online Gesture Recognition on Workstations. " doi
  119. (2005). TGS 2005: lwata speaks, "
  120. (1986). The Biological Foundatioll-S (ý/ Gestures: Motor and Semiotic Aspects. New Jersey London: Lawrence FribaLini associates,
  121. (2004). TouchLight: An Imaging Touch Screen and Display for Gesture-Based Interaction, " doi
  122. Towards 3D Hand Tracking using a Defori-nable Model, "
  123. (2004). Tracking Multiple Vehicles using Forground, Background and Motion Models, " Image and vision computing, doi
  124. Tracking of Multi-state Hand Models Using Particle Filtering and a Hierarchy of Multi-scale Image Features, " presented at doi
  125. (2006). Tracking using Dynamic Programming for Appearance-based Sign Language Recognition-" doi
  126. (2007). Traning Smulators and VR Per, pheral. s. " htti): //www. ml
  127. (2005). Two-Handed Gesture Recognition, "
  128. (2008). University of Waikato, Weka 3: Data Mining Software
  129. View- independent Recognition of Hand Posture". "
  130. (2003). Vision based Intcrpretation of Natural Sign Languages, "
  131. (1999). Vision-Based Gesture Recognition: A Review, " doi
  132. Visual Hand Tracking Using Nonparametric Belief doi
  133. Visual Interpretation of liand Gestures for Human-Computer Interaction: A Review, " doi
  134. Visual Pattern Recognition by Moment Invariants, " doi
  135. (1995). Visual Recognition of American Sign Language Using Hidden Markov Models, " doi
  136. (2002). What HMMs Can Do, " Tech Report. Universm of Washingtoti
  137. (2007). Wii are getting fitter: Retirement home installs computer game to k-ccp residents trim, "
  138. Xislonbased Hand Pose Estimation: A review, "
  139. Zemike Velocity %Ioments for Sequence- Býi. scd Description of Moving Features, " Image and VISiOn COMP11ting. N-01.24,
  140. Zemike Velocity Moments for the Descriptioll and Recognition of Moving Shapes, " doi

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.