Fingers micro-gesture recognition based on holoscopic 3D imaging system

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonMicro-gesture recognition has been widely research in recent years, in particular there has been a great focus on 3D micro-gesture recognition which consists of classifying the micro-gesture movements of the fingers for touch-less control applications. Holoscopic 3D imaging system mimics fly’s eye technique to capture true 3D scene which is enrich in both texture and motion information. As a result, holoscopic 3D imaging system shall be a suitable approach for robust recognition application. This PhD research focuses on innovative 3D micro-gesture recognition based on holoscopic 3D system which delivers robust and reliable performance with precision for 3D micro-gestures. Indeed this can be applied to other wide range of applications such as Internet of things (IoT), AR/VR, robotics and other touch-less interaction. Due to lack of holoscopic 3D dataset, a comprehensive 3D micro-gesture dataset (HoMG) includes both holoscopic 3D images and videos is prepared. It is a reasonable size holoscopic 3D dataset which is captured with different camera settings and conditions from 40 participants. Innovative 3D micro-gesture recognition is proposed based on 2D feature extraction methods with basic classification methods, the recognition accuracy can reach around 50.9%. For video-based data, the 3D feature extraction methods are achieved 66.7% recognition accuracy over 50.9% accuracy for micro-gesture images as the initial investigation. HoMG database held a challenge in IEEE International automatic face and gesture 2018, and 4 groups from the international research institutes joined the challenge and contributed many new methods as further development where the proposed method was published. The holoscopic 3D dataset further enrich innovative micro-gesture 3D recognition system is proposed and its performance is evaluated by carrying out like to like comparison with state of the art methods. In addition, a fast and efficient pre-processing algorithm for H3D images to extract the element images. Simplified viewpoint image extraction method are presented. A pre-trained CNN model with the attention mechanics is implemented based on VP image for the predicted probabilities of gesture. The proposed approached is further improved using voting strategy. The proposed approach achieves 87% accuracy, which outperform all existing state of the art methods on the image-based database. Advanced 3D micro-gesture recognition is investigated based on sequence video database, the end-to-end model has been used on effective H3D based micro-gesture recognition system. For front-end network, there are two method of traditional viewpoint image extraction and novel pseudo viewpoint image extraction have been used and evaluated. The pseudo viewpoint (PVP) front-end has been created, which used to deep learning networks understanding the implied 3D information of H3D imaging system. The viewpoint (VP) front-end follows the traditional H3D image method to extract and reconstruct the multi-viewpoint images. Both front-end have been feed in four popular advanced deep networks using for learning and classification. This experiments evaluated the performance of 2D/3D convolutional, mixing 2D and 3D convolutional and LSTM on the HoMG video database, which is beneficial to H3D imaging system using deep learning network. Finally, in order to obtain the high accuracies, the majority voting has been applied for further improve. The final results show that the performance is not only better than the traditional methods, but also superior to the existing deep learning based approaches, which clearly demonstrates the effectiveness of the proposed approach

Attention Based Residual Network for Micro-Gesture Recognition

Finger micro-gesture recognition is increasingly become an important part of human-computer interaction (HCI) in applications of augmented reality (AR) and virtual reality (VR) technologies. To push the boundary of microgesture recognition, a novel Holoscopic 3D Micro-Gesture Database (HoMG) was established for research purpose. HoMG has an image subset and a video subset. This paper is to demonstrate the result achieved on the image subset for Holoscopic Micro-Gesture Recognition Challenge 2018 (HoMGR 2018). The proposed method utilized the state-of-the-art residual network with an attention-involved design. In every block of the network, an attention branch is added to the output of the last convolution layer. The attention branch is designed to spotlight the finger micro-gesture and reduce the noise introduced from the wrist and background. With an extensive analysis on HoMG, the proposed model achieved a recognition accuracy of 80.5% on the validation set and 82.1% on the testing set

Recognition of Holoscopic 3D Video Hand Gesture Using Convolutional Neural Networks

Copyright © 2020 by the authors. The convolutional neural network (CNN) algorithm is one of the efficient techniques to recognize hand gestures. In human–computer interaction, a human gesture is a non-verbal communication mode, as users communicate with a computer via input devices. In this article, 3D micro hand gesture recognition disparity experiments are proposed using CNN. This study includes twelve 3D micro hand motions recorded for three different subjects. The system is validated by an experiment that is implemented on twenty different subjects of different ages. The results are analysed and evaluated based on execution time, training, testing, sensitivity, specificity, positive and negative predictive value, and likelihood. The CNN training results show an accuracy as high as 100%, which present superior performance in all factors. On the other hand, the validation results average about 99% accuracy. The CNN algorithm has proven to be the most accurate classification tool for micro gesture recognition.Imam Abdulrahman bin Faisal Universit

Hand gesture recognition using deep learning neural networks

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonHuman Computer Interaction (HCI) is a broad field involving different types of interactions including gestures. Gesture recognition concerns non-verbal motions used as a means of communication in HCI. A system may be utilised to identify human gestures to convey information for device control. This represents a significant field within HCI involving device interfaces and users. The aim of gesture recognition is to record gestures that are formed in a certain way and then detected by a device such as a camera. Hand gestures can be used as a form of communication for many different applications. It may be used by people who possess different disabilities, including those with hearing-impairments, speech impairments and stroke patients, to communicate and fulfil their basic needs. Various studies have previously been conducted relating to hand gestures. Some studies proposed different techniques to implement the hand gesture experiments. For image processing there are multiple tools to extract features of images, as well as Artificial Intelligence which has varied classifiers to classify different types of data. 2D and 3D hand gestures request an effective algorithm to extract images and classify various mini gestures and movements. This research discusses this issue using different algorithms. To detect 2D or 3D hand gestures, this research proposed image processing tools such as Wavelet Transforms and Empirical Mode Decomposition to extract image features. The Artificial Neural Network (ANN) classifier which used to train and classify data besides Convolutional Neural Networks (CNN). These methods were examined in terms of multiple parameters such as execution time, accuracy, sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood, negative likelihood, receiver operating characteristic, area under ROC curve and root mean square. This research discusses four original contributions in the field of hand gestures. The first contribution is an implementation of two experiments using 2D hand gesture video where ten different gestures are detected in short and long distances using an iPhone 6 Plus with 4K resolution. The experiments are performed using WT and EMD for feature extraction while ANN and CNN for classification. The second contribution comprises 3D hand gesture video experiments where twelve gestures are recorded using holoscopic imaging system camera. The third contribution pertains experimental work carried out to detect seven common hand gestures. Finally, disparity experiments were performed using the left and the right 3D hand gesture videos to discover disparities. The results of comparison show the accuracy results of CNN being 100% compared to other techniques. CNN is clearly the most appropriate method to be used in a hand gesture system.Imam Abdulrahman bin Faisal Universit

ReSCon '12, Research Student Conference: Book of Abstracts

The fifth SED Research Student Conference (ReSCon2012) was hosted over three days, 18-20 June 2012, in the Hamilton Centre at Brunel University. The conference consisted of 130 oral and 70 poster presentations, based on the high quality and diverse research being conducted within the School of Engineering and Design by postgraduate research students. The conference is held annually, and ReSCon plays a key role in contributing to research and innovations within the School

Holoscopic 3D Micro-Gesture Database for Wearable Device Interaction

With the rapid development of augmented reality (AR) and virtual reality (VR) technology, human-computer interaction (HCI) has been greatly improved for gaming interaction of AR and VR control. The finger micro-gesture is one of the important interactive methods for HCI applications such as in the Google Soli and Microsoft Kinect projects. However, the progress in this research is slow due to the lack of high quality public available database. In this paper, holoscopic 3D camera is used to capture high quality micro-gesture images and a new unique holoscopic 3D micro-gesture (HoMG) database is produced. The principle of the holoscopic 3D camera is based on the flys viewing system to see the objects. HoMG database recorded the image sequence of 3 conventional gestures from 40 participants under different settings and conditions. For the purpose of micro-gesture recognition, HoMG has a video subset with 960 videos and a still image subset with 30635 images. Initial micro-gesture recognition on both subsets has been conducted using traditional 2D image and video features and popular classifiers and some encouraging performance has been achieved. The database will be available for the research communities and speed up the research in this area.NVIDIA Corporatio