This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonMicro-gesture recognition has been widely research in recent years, in particular there
has been a great focus on 3D micro-gesture recognition which consists of classifying the
micro-gesture movements of the fingers for touch-less control applications. Holoscopic
3D imaging system mimics fly’s eye technique to capture true 3D scene which is enrich
in both texture and motion information. As a result, holoscopic 3D imaging system shall
be a suitable approach for robust recognition application. This PhD research focuses on
innovative 3D micro-gesture recognition based on holoscopic 3D system which delivers
robust and reliable performance with precision for 3D micro-gestures. Indeed this can
be applied to other wide range of applications such as Internet of things (IoT), AR/VR,
robotics and other touch-less interaction.
Due to lack of holoscopic 3D dataset, a comprehensive 3D micro-gesture dataset (HoMG)
includes both holoscopic 3D images and videos is prepared. It is a reasonable size holoscopic
3D dataset which is captured with different camera settings and conditions from
40 participants. Innovative 3D micro-gesture recognition is proposed based on 2D feature
extraction methods with basic classification methods, the recognition accuracy can reach
around 50.9%. For video-based data, the 3D feature extraction methods are achieved
66.7% recognition accuracy over 50.9% accuracy for micro-gesture images as the initial
investigation. HoMG database held a challenge in IEEE International automatic face and
gesture 2018, and 4 groups from the international research institutes joined the challenge
and contributed many new methods as further development where the proposed method
was published.
The holoscopic 3D dataset further enrich innovative micro-gesture 3D recognition system
is proposed and its performance is evaluated by carrying out like to like comparison
with state of the art methods. In addition, a fast and efficient pre-processing algorithm
for H3D images to extract the element images. Simplified viewpoint image extraction
method are presented. A pre-trained CNN model with the attention mechanics is implemented
based on VP image for the predicted probabilities of gesture. The proposed
approached is further improved using voting strategy. The proposed approach achieves
87% accuracy, which outperform all existing state of the art methods on the image-based
database. Advanced 3D micro-gesture recognition is investigated based on sequence video database,
the end-to-end model has been used on effective H3D based micro-gesture recognition
system. For front-end network, there are two method of traditional viewpoint image
extraction and novel pseudo viewpoint image extraction have been used and evaluated.
The pseudo viewpoint (PVP) front-end has been created, which used to deep learning
networks understanding the implied 3D information of H3D imaging system. The viewpoint
(VP) front-end follows the traditional H3D image method to extract and reconstruct
the multi-viewpoint images. Both front-end have been feed in four popular advanced
deep networks using for learning and classification. This experiments evaluated the performance
of 2D/3D convolutional, mixing 2D and 3D convolutional and LSTM on the
HoMG video database, which is beneficial to H3D imaging system using deep learning
network. Finally, in order to obtain the high accuracies, the majority voting has been applied
for further improve. The final results show that the performance is not only better
than the traditional methods, but also superior to the existing deep learning based approaches,
which clearly demonstrates the effectiveness of the proposed approach