8 research outputs found
Recommended from our members
Fingers micro-gesture recognition based on holoscopic 3D imaging system
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonMicro-gesture recognition has been widely research in recent years, in particular there
has been a great focus on 3D micro-gesture recognition which consists of classifying the
micro-gesture movements of the fingers for touch-less control applications. Holoscopic
3D imaging system mimics fly’s eye technique to capture true 3D scene which is enrich
in both texture and motion information. As a result, holoscopic 3D imaging system shall
be a suitable approach for robust recognition application. This PhD research focuses on
innovative 3D micro-gesture recognition based on holoscopic 3D system which delivers
robust and reliable performance with precision for 3D micro-gestures. Indeed this can
be applied to other wide range of applications such as Internet of things (IoT), AR/VR,
robotics and other touch-less interaction.
Due to lack of holoscopic 3D dataset, a comprehensive 3D micro-gesture dataset (HoMG)
includes both holoscopic 3D images and videos is prepared. It is a reasonable size holoscopic
3D dataset which is captured with different camera settings and conditions from
40 participants. Innovative 3D micro-gesture recognition is proposed based on 2D feature
extraction methods with basic classification methods, the recognition accuracy can reach
around 50.9%. For video-based data, the 3D feature extraction methods are achieved
66.7% recognition accuracy over 50.9% accuracy for micro-gesture images as the initial
investigation. HoMG database held a challenge in IEEE International automatic face and
gesture 2018, and 4 groups from the international research institutes joined the challenge
and contributed many new methods as further development where the proposed method
was published.
The holoscopic 3D dataset further enrich innovative micro-gesture 3D recognition system
is proposed and its performance is evaluated by carrying out like to like comparison
with state of the art methods. In addition, a fast and efficient pre-processing algorithm
for H3D images to extract the element images. Simplified viewpoint image extraction
method are presented. A pre-trained CNN model with the attention mechanics is implemented
based on VP image for the predicted probabilities of gesture. The proposed
approached is further improved using voting strategy. The proposed approach achieves
87% accuracy, which outperform all existing state of the art methods on the image-based
database. Advanced 3D micro-gesture recognition is investigated based on sequence video database,
the end-to-end model has been used on effective H3D based micro-gesture recognition
system. For front-end network, there are two method of traditional viewpoint image
extraction and novel pseudo viewpoint image extraction have been used and evaluated.
The pseudo viewpoint (PVP) front-end has been created, which used to deep learning
networks understanding the implied 3D information of H3D imaging system. The viewpoint
(VP) front-end follows the traditional H3D image method to extract and reconstruct
the multi-viewpoint images. Both front-end have been feed in four popular advanced
deep networks using for learning and classification. This experiments evaluated the performance
of 2D/3D convolutional, mixing 2D and 3D convolutional and LSTM on the
HoMG video database, which is beneficial to H3D imaging system using deep learning
network. Finally, in order to obtain the high accuracies, the majority voting has been applied
for further improve. The final results show that the performance is not only better
than the traditional methods, but also superior to the existing deep learning based approaches,
which clearly demonstrates the effectiveness of the proposed approach
Recognition of Holoscopic 3D Video Hand Gesture Using Convolutional Neural Networks
Copyright © 2020 by the authors. The convolutional neural network (CNN) algorithm is one of the efficient techniques to recognize hand gestures. In human–computer interaction, a human gesture is a non-verbal communication mode, as users communicate with a computer via input devices. In this article, 3D micro hand gesture recognition disparity experiments are proposed using CNN. This study includes twelve 3D micro hand motions recorded for three different subjects. The system is validated by an experiment that is implemented on twenty different subjects of different ages. The results are analysed and evaluated based on execution time, training, testing, sensitivity, specificity, positive and negative predictive value, and likelihood. The CNN training results show an accuracy as high as 100%, which present superior performance in all factors. On the other hand, the validation results average about 99% accuracy. The CNN algorithm has proven to be the most accurate classification tool for micro gesture recognition.Imam Abdulrahman bin Faisal Universit
Recommended from our members
Hand gesture recognition using deep learning neural networks
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonHuman Computer Interaction (HCI) is a broad field involving different types of interactions including gestures. Gesture recognition concerns non-verbal motions used as a means of communication in HCI. A system may be utilised to identify human gestures to convey information for device control. This represents a significant field within HCI involving device interfaces and users. The aim of gesture recognition is to record gestures that are formed in a certain way and then detected by a device such as a camera. Hand gestures can be used as a form of communication for many different applications. It may be used by people who possess different disabilities, including those with hearing-impairments, speech impairments and stroke patients, to communicate and fulfil their basic needs.
Various studies have previously been conducted relating to hand gestures. Some studies proposed different techniques to implement the hand gesture experiments. For image processing there are multiple tools to extract features of images, as well as Artificial Intelligence which has varied classifiers to classify different types of data. 2D and 3D hand gestures request an effective algorithm to extract images and classify various mini gestures and movements. This research discusses this issue using different algorithms. To detect 2D or 3D hand gestures, this research proposed image processing tools such as Wavelet Transforms and Empirical Mode Decomposition to extract image features. The Artificial Neural Network (ANN) classifier which used to train and classify data besides Convolutional Neural Networks (CNN). These methods were examined in terms of multiple parameters such as execution time, accuracy, sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood, negative likelihood, receiver operating characteristic, area under ROC curve and root mean square. This research discusses four original contributions in the field of hand gestures. The first contribution is an implementation of two experiments using 2D hand gesture video where ten different gestures are detected in short and long distances using an iPhone 6 Plus with 4K resolution. The experiments are performed using WT and EMD for feature extraction while ANN and CNN for classification. The second contribution comprises 3D hand gesture video experiments where twelve gestures are recorded using holoscopic imaging system camera. The third contribution pertains experimental work carried out to detect seven common hand gestures. Finally, disparity experiments were performed using the left and the right 3D hand gesture videos to discover disparities. The results of comparison show the accuracy results of CNN being 100% compared to other techniques. CNN is clearly the most appropriate method to be used in a hand gesture system.Imam Abdulrahman bin Faisal Universit
Recommended from our members
Holoscopic 3D perception for autonomous vehicles
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonAutonomous mobile platforms are going to be huge part of the future transportation and autonomous navigation is the critical part of autonomous platforms. An autonomous mobile platform navigates the vehicle by perceiving the environment through the sensors mount on the vehicle, and acting on the data it receives from these sensors by making sense of the environmental and surroundings. As a result, an autonomous mobile platform
consists of localisation aka positioning and path planning. Both of them require very accurate sensor measurements. In terms of accuracy, sensor can generally be divided into two groups (a) High accuracy sensors like the state-of-the-art in LiDAR and vision sensors e.g. mobile-eye sensor. (b) Low accuracy sensors whereas GPS (accurate within 2-10 metres) sensor and IMU (suffering from drifts) could be fused to improve the other method of positioning. These are expensive process due to offline map creation. To deal with low
accuracy sensors, researchers normally use very complex models, which again run into performance reliability and consistency issue. Furthermore, it is common believe, that when navigating autonomously, perception and
situation cognisance is an important component to navigate safely and there have been a huge research on AI enabled perception such as Mobile Eye and Tesla car which uses 2D cameras for its perception. In this research, an innovative method is proposed to use rich vision sensor holoscopic 3D camera for environment perception with artificial intelligent algorithms to observe road objects and learn their 3D behavioural for reliable detection and recognition. The sensor provides rich information - 3D cubic visual information about the
environment including the very valuable “depth information” to imitate third coordinate of real world. To learn the objects, different AI algorithms are studied and in particular deep learning model is proposed that provides a reasonable good result. To evaluate the innovative holoscopic 3D sensor, we applied to face recognition challenge under different face expression where 2D images are considered to fail. However the holoscopic 3D sensor outperform and delivered outstanding performance by recognising faces under different expression by only training on the neutral face using a simple AI algorithm. Then we design and develop holoscopic perception database of 200000 frames for autonomous car. The experimental result has shown a promising result that AI algorithm, particularly deep learning algorithm learns effectively from holoscopic 3D content compared to traditional 2D images even those DL models which were designed for visual features yet holoscopic 3D images contain motion data which shall be exploited
Recommended from our members
Depth Estimation from a Single Holoscopic 3D Image and Image Up-sampling with Deep-learning
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University London3D depth information is widely utilized in industries such as security, autonomous vehicles, robotics, 3D printing, AR/VR entertainment, cinematography and medical science. However, state-of-the-art imaging and 3D depth-sensing technologies are rather complicated or expensive and still lack scalability and interoperability. The research identified, entails the development of an innovative technique for reliable and efficient 3D depth estimation that deliver better accuracy. The proposed (1) multilayer Holoscopic 3D encoding technique reduces the computational cost of extracting viewpoint images from complex structured Holoscopic 3D data by 95%, by using labelled multilayer elemental images. It also addresses misplacement of elemental image pixels due to lens distortion error. The multilayer Holoscopic 3D encoding computing efficiency leads to the implementation of real-time 3D depth-dependent applications. Also, (2) an innovative approach of a deep learning-based single image super-resolution framework is developed and evaluated. It identified that learning-based image up-sampling techniques could be used regardless of inadequate 3D training data, as 2D training data can yield the same results.
(3) The research is extended further by implementation of an H3D depth disparity -based framework, where a Holoscopic content adaptation technique for extracting semi-segmented stereo viewpoint image is introduced, and the design of a smart 3D depth mapping technique is proposed. Particularly, it provides a somewhat accurate 3D depth estimation from H3D images in near real-time. Holoscopic 3D image has thousands of perspective elemental images from omnidirectional viewpoint images and (4) a novel 3D depth estimation technique is developed to estimates 3D depth information directly from a single Holoscopic 3D image without the loss of any angular information and the introduction of unwanted artefacts. The proposed 3D depth measurement techniques are computationally efficient and robust with high accuracy; these can be incorporated in real-time applications of autonomous vehicles, security and AR/VR for real-time interaction
Towards key-frame extraction methods for 3D video: a review
The increasing rate of creation and use of 3D video content leads to a pressing need for methods capable of lowering
the cost of 3D video searching, browsing and indexing operations, with improved content selection performance.
Video summarisation methods specifically tailored for 3D video content fulfil these requirements. This paper presents
a review of the state-of-the-art of a crucial component of 3D video summarisation algorithms: the key-frame
extraction methods. The methods reviewed cover 3D video key-frame extraction as well as shot boundary detection
methods specific for use in 3D video. The performance metrics used to evaluate the key-frame extraction methods
and the summaries derived from those key-frames are presented and discussed. The applications of these methods
are also presented and discussed, followed by an exposition about current research challenges on 3D video
summarisation methods
Recommended from our members
ReSCon '12, Research Student Conference: Book of Abstracts
The fifth SED Research Student Conference (ReSCon2012) was hosted over three days, 18-20 June 2012, in the Hamilton Centre at Brunel University. The conference consisted of 130 oral and 70 poster presentations, based on the high quality and diverse research being conducted within the School of Engineering and Design by postgraduate research students. The conference is held annually, and ReSCon plays a key role in contributing to research and innovations within the School
Exploration of smart infrastructure for drivers of autonomous vehicles
The connection between vehicles and infrastructure is an integral part of providing autonomous vehicles information about the environment.
Autonomous vehicles need to be safe and users need to trust their driving decision. When smart infrastructure information is integrated into the vehicle, the driver needs to be informed in an understandable manner what the smart infrastructure detected.
Nevertheless, interactions that benefit from smart infrastructure have not been the focus of research, leading to knowledge gaps in the integration of smart infrastructure information in the vehicle. For example, it is unclear, how the information from two complex systems can be presented, and if decisions are made, how these can be explained.
Enriching the data of vehicles with information from the infrastructure opens unexplored opportunities.
Smart infrastructure provides vehicles with information to predict traffic flow and traffic events.
Additionally, it has information about traffic events in several kilometers distance and thus enables a look ahead on a traffic situation, which is not in the immediate view of drivers.
We argue that this smart infrastructure information can be used to enhance the driving experience. To achieve this, we explore designing novel interactions, providing warnings and visualizations about information that is out of the view of the driver, and offering explanations for the cause of changed driving behavior of the vehicle.
This thesis focuses on exploring the possibilities of smart infrastructure information with a focus on the highway.
The first part establishes a design space for 3D in-car augmented reality applications that profit from smart infrastructure information. Through the input of two focus groups and a literature review, use cases are investigated that can be introduced in the vehicle's interaction interface which, among others, rely on environment information. From those, a design space that can be used to design novel in-car applications is derived.
The second part explores out-of-view visualizations before and during take over requests to increase situation awareness. With three studies, different visualizations for out-of-view information are implemented in 2D, stereoscopic 3D, and augmented reality. Our results show that visualizations improve the situation awareness about critical events in larger distances during take over request situations.
In the third part, explanations are designed for situations in which the vehicle drives unexpectedly due to unknown reasons. Since smart infrastructure could provide connected vehicles with out-of-view or cloud information, the driving maneuver of the vehicle might remain unclear to the driver. Therefore, we explore the needs of drivers in those situations and derive design recommendations for an interface which displays the cause for the unexpected driving behavior.
This thesis answers questions about the integration of environment information in vehicles'.
Three important aspects are explored, which are essential to consider when implementing use cases with smart infrastructure in mind. It enables to design novel interactions, provides insights on how out-of-view visualizations can improve the drivers' situation awareness and explores unexpected driving situations and the design of explanations for them.
Overall, we have shown how infrastructure and connected vehicle information can be introduced in vehicles' user interface and how new technology such as augmented reality glasses can be used to improve the driver's perception of the environment.Autonome Fahrzeuge werden immer mehr in den alltäglichen Verkehr integriert. Die Verbindung von Fahrzeugen mit der Infrastruktur ist ein wesentlicher Bestandteil der Bereitstellung von Umgebungsinformationen in autonome Fahrzeugen.
Die Erweiterung der Fahrzeugdaten mit Informationen der Infrastruktur eröffnet ungeahnte Möglichkeiten.
Intelligente Infrastruktur übermittelt verbundenen Fahrzeugen Informationen über den prädizierten Verkehrsfluss und Verkehrsereignisse.
Zusätzlich können Verkehrsgeschehen in mehreren Kilometern Entfernung übermittelt werden, wodurch ein Vorausblick auf einen Bereich ermöglicht wird, der für den Fahrer nicht unmittelbar sichtbar ist.
Mit dieser Dissertation wird gezeigt, dass Informationen der intelligenten Infrastruktur benutzt werden können, um das Fahrerlebnis zu verbessern. Dies kann erreicht werden, indem innovative Interaktionen gestaltet werden, Warnungen und Visualisierungen über Geschehnisse außerhalb des Sichtfelds des Fahrers vermittelt werden und indem Erklärungen über den Grund eines veränderten Fahrzeugverhaltens untersucht werden.
Interaktionen, welche von intelligenter Infrastruktur profitieren, waren jedoch bisher nicht im Fokus der Forschung. Dies führt zu Wissenslücken bezüglich der Integration von intelligenter Infrastruktur in das Fahrzeug.
Diese Dissertation exploriert die Möglichkeiten intelligenter Infrastruktur, mit einem Fokus auf die Autobahn.
Der erste Teil erstellt einen Design Space für Anwendungen von augmentierter Realität (AR) in 3D innerhalb des Autos, die unter anderem von Informationen intelligenter Infrastruktur profitieren. Durch das Ergebnis mehrerer Studien werden Anwendungsfälle in einem Katalog gesammelt, welche in die Interaktionsschnittstelle des Autos einfließen können. Diese Anwendungsfälle bauen unter anderem auf Umgebungsinformationen. Aufgrund dieser Anwendungen wird der Design Space entwickelt, mit Hilfe dessen neuartige Anwendungen für den Fahrzeuginnenraum entwickelt werden können.
Der zweite Teil exploriert Visualisierungen für Verkehrssituationen, die außerhalb des Sichtfelds des Fahrers sind. Es wird untersucht, ob durch diese Visualisierungen der Fahrer besser auf ein potentielles Übernahmeszenario vorbereitet wird. Durch mehrere Studien wurden verschiedene Visualisierungen in 2D, stereoskopisches 3D und augmentierter Realität implementiert, die Szenen außerhalb des Sichtfelds des Fahrers darstellen. Diese Visualisierungen verbessern das Situationsbewusstsein über kritische Szenarien in einiger Entfernung während eines Übernahmeszenarios.
Im dritten Teil werden Erklärungen für Situationen gestaltet, in welchen das Fahrzeug ein unerwartetes Fahrmanöver ausführt. Der Grund des Fahrmanövers ist dem Fahrer dabei unbekannt. Mit intelligenter Infrastruktur verbundene Fahrzeuge erhalten Informationen, die außerhalb des Sichtfelds des Fahrers liegen oder von der Cloud bereit gestellt werden. Dadurch könnte der Grund für das unerwartete Fahrverhalten unklar für den Fahrer sein.
Daher werden die Bedürfnisse des Fahrers in diesen Situationen erforscht und Empfehlungen für die Gestaltung einer Schnittstelle, die Erklärungen für das unerwartete Fahrverhalten zur Verfügung stellt, abgeleitet.
Zusammenfassend wird gezeigt wie Daten der Infrastruktur und Informationen von verbundenen Fahrzeugen in die Nutzerschnittstelle des Fahrzeugs implementiert werden können. Zudem wird aufgezeigt, wie innovative Technologien wie AR Brillen, die Wahrnehmung der Umgebung des Fahrers verbessern können.
Durch diese Dissertation werden Fragen über Anwendungsfälle für die Integration von Umgebungsinformationen in Fahrzeugen beantwortet.
Drei wichtige Themengebiete wurden untersucht, welche bei der Betrachtung von Anwendungsfällen der intelligenten Infrastruktur essentiell sind. Durch diese Arbeit wird die Gestaltung innovativer Interaktionen ermöglicht, Einblicke in Visualisierungen von Informationen außerhalb des Sichtfelds des Fahrers gegeben und es wird untersucht, wie Erklärungen für unerwartete Fahrsituationen gestaltet werden können