Dynamic Hand Gesture Recognition Using Ultrasonic Sonar Sensors and Deep Learning

The space of hand gesture recognition using radar and sonar is dominated mostly by radar applications. In addition, the machine learning algorithms used by these systems are typically based on convolutional neural networks with some applications exploring the use of long short term memory networks. The goal of this study was to build and design a Sonar system that can classify hand gestures using a machine learning approach. Secondly, the study aims to compare convolutional neural networks to long short term memory networks as a means to classify hand gestures using sonar. A Doppler Sonar system was designed and built to be able to sense hand gestures. The Sonar system is a multi-static system containing one transmitter and three receivers. The sonar system can measure the Doppler frequency shifts caused by dynamic hand gestures. Since the system uses three receivers, three different Doppler frequency channels are measured. Three additional differential frequency channels are formed by computing the differences between the frequency of each of the receivers. These six channels are used as inputs to the deep learning models. Two different deep learning algorithms were used to classify the hand gestures; a Doppler biLSTM network [1] and a CNN [2]. Six basic hand gestures, two in each x- y- and z-axis, and two rotational hand gestures are recorded using both left and right hand at different distances. The gestures were also recorded using both left and right hands. Ten-Fold cross-validation is used to evaluate the networks' performance and classification accuracy. The LSTM was able to classify the six basic gestures with an accuracy of at least 96% but with the addition of the two rotational gestures, the accuracy drops to 47%. This result is acceptable since the basic gestures are more commonly used gestures than rotational gestures. The CNN was able to classify all the gestures with an accuracy of at least 98%. Additionally, The LSTM network is also able to classify separate left and right-hand gestures with an accuracy of 80% and The CNN with an accuracy of 83%. The study shows that CNN is the most widely used algorithm for hand gesture recognition as it can consistently classify gestures with various degrees of complexity. The study also shows that the LSTM network can also classify hand gestures with a high degree of accuracy. More experimentation, however, needs to be done in order to increase the complexity of recognisable gestures

Stay-At-Home Motor Rehabilitation: Optimizing Spatiotemporal Learning on Low-Cost Capacitive Sensor Arrays

Repeated, consistent, and precise gesture performance is a key part of recovery for stroke and other motor-impaired patients. Close professional supervision to these exercises is also essential to ensure proper neuromotor repair, which consumes a large amount of medical resources. Gesture recognition systems are emerging as stay-at-home solutions to this problem, but the best solutions are expensive, and the inexpensive solutions are not universal enough to tackle patient-to-patient variability. While many methods have been studied and implemented, the gesture recognition system designer does not have a strategy to effectively predict the right method to fit the needs of a patient. This thesis establishes such a strategy by outlining the strengths and weaknesses of several spatiotemporal learning architectures combined with deep learning, specifically when low-cost, low-resolution capacitive sensor arrays are used. This is done by testing the immunity and robustness of those architectures to the type of variability that is common among stroke patients, investigating select hyperparameters and their impact on the architectures’ training progressions, and comparing test performance in different applications and scenarios. The models analyzed here are trained on a mixture of high-quality, healthy gestures and personalized, imperfectly performed gestures using a low-cost recognition system

Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities

The vast proliferation of sensor devices and Internet of Things enables the applications of sensor-based activity recognition. However, there exist substantial challenges that could influence the performance of the recognition system in practical scenarios. Recently, as deep learning has demonstrated its effectiveness in many areas, plenty of deep methods have been investigated to address the challenges in activity recognition. In this study, we present a survey of the state-of-the-art deep learning methods for sensor-based human activity recognition. We first introduce the multi-modality of the sensory data and provide information for public datasets that can be used for evaluation in different challenge tasks. We then propose a new taxonomy to structure the deep methods by challenges. Challenges and challenge-related deep methods are summarized and analyzed to form an overview of the current research progress. At the end of this work, we discuss the open issues and provide some insights for future directions

Techniques for subtle mid-air gestural interaction using mmWave radar

Users need to be able to interact with mid-air gesture systems in ways that are efficient, precise, and socially acceptable. Subtle mid-air micro gestures can provide low-effort and discreet ways of interaction. This thesis contributes techniques for recognizing and utilizing subtle mid-air gestures with millimeter wave radars, a rapidly emerging sensing technology in human-computer interaction. The first contribution focused on the problem of addressing a system. By analyzing the frequency components of various hand motions, subtle activation gestures were identified which produced high-frequency signals through deliberate, rhythmic movements. A novel activation gesture recognition pipeline was then developed using frequency analysis to recognize these gestures and ignore incidental hand motions. Tested across three types of sensors, the pipeline demonstrated robust performance in recognizing subtle high-frequency activation gestures and producing zero false activations for broad hand motions. Further improvements were also explored to enhance robustness to reduce false activations during activities like typing, writing, and phone usage. The second contribution focused on recognition of subtle gestures from mmWave radar data using deep learning. A new dataset was developed, capturing the temporal dynamics and motion patterns of 10 different subtle gestures from 8 users with a mmWave radar. Multiple neural network architectures were trained and evaluated using the dataset, achieving a high recognition accuracy of 90%. The results demonstrated that hybrid neural networks combining convolutional and recurrent layers can effectively recognize subtle gestures from mmWave radar signals and generalize across different users. The final contribution progressed from offline evaluations to practical, real-time assessments. The neural network models were integrated into prototype applications that enabled real-time subtle gesture interactions for tasks such as selecting photos and adjusting media playback. A user study demonstrated significant improvements in task completion, accuracy, and user experience compared to traditional macro gestures. The findings suggest that subtle gestural interaction, enabled by mmWave radar sensors, signal processing, and deep learning, can significantly enhance usability of virtual interfaces

Reconocimiento biométrico basado en la interacción con dispositivos móviles

Tesis Doctoral inédita leída en la Universidad Autónoma de Madrid, Escuela Politécnica Superior, Departamento de Tecnología Electrónica y de las Comunicaciones. Fecha de Lectura: 24-05-2024The research described in this Thesis was carried out within the Biometrics and Data Pattern Analytics Laboratory - BiDA Lab at the Dept. of Tecnología Electrónica y de las Comunicaciones, Escuela Politécnica Superior, Universidad Autónoma de Madrid (from 2020 to 2024). This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant agreement No. 86031

Multimodal radar sensing for ambient assisted living

Data acquired from health and behavioural monitoring of daily life activities can be exploited to provide real-time medical and nursing service with affordable cost and higher efficiency. A variety of sensing technologies for this purpose have been developed and presented in the literature, for instance, wearable IMU (Inertial Measurement Unit) to measure acceleration and angular speed of the person, cameras to record the images or video sequence, PIR (Pyroelectric infrared) sensor to detect the presence of the person based on Pyroelectric Effect, and radar to estimate distance and radial velocity of the person. Each sensing technology has pros and cons, and may not be optimal for the tasks. It is possible to leverage the strength of all these sensors through information fusion in a multimodal fashion. The fusion can take place at three different levels, namely, i) signal level where commensurate data are combined, ii) feature level where feature vectors of different sensors are concatenated and iii) decision level where confidence level or prediction label of classifiers are used to generate a new output. For each level, there are different fusion algorithms, the key challenge here is mainly on choosing the best existing fusion algorithm and developing novel fusion algorithms that more suitable for the current application. The fundamental contribution of this thesis is therefore exploring possible information fusion between radar, primarily FMCW (Frequency Modulated Continuous Wave) radar, and wearable IMU, between distributed radar sensors, and between UWB impulse radar and pressure sensor array. The objective is to sense and classify daily activities patterns, gait styles and micro-gestures as well as producing early warnings of high-risk events such as falls. Initially, only “snapshot” activities (single activity within a short X-s measurement) have been collected and analysed for verifying the accuracy improvement due to information fusion. Then continuous activities (activities that are performed one after another with random duration and transitions) have been collected to simulate the real-world case scenario. To overcome the drawbacks of conventional sliding-window approach on continuous data, a Bi-LSTM (Bidirectional Long Short-Term Memory) network is proposed to identify the transitions of daily activities. Meanwhile, a hybrid fusion framework is presented to exploit the power of soft and hard fusion. Moreover, a trilateration-based signal level fusion method has been successfully applied on the range information of three UWB (Ultra-wideband) impulse radar and the results show comparable performance as using micro-Doppler signature, at the price of much less computation loads. For classifying ‘snapshot’ activities, fusion between radar and wearable shows approximately 12% accuracy improvement compared to using radar only, whereas for classifying continuous activities and gaits, our proposed hybrid fusion and trilateration-based signal level improves roughly 6.8% (before 89%, after 95.8%) and 7.3% (before 85.4%, after 92.7%), respectively

Hand gesture recognition and hand tracking for medical applications

Hand gestures are a mean of communication and a prevalent type of body language that conveys messages through different shapes constructed by palm and fingers. Hand gesture recognition (HGR) has been of interest in many research fields such as sign language translation, musical creation, and virtual environment control. There are also several studies on HGR for robotics, prosthetic, and rehabilitation applications. In this dissertation, the application of HGR for addressing two challenges in the medical field is presented. The first challenge is to develop a quantitative metric to improve rehabilitation of neurological conditions, with a focus on improvement in performing activities of daily living (ADL), while the second challenge is to develop ATTENTIVE, an automated and quantitative assessment system, to enhance a better evaluation of surgical skills proficiency. Many neurological conditions lead to motor impairment of upper extremity that includes muscle weakness, altered muscle tone, joint laxity, and impaired motor control. As a result, common activities such as reaching, picking up objects, and holding onto them are compromised. Therefore, such patients will experience disability in performing ADL such as eating, writing, performing housework, and so on. Several evaluation methods are commonly used to assess problems in performing ADL. Despite the wide application of these methods, all of them are subjective techniques, i.e. they are either questionnaires or qualitative scores assigned by a medical professional. We hypothesize that providing a more quantitative metric can enhance evaluation of the rehabilitation progress, and lead to a more efficient rehabilitation regimen tailored to the specific needs of each individual patient. Since the first step of developing a metric is to distinguish different ADL activities using hand gesture data, in this dissertation the focus is on classification of ADL tasks using hand gestures. Data analysis pipelines were developed to take in data, collected by the leap motion controller as well as the electromyography and inertial measurement unit sensors, from the lower arm during completion of certain ADL tasks. These pipelines output classification accuracies to distinguish the ADL tasks. Different preprocessing, feature extraction, and classification methods were tested on the data from healthy adults to detect the best structure and parameters for the proposed pipelines. The developed pipelines can be trained and their parameters can be tuned based on data from an intact-adult population. Then, The tuned pipelines can be set as the references. Subsequently, hand motion data from a neurological patient completing the same tasks in the same data collection setup can be fed into the reference pipelines to obtain the classification accuracies. The achieved accuracies indicate how close a patient’s hand motions and muscle activation are to the hand motions and muscle activation of the healthy population. This method enhances assessment of the overall performance of a patient in a quantitative fashion. In addition, the acquired confusion matrices provide insight into the patient’s performance in completing each individual task. The second section of this dissertation includes the design of ATTENTIVE; an evidence-supported, automated, robust, real-time, comprehensive, quantitative assessment system for evaluating proficiency in basic surgical skills. Since ATTENTIVE provides quantitative feedback, it can have a variety of applications in teaching surgical skills either in traditional settings or within incorporation of the augmented reality systems. As of now, the presence of an automated and quantitative assessment system to provide feedback on surgical tasks performance is lacking, and expert surgeon’s involvement is necessary to provide feedback to the surgical trainees. As a result, a trainee’s opportunities to receive feedback on one’s performance is restricted to the availability of an expert surgeon, which is limited due to pre-existing high workload of the expert surgeons. ATTENTIVE can eliminate such restriction that in turn may result in surgical trainees’ performance improvement and superior surgical outcomes over the long run. In this work, the idea and pipeline for developing ATTENTIVE are presented. Next, the apparatus and experimental setup and protocol to investigate the feasibility of ATTENTIVE were designed and built. Afterwards, data was collected from 65 participants completing four basic surgical tasks. The participants were students, residents, and expert surgeons in the fields of veterinary and human medicine. To benefit from both sensor-based and vision-based HGR methods for solving the problem in hand, Azure Kinect DK, Leap Motion Controller, and Myo armband were used to collect data from the lower arm of the participants. ATTENTIVE’s workflow consists of three major steps including separating the main task part from the preparation and cleanup after the task completion, classifying the input surgical task, and assigning a performance score to the input task. In this dissertation, many parts of these three steps are completed, and algorithms to complete the rest are determined and implemented to a vast extend. The details of the data analysis steps are beyond the scope of the abstract and are presented in the second section’s chapters. The last chapter of the current dissertation contains preliminary work on design and fabrication of a wearable device, named iBand, to collect biosignals and kinematic data from the lower arm. Different components of iBand have been selected and calibrated to read synchronized data from the lower arm, transfer them to a computer via Bluetooth, and save them as separate files. An easy-to-work user interface has been developed for iBand to enable user to save the data in the desired folder and with the desired file name. In addition, the user interface enhances a real-time data observation in which the user can choose the sensor from which the collected signal is displayed. Upon completion, the iBand can replace the discontinued Myo armband for research and daily life applications.Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo termsThe student, Hajar Sharif, accepted the attached license on 2022-11-24 at 09:15.The student, Hajar Sharif, submitted this Dissertation for approval on 2022-11-29 at 11:53.This Dissertation was approved for publication on 2022-12-02 at 10:54.DSpace SAF Submission Ingestion Package generated from Vireo submission #18631 on 2023-04-12 at 07:32:3