ULTRASOUND BASED GESTURE RECOGNITION

ABSTRACT In this study, we explore the possibility of recognizing hand gestures using ultrasonic depth imaging. The ultrasonic device consists of a single piezoelectric transducer and an 8 -element microphone array. Using carefully designed transmit pulse, and a combination of beamforming, matched filtering, and cross-correlation methods, we construct ultrasound images with depth and intensity pixels. Thereafter, we use a combined Convolutional (CNN) and Long Short-Term Memory (LSTM) network to recognize gestures from the ultrasound images. We report gesture recognition accuracies in the range 64.5-96.9%, based on the number of gestures to be recognized, and show that ultrasound sensors have the potential to become low power, low computation, and low cost alternatives to existing optical sensors

Intelligent ultrasound hand gesture recognition system

With the booming development of technology, hand gesture recognition has become a hotspot in Human-Computer Interaction (HCI) systems. Ultrasound hand gesture recognition is an innovative method that has attracted ample interest due to its strong real-time performance, low cost, large field of view, and illumination independence. Well-investigated HCI applications include external digital pens, game controllers on smart mobile devices, and web browser control on laptops. This thesis probes gesture recognition systems on multiple platforms to study the behavior of system performance with various gesture features. Focused on this topic, the contributions of this thesis can be summarized from the perspectives of smartphone acoustic field and hand model simulation, real-time gesture recognition on smart devices with speed categorization algorithm, fast reaction gesture recognition based on temporal neural networks, and angle of arrival-based gesture recognition system. Firstly, a novel pressure-acoustic simulation model is developed to examine its potential for use in acoustic gesture recognition. The simulation model is creating a new system for acoustic verification, which uses simulations mimicking real-world sound elements to replicate a sound pressure environment as authentically as possible. This system is fine-tuned through sensitivity tests within the simulation and validate with real-world measurements. Following this, the study constructs novel simulations for acoustic applications, informed by the verified acoustic field distribution, to assess their effectiveness in specific devices. Furthermore, a simulation focused on understanding the effects of the placement of sound devices and hand-reflected sound waves is properly designed. Moreover, a feasibility test on phase control modification is conducted, revealing the practical applications and boundaries of this model. Mobility and system accuracy are two significant factors that determine gesture recognition performance. As smartphones have high-quality acoustic devices for developing gesture recognition, to achieve a portable gesture recognition system with high accuracy, novel algorithms were developed to distinguish gestures using smartphone built-in speakers and microphones. The proposed system adopts Short-Time-Fourier-Transform (STFT) and machine learning to capture hand movement and determine gestures by the pretrained neural network. To differentiate gesture speeds, a specific neural network was designed and set as part of the classification algorithm. The final accuracy rate achieves 96% among nine gestures and three speed levels. The proposed algorithms were evaluated comparatively through algorithm comparison, and the accuracy outperformed state-of-the-art systems. Furthermore, a fast reaction gesture recognition based on temporal neural networks was designed. Traditional ultrasound gesture recognition adopts convolutional neural networks that have flaws in terms of response time and discontinuous operation. Besides, overlap intervals in network processing cause cross-frame failures that greatly reduce system performance. To mitigate these problems, a novel fast reaction gesture recognition system that slices signals in short time intervals was designed. The proposed system adopted a novel convolutional recurrent neural network (CRNN) that calculates gesture features in a short time and combines features over time. The results showed the reaction time significantly reduced from 1s to 0.2s, and accuracy improved to 100% for six gestures. Lastly, an acoustic sensor array was built to investigate the angle information of performed gestures. The direction of a gesture is a significant feature for gesture classification, which enables the same gesture in different directions to represent different actions. Previous studies mainly focused on types of gestures and analyzing approaches (e.g., Doppler Effect and channel impulse response, etc.), while the direction of gestures was not extensively studied. An acoustic gesture recognition system based on both speed information and gesture direction was developed. The system achieved 94.9% accuracy among ten different gestures from two directions. The proposed system was evaluated comparatively through numerical neural network structures, and the results confirmed that incorporating additional angle information improved the system's performance. In summary, the work presented in this thesis validates the feasibility of recognizing hand gestures using remote ultrasonic sensing across multiple platforms. The acoustic simulation explores the smartphone acoustic field distribution and response results in the context of hand gesture recognition applications. The smartphone gesture recognition system demonstrates the accuracy of recognition through ultrasound signals and conducts an analysis of classification speed. The fast reaction system proposes a more optimized solution to address the cross-frame issue using temporal neural networks, reducing the response latency to 0.2s. The speed and angle-based system provides an additional feature for gesture recognition. The established work will accelerate the development of intelligent hand gesture recognition, enrich the available gesture features, and contribute to further research in various gestures and application scenarios

Proficient ambient intelligence based on the wisely selection of data sources in heterogeneous environments

Nowadays society tends to be more and more technologically evolved, with that a lot of personal information is shared with computational systems through online records; microphones that perform voice recognition in which it is processed remotely; application of filters on person portraits or recognition of the age at which the image is saved on a server to improve the standard of an Artificial Intelligence (AI). The devices we currently interact with end up compiling information about our consumption habits, routines, collected photographs, videos or personal content. Sometimes exposing our privacy to third parties, it is not noticeable where that data is been saved, or that is being stored. And if we could have a simpler device to perform some tasks, in which there was no camera, microphone or storage. No personal data will be processed locally or remotely, so the regulation European Union (EU) law on data protection and privacy, General Data Protection Regulation (GDPR), will not affect this kind of device. Furthermore, if a device of this nature was located in a public place, there would be no need for any requirement, consent or authorization to the intervening users or only those who approach the device, because would not differentiate the user or even recognize him, since the collection of gesture commands is performed through non-evasive sensors. Recently the world was surprised by the cases of a new coronavirus which caused a pandemic on a planetary scale. New rules had to be applied, to the way we interact with objects in order not to spread the disease. It was created a prototype discriminated in this study, using a grid of ultrasonic sensors, capable of recognize trained gestures. This interaction can be made from a distance of the prototype, and do not require contact, in which it solves a human interaction with the computer avoiding that the user has to use the touch on a component, thus avoiding the contagion of Corona virus disease of 2019 (COVID-19). The early stage results evidence that the proposed system is suitable to create a new input type of Human Interface Device (HID), and may replace devices as a remote control of television, to a new way to interact with information panels in public places, like a shopping mall, airports, train and bus stations.Hoje em dia a sociedade tende a estar cada vez mais evoluída tecnologicamente, consequentemente muitas informações pessoais são compartilhadas com sistemas computacionais através de registros online; microfones que realizam reconhecimento de voz, em que é processado remotamente; aplicação de filtros em retratos de pessoas ou reconhecimento da idade, em que a foto é guardada em um servidor para melhorar o padrão de uma Inteligência Artificial (AI). Atualmente os dispositivos com os quais interagimos acabam compilando informações sobre os nossos hábitos de consumo, rotinas, fotos e vídeos armazenados ou conteúdos pessoais. Por vezes, expondo nossa privacidade a terceiros, não é perceptível onde os dados foram guardados ou onde estão a ser armazenados. E se pudéssemos ter um dispositivo mais simples para realizar algumas tarefas, em que não possuísse câmera, microfone ou armazenamento. Nenhum dado pessoal será processado local ou remotamente, portanto, a legislação da União Europeia (EU) sobre proteção e privacidade de dados, Regulamento Geral de Proteção de Dados (GDPR), não afetará este tipo de dispositivo. Além disso, se um dispositivo desta natureza estivesse localizado em local público, não haveria necessidade de qualquer exigência, consentimento ou autorização aos utilizadores intervenientes ou até mesmo aos que apenas se aproximam do dispositivo, pois não distinguiria o utilizador e também não o reconheceria, já que a recolha de comandos por gestos é realizada por via de sensores não evasivos. Recentemente, o mundo foi surpreendido por vários casos de um novo coronavírus que causou uma pandemia em escala mundial. Novas regras tiveram que ser aplicadas, à forma como interagimos com os objetos para não espalhar a doença. Foi criado um protótipo discriminado neste estudo, utilizando uma grelha de sensores ultrassônicos, capaz de reconhecer gestos treinados. Esta interação pode ser feita à distância do protótipo, e não requer contato, no qual se resolve uma interação humana com o computador evitando que o utilizador use o tato no dispositivo, evitando assim o contágio de (COVID-19). Os resultados da fase inicial evidenciam que o sistema proposto é adequado para criar um novo tipo de Dispositivo de Interface Humana (HID), podendo substituir dispositivos como um controle remoto de televisão, ou uma nova forma de interagir com painéis de informação em locais públicos, como um shopping center, aeroportos, estações de comboios e de autocarros

ULTRASONIC IMAGING AND TACTILE SENSING FOR ROBOTIC SYSTEMS

This research develops several novel algorithms that enhance the operation of ultrasonic and tactile sensors for robotic applications. The emphasis is on reducing the overall cost, system complexity, and enabling operation on resource-constrained embedded devices with the main focus on ultrasonics. The research improves key performance characteristics of pulse-echo sensor systems -- the minimum range, range resolution, and multi-object localization. The former two aspects are improved through the application of model-based and model-free techniques. Time optimal principles precisely control the oscillations of transmitting and receiving ultrasonic transducers, influencing the shape of the pressure waves. The model-free approach develops simple learning procedures to manipulate transducer oscillations, resulting in algorithms that are insensitive to parameter variations. Multi-object localization is achieved through phased array techniques that determine the positions of reflectors in 3-D space using a receiver array consisting of a small number of elements. The array design and the processing algorithm allow simultaneous determination of the reflector positions, achieving high sensor throughputs. Tactile sensing is a minor focus of this research that leverages machine learning in combination with an exploratory procedure to estimate the unknown stiffness of a grasped object. Gripper mechanisms with full-actuation and under-actuation are studied, and the object stiffness is estimated using regression. Sensor measurements use actuator position and current as the inputs. Regressor design, dataset generation, and the estimation performance under nonlinear effects, such as dry friction, parameter variations, and under-actuated transmission mechanisms are addressed.Ph.D

Low complexity multi-directional in-air ultrasonic gesture recognition using a TCN

On the trend of ultrasound-based gesture recognition, this study introduces the concept of time-sequence classification of ultrasonic patterns induced by hand movements on a microphone array. We refer to time-sequence ultrasound echoes as continuous frequency patterns being received in real-time at different steering angles. The ultrasound source is a single tone continuously being emitted from the center of the microphone array. In the interim, the array beamforms and locates an ultrasonic activity (induced echoes) after which a processing pipeline is initiated to extract band-limited frequency features. These beamformed features are organized in a 2D matrix of size 11 × 30 updated every 10ms on which a Temporal Convolutional Network (TCN) outputs continuous classification. Prior to that, the same TCN is trained to classify Doppler shift variability rate. Using this approach, we show that a user can easily achieve 49 gestures at different steering angles by means of sequence detection. To make it simple to users, we define two Doppler shift variability rates; very slow and very fast which the TCN detects 95-99% of the time. Not only a gesture can be performed at different directions but also the length of each performed gesture can be measured. This leverages the diversity of inair ultrasonic gestures allowing more control capabilities. The process is designed under low-resource settings; that is, given the fact that this real-time process is always-on, the power and memory resources should be optimized. The proposed solution needs 6:2 − 10:2 MMACs and a memory footprint of 6KB allowing such gesture recognition system to be hosted by energyconstrained edge devices such as smart-speakers