Novel Hybrid-Learning Algorithms for Improved Millimeter-Wave Imaging Systems

Increasing attention is being paid to millimeter-wave (mmWave), 30 GHz to 300 GHz, and terahertz (THz), 300 GHz to 10 THz, sensing applications including security sensing, industrial packaging, medical imaging, and non-destructive testing. Traditional methods for perception and imaging are challenged by novel data-driven algorithms that offer improved resolution, localization, and detection rates. Over the past decade, deep learning technology has garnered substantial popularity, particularly in perception and computer vision applications. Whereas conventional signal processing techniques are more easily generalized to various applications, hybrid approaches where signal processing and learning-based algorithms are interleaved pose a promising compromise between performance and generalizability. Furthermore, such hybrid algorithms improve model training by leveraging the known characteristics of radio frequency (RF) waveforms, thus yielding more efficiently trained deep learning algorithms and offering higher performance than conventional methods. This dissertation introduces novel hybrid-learning algorithms for improved mmWave imaging systems applicable to a host of problems in perception and sensing. Various problem spaces are explored, including static and dynamic gesture classification; precise hand localization for human computer interaction; high-resolution near-field mmWave imaging using forward synthetic aperture radar (SAR); SAR under irregular scanning geometries; mmWave image super-resolution using deep neural network (DNN) and Vision Transformer (ViT) architectures; and data-level multiband radar fusion using a novel hybrid-learning architecture. Furthermore, we introduce several novel approaches for deep learning model training and dataset synthesis.Comment: PhD Dissertation Submitted to UTD ECE Departmen

Doctor of Philosophy

dissertationLow-cost wireless embedded systems can make radio channel measurements for the purposes of radio localization, synchronization, and breathing monitoring. Most of those systems measure the radio channel via the received signal strength indicator (RSSI), which is widely available on inexpensive radio transceivers. However, the use of standard RSSI imposes multiple limitations on the accuracy and reliability of such systems; moreover, higher accuracy is only accessible with very high-cost systems, both in bandwidth and device costs. On the other hand, wireless devices also rely on synchronized notion of time to coordinate tasks (transmit, receive, sleep, etc.), especially in time-based localization systems. Existing solutions use multiple message exchanges to estimate time offset and clock skew, which further increases channel utilization. In this dissertation, the design of the systems that use RSSI for device-free localization, device-based localization, and breathing monitoring applications are evaluated. Next, the design and evaluation of novel wireless embedded systems are introduced to enable more fine-grained radio signal measurements to the application. I design and study the effect of increasing the resolution of RSSI beyond the typical 1 dB step size, which is the current standard, with a couple of example applications: breathing monitoring and gesture recognition. Lastly, the Stitch architecture is then proposed to allow the frequency and time synchronization of multiple nodes' clocks. The prototype platform, Chronos, implements radio frequency synchronization (RFS), which accesses complex baseband samples from a low-power low-cost narrowband radio, estimates the carrier frequency offset, and iteratively drives the difference between two nodes' main local oscillators (LO) to less than 3 parts per billion (ppb). An optimized time synchronization and ranging protocols (EffToF) is designed and implemented to achieve the same timing accuracy as the state-of-the-art but with 59% less utilization of the UWB channel. Based on this dissertation, I could foresee Stitch and RFS further improving the robustness of communications infrastructure to GPS jamming, allow exploration of applications such as distributed beamforming and MIMO, and enable new highly-synchronous wireless sensing and actuation systems

Fragility, robustness, and antifragility in deep neural networks

This PhD thesis investigates the relationship between network architectures and the robustness against adversarial attacks using a novel methodology that considers both aspects as part of the robustness analysis. Through an investigation on the adversarial targeting of neurons, specifically in the first convolutional layer of a deep neural network (DNN), we observe a relationship between neurons that affect the test accuracy of the DNN, when inferring on a clean, unperturbed dataset, subsequently characterising them as fragile, and those neurons targeted by a potential adversarial attack. We show how the fragile neurons of a DNN convolutional layer evolve over the network training procedure and propose an algorithm to show the targeting of fragile neurons by adversarial attacks. Using the developed adversarial targeting algorithm we show that adversarial attacks focus on specific components of the convolutional layer, framing the adversarial perturbations as attacks on fragile neurons. The task of analysing the robustness of DNNs, thus, leads us to the identification of fragile and non-fragile network parameters, where non-fragile refers to any parameters that do not degrade the performance when subjected to perturbations, as opposed to fragile parameters that do degrade network performance. When discussing perturbations, we consider both variations to the network parameters and the input dataset, in the form of adversarial attacks. We further extend the analysis to characterise the parameters of deep neural networks as either fragile, robust, or antifragile, and show that network accuracy is impacted negatively, invariantly, or positively w.r.t. defined global and local robustness scores that are computed using a baseline network performance. We design a signal processing technique in the form of synaptic filters that identify the fragility, robustness and antifragility characteristics of deep neural network parameters. We subject a network to synaptic filters and compare the network responses for both clean and adversarial datasets, subsequently exposing parameters targeted by the adversary. Our results identify the structural fragility of network architectures and show how they evolve over the training process, thus informing us on the learning landscapes of DNNs. We find that, for a given network architecture, global and local filtering responses have invariant features to different datasets over the learning landscape. Vice-versa, for a given dataset we identify invariant features across different network architectures. Our proposed analysis of fragility, robustness and antifragility of deep neural networks is useful for designing compact networks by removing particularly the antifragile parameters. We improve the adversarial robustness of networks using a selective backpropagation method that, upon identification of parameter characterisations, retrains only the robust and antifrgaile parameter updates, whilst omitting the fragile parameter updates during the training procedure. Following this, we develop DNNs for two novel, real-world applications; a DNN designed to identify the the optimum denoising filter for noisy ECG waveforms, and DNNs designed to classify human activities and motion intensities from signals measured using an ultra wide-band radar system. We use original datasets for both tasks and develop novel DNN architectures for the classification tasks. Subsequently, we apply the developed selective backpropagation method to train the custom-designed DNNs and observed an increase in adversarial robustness for the DNNs evaluated. Furthermore, for both the signal denoising filter selection and activity classification tasks, we discern an improvement in the test accuracy when applied to the clean, unperturbed dataset. We successfully show that the proposed selective backpropagation method is capable of improving the adversarial robustness of networks, and in certain instances, also the regular test accuracy. Supporting results for these findings are presented across the chapters of this thesis

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

Mobility classification of cattle with micro-Doppler radar

Lameness in dairy cattle is a welfare concern that negatively impacts animal productivity and farmer profitability. Micro-Doppler radar sensing has been previously suggested as a potential system for automating lameness detection in ruminants. This thesis investigates the refinement of the proposed automated system by analysing and enhancing the repeatability and accuracy of the existing scoring method in cattle mobility scoring, used to provide labels in machine learning. The main aims of the thesis were (1) to quantify the performance of the micro-Doppler radar sensing method for the assessment of mobility, (2) to characterise and validate micro-Doppler radar signatures of dairy cattle with varying degrees of gait impairment, and (3) to develop machine learning algorithms that can infer the mobility status of the animals under test from their radar signatures and support automatic contactless classification. The first study investigated inter-assessor agreement using a 4-level system and modifications to it, as well as the impact of factors such as mobility scoring experience, confidence in scoring decisions, and video characteristics. The results revealed low levels of agreement between assessors' scores, with kappa values ranging from 0.16 to 0.53. However, after transforming and reducing the mobility scoring system levels, an improvement was observed, with kappa values ranging from 0.2 to 0.67. Subsequently, a longitudinal study was conducted using good-agreement scores as ground truth labels in supervised machine-learning models. However, the accuracy of the algorithmic models was found to be insufficient, ranging from 0.57 to 0.63. To address this issue, different labelling systems and data pre-processing techniques were explored in a cross-sectional study. Nonetheless, the inter-assessor agreement remained challenging, with an average kappa value of 0.37 (SD = 0.16), and high-accuracy algorithmic predictions remained elusive, with an average accuracy of 56.1 (SD =16.58). Finally, the algorithms' performance was tested with high-confidence labels, which consisted of only scores 0 and 3 of the AHDB system. This testing resulted in good classification accuracy (0.82), specificity (0.79), and sensitivity (0.85). This led to the proposal of a new approach to producing labels, testing vantage point changes, and improving the performance of machine learning models (average accuracy = 0.7 & SD = 0.17, average sensitivity = 0.68 & SD = 0.27, average specificity = 0.75 & SD = 0.17). The research identified a challenge in creating high-confidence diagnostic labels for supervised machine learning-based algorithms to automate the detection and classification of lameness in dairy cows. As a result, the original goals were partially overridden, with the focus shifted to creating reliable labels that would perform well with radar data and machine learning. This point was considered necessary for smooth system development and process automation. Nevertheless, we managed to quantify the performance of the micro-Doppler radar system, partially develop the supervised machine learning algorithms, compare levels of agreement among multiple assessors, evaluate the assessment tools, assess the mobility evaluation process and gather a valuable data set which can be used as a foundation for subsequent studies. Finally, the thesis suggests changes in the assessment process to improve the prediction accuracy of algorithms based on supervised machine learning with radar data

Remote Sensing

This dual conception of remote sensing brought us to the idea of preparing two different books; in addition to the first book which displays recent advances in remote sensing applications, this book is devoted to new techniques for data processing, sensors and platforms. We do not intend this book to cover all aspects of remote sensing techniques and platforms, since it would be an impossible task for a single volume. Instead, we have collected a number of high-quality, original and representative contributions in those areas

Sensors and Systems for Indoor Positioning

This reprint is a reprint of the articles that appeared in Sensors' (MDPI) Special Issue on “Sensors and Systems for Indoor Positioning". The published original contributions focused on systems and technologies to enable indoor applications

Sensor Signal and Information Processing II

In the current age of information explosion, newly invented technological sensors and software are now tightly integrated with our everyday lives. Many sensor processing algorithms have incorporated some forms of computational intelligence as part of their core framework in problem solving. These algorithms have the capacity to generalize and discover knowledge for themselves and learn new information whenever unseen data are captured. The primary aim of sensor processing is to develop techniques to interpret, understand, and act on information contained in the data. The interest of this book is in developing intelligent signal processing in order to pave the way for smart sensors. This involves mathematical advancement of nonlinear signal processing theory and its applications that extend far beyond traditional techniques. It bridges the boundary between theory and application, developing novel theoretically inspired methodologies targeting both longstanding and emergent signal processing applications. The topic ranges from phishing detection to integration of terrestrial laser scanning, and from fault diagnosis to bio-inspiring filtering. The book will appeal to established practitioners, along with researchers and students in the emerging field of smart sensors processing

An Assessment of Indoor Geolocation Systems

Currently there is a need to design, develop, and deploy autonomous and portable indoor geolocation systems to fulfil the needs of military, civilian, governmental and commercial customers where GPS and GLONASS signals are not available due to the limitations of both GPS and GLONASS signal structure designs. The goal of this dissertation is (1) to introduce geolocation systems; (2) to classify the state of the art geolocation systems; (3) to identify the issues with the state of the art indoor geolocation systems; and (4) to propose and assess four WPI indoor geolocation systems. It is assessed that the current GPS and GLONASS signal structures are inadequate to overcome two main design concerns; namely, (1) the near-far effect and (2) the multipath effect. We propose four WPI indoor geolocation systems as an alternative solution to near-far and multipath effects. The WPI indoor geolocation systems are (1) a DSSS/CDMA indoor geolocation system, (2) a DSSS/CDMA/FDMA indoor geolocation system, (3) a DSSS/OFDM/CDMA/FDMA indoor geolocation system, and (4) an OFDM/FDMA indoor geolocation system. Each system is researched, discussed, and analyzed based on its principle of operation, its transmitter, the indoor channel, and its receiver design and issues associated with obtaining an observable to achieve indoor navigation. Our assessment of these systems concludes the following. First, a DSSS/CDMA indoor geolocation system is inadequate to neither overcome the near-far effect not mitigate cross-channel interference due to the multipath. Second, a DSSS/CDMA/FDMA indoor geolocation system is a potential candidate for indoor positioning, with data rate up to 3.2 KBPS, pseudorange error, less than to 2 m and phase error less than 5 mm. Third, a DSSS/OFDM/CDMA/FDMA indoor geolocation system is a potential candidate to achieve similar or better navigation accuracy than a DSSS/CDMA indoor geolocation system and data rate up to 5 MBPS. Fourth, an OFDM/FDMA indoor geolocation system is another potential candidate with a totally different signal structure than the pervious three WPI indoor geolocation systems, but with similar pseudorange error performance