Classification of Human Activities with Distributed Radar Systems

This thesis introduces the relevance of radar systems in the realm of human activity recognition (HAR) in Chapter 1. The study touches upon the complex understanding of continuous human activities and the existing challenges and gaps in current methodologies, hinting at the innovative technical approaches that are to be detailed in the following chapters. The technical foundation of the research is given in Chapter 2 by introducing distributed ultrawideband (UWB) radar systems. These systems, especially when spatially distributed, bring a depth of information by integrating data from multiple radar nodes and spatial perspectives. There is a significant emphasis on how different fusion techniques, both late and early, play a crucial role in harnessing data effectively, particularly in the context of HAR.A critical contribution in the study is the potential to deviate from conventional radar data domains, such as microDoppler spectrograms for activity recognition. The research in Chapter 3 highlights an alternative approach, rooted in the radar phase information from a highresolution rangetime map, which bypasses the limitations of common FFTbased radar data domains. This methodology, paired with the histogram of oriented gradients (HOG) algorithm, showcases promising results that can be particularly interesting for realtime applications with computational constraints.The research in Chapter 4 underlines the efficacy of employing a network of spatially distributed UWB radars for continuous HAR. These networks address the downsides of using a single sensor, like unfavorable aspectangle observations. The study delves into fusion methodologies and their implementation in classifying activities, particularly using recurrent neural networks. To assess these continuous recognition systems, novel evaluation metrics are proposed, offering a deeper insight into the practicality and effectiveness of such systems with temporal classification capabilities.Indoor radar networks often face multipath challenges. The study in Chapter 5 not only identifies this challenge, but also uses the multipath components by leveraging these typically unwanted phenomena to enhance classification capabilities. Through a pipeline that isolates, determines, and analyzes different propagation pathways, there is an evident boost in the network’s perception. This novel approach showcases a significant performance upward trend, especially when employing convolutional neural networks.Chapter 6 of the research focuses on the complexities of HAR in crowded environments. The study introduces the challenges of differentiating the activities of walking versus standing idle for multiple individuals simultaneously. The investigation shows initial promising results by using synthetic data generated from experimental recordings, by employing a regressionbased approach and leveraging diverse techniques such as LSTM, CNN, SVM, and linear regression.In conclusion, the research offers a reflective glance at the breakthroughs achieved in the domain of radarbased HAR in Chapter 7. The significant contributions and advancements of the study are highlighted. Looking ahead, the chapter identifies research areas for exploration and further improvement.Microwave Sensing, Signals & System

Derivative Target Line (DTL) for Continuous Human Activity Detection and Recognition

In this paper, we investigate the classification of Activities of Daily Living (ADL) by using a pulsed ultra-wideband radar. Specifically, we focus on contiguous activities that can be inseparable in time and share a common transition, such as walking and falling. The range-time data domain is deliberately exploited to determine transitions from translation activities to in-place activities and vice versa, using a simple, yet effective approach based on the proposed Derivative Target Line (DTL). The separation of different in-place activities is then addressed using an energy detector finding the onset and offset times. Furthermore, the possible ADL for classification are limited at any decision stage based on kinematic constraints of human movements. We show that such limitation of classes at any given time leads to a classification improvement over a classifier containing always all ADL classes.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Evaluation Metrics for Continuous Human Activity Classification Using Distributed Radar Networks

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated using 5 spatially distributed pulsed Ultra-Wideband (UWB) radars. Such activities performed in arbitrary and unconstrained trajectories render a more natural occurrence of Activities of Daily Living (ADL) to be recognized. An innovative signal level fusion method was applied on the Range-Time (RT) maps, and deep learning classification via Recurrent Neural Networks (RNN) with and without bidi-rectionality was used on the computed micro-Doppler (μD) spectrogram. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and for imbalances in the dataset are proposed and compared with existing metrics. It is shown that conventional accuracy evaluation is too coarse, and that the proposed metrics need to be considered for a more comprehensive evaluation.Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Distributed radar fusion and recurrent networks for classification of continuous human activities

Continuous Human Activity Recognition (HAR) in arbitrary directions is investigated in this paper using a network of five spatially distributed pulsed Ultra-Wideband radars. While activities performed continuously and in unconstrained trajectories provide a more realistic and natural scenario for HAR, the network of radar sensors is proposed to address the issue of unfavourable or occluded perspectives when using only a single sensor. Different techniques to combine the relevant information from the multiple radars in the network are investigated, focussing on signal level fusion directly applied on Range-Time maps, and the selection of radar nodes based on location and velocity of the target derived from multilateration processing and tracking. Recurrent Neural Networks with and without bidirectionality are used to classify the activities based on the micro-Doppler (μD) spectrograms obtained for sensor fusion techniques. To assess classification performances, novel evaluation metrics accounting for the continuous nature of the sequence of activities and inherent imbalances in the dataset are proposed and compared with existing metrics. It is shown that the conventional accuracy metric may not capture all the important aspects for continuous HAR, and the proposed metrics can be considered for a more comprehensive evaluation.Microwave Sensing, Signals & System

Phase-based Classification for Arm Gesture and Gross-Motor Activities using Histogram of Oriented Gradients

Micro-Doppler spectrograms are a conventional data representation domain for movement recognition such as Human Activity Recognition (HAR) or gesture detection. However, they present the problem of time-frequency resolution trade-offs of Short-Time Fourier Transform (STFT), which may have limitations due to unambiguous Doppler frequency, and the STFT computation may be onerous in constrained embedded environments. We propose in this paper an alternative classification approach based on the radar phase information directly extracted from high-resolution Range Map (RM). This novel approach does not use the aforementioned micro-Doppler processing, and yet achieves equivalent or even superior classification results. This shows a potential advantage for low-latency, real-time applications, or computationally constrained scenarios. The proposed method exploits the Histogram of Oriented Gradients (HOG) algorithm as an effective feature extraction algorithm, specifically its capability to capture the unique shape and patterns present in the wrapped phase domains, such as their contour intensity and distributions. Validation results consistently above 92% demonstrate the effectiveness of this method on two independent datasets of arm gestures and gross-motor activities. These were classified with three algorithms, namely the Nearest Neighbor (NN), the linear Support Vector Machine (SVM), and the Gaussian SVM classifiers using the proposed phase information. Feature fusion of different data domains, e.g. the modulus of the RM fused with the RM phase information, is also investigated and shows classification improvement specifically for the robustness of activity performances, such as the aspect angle and the speed of performance.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Radar Sensing in Healthcare: Challenges and Achievements in Human Activity Classification & Vital Signs Monitoring

Driven by its contactless sensing capabilities and the lack of optical images being recorded, radar technology has been recently investigated in the context of human healthcare. This includes a broad range of applications, such as human activity classification, fall detection, gait and mobility analysis, and monitoring of vital signs such as respiration and heartbeat. In this paper, a review of notable achievements in these areas and open research challenges is provided, showing the potential of radar sensing for human healthcare and assisted living.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Exploiting Radar Data Domains for Classification with Spatially Distributed Nodes

Recognition of continuous human activities is investigated in unconstrained movement directions using multiple spatially distributed radar nodes, where activities can occur at unfavourable aspect angles or occluded perspectives when using a single node. Furthermore, such networks are favourable not only for the aforementioned aim, but also for larger controlled surveillance areas that may require more than just one sensor. Specifically, a distributed network can show significant differences in signature between the nodes when targets are located at long distances and different aspect angles. Radar data can be represented in various domains, where a widely known domain for Human Activity Recognition (HAR) is the microDoppler spectrogram. However, other domains might be more suitable for better classification performance or are superior for low-cost hardware with limited computational resources, such as the Range-Time or the Range-Doppler domain. An open question is how to take advantage of the diversity of information extractable from the aforesaid data domains, as well as from different distributed radar nodes that simultaneously observe a surveillance area. For this, data fusion techniques can be used at both the level of data representations for each radar node, and across the different nodes in the network. The introduced methods of decision fusion, where typically one classifier operates on each node, or feature fusion, where the data is concatenated before using one single classifier, will be exploited, investigating their performance for continuous sequence classification, a more naturalistic and realistic way of classifying human movements, also accounting for inherent imbalances in the dataset.Microwave Sensing, Signals & System

Continuous Human Activity Recognition With Distributed Radar Sensor Networks and CNN–RNN Architectures

Unconstrained human activities recognition with a radar network is considered. A hybrid classifier combining both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for spatial–temporal pattern extraction is proposed. The 2-D CNNs (2D-CNNs) are first applied to the radar data to perform spatial feature extraction on the input spectrograms. Subsequently, gated recurrent units with bidirectional implementations are used to capture the long- and short-term temporal dependencies in the feature maps generated by the 2D-CNNs. Three NN-based data fusion methods were explored and compared with utilize the rich information provided by the different radar nodes. The performance of the proposed classifier was validated rigorously using the K-fold cross-validation (CV) and leave-one-person-out (L1PO) methods. Unlike competitive research, the dataset with continuous human activities with seamless interactivity transitions that can occur at any time and unconstrained moving trajectories of the participants has been collected and used for evaluation purposes. Classification accuracy of about 90.8% is achieved for nine-class human activity recognition (HAR) by the proposed classifier with the halfway fusion methodGreen Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Point Transformer-Based Human Activity Recognition Using High-Dimensional Radar Point Clouds

Radar-based Human Activity Recognition(HAR) is considered by using snapshots of point clouds. Such point cloudsinterpret 2D images generated by an mm-wave FMCW MIMO radar enriched byincluding Doppler and temporal information. We use the similarity between suchradar data representation and the core of the self-attention concept inartificial intelligence. Three self-attention models (Point Transformer) areinvestigated to classify Activities of Daily Living (ADL). An experimentaldataset collected at TU Delft is used to explore the best combination ofdifferent input features, the effect of a proposed Adaptive ClutterCancellation (ACC) method, and the robustness in a leave-one-subject-outscenario. Results with a macro F1 score in the order of 90% are demonstratedwith the proposed method, including activities that are static postures withlittle associated Doppler.Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System

Radar-based Human Activities Classification with Complex-valued Neural Networks

Human activities classification in assisted living is one of the emerging applications of radar. The conventional analysis considers micro-Doppler signatures as the chosen input for feature extraction or deep learning classification algorithms, or, less frequently, other radar data formats such as the range-time, the range-Doppler, or the Cadence Velocity Diagram. However, these data are typically used as real-valued images, whereas they are actually complex-valued data structures. In this paper, neural networks processing radar data as complex data structures are investigated, with a focus on spectrograms, range-time, and range-Doppler plots as the data formats of choice. Different network architectures are explored both in terms of complex numbers' representations and the depth/complexity of the architecture itself. Experimental data with 9 activities and 15 volunteers collected using an UWB radar are used to test the networks' performances. It is shown that for certain data formats and network architectures, there is an advantage in using complex-valued networks compared to their real-valued counterparts.Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System