Radar-based Application of Pedestrian and Cyclist Micro-Doppler Signatures for Automotive Safety Systems

Die sensorbasierte Erfassung des Nahfeldes im Kontext des hochautomatisierten Fahrens erfährt einen spürbaren Trend bei der Integration von Radarsensorik. Fortschritte in der Mikroelektronik erlauben den Einsatz von hochauflösenden Radarsensoren, die durch effiziente Verfahren sowohl im Winkel als auch in der Entfernung und im Doppler die Messgenauigkeit kontinuierlich ansteigen lassen. Dadurch ergeben sich neuartige Möglichkeiten bei der Bestimmung der geometrischen und kinematischen Beschaffenheit ausgedehnter Ziele im Fahrzeugumfeld, die zur gezielten Entwicklung von automotiven Sicherheitssystemen herangezogen werden können. Im Rahmen dieser Arbeit werden ungeschützte Verkehrsteilnehmer wie Fußgänger und Radfahrer mittels eines hochauflösenden Automotive-Radars analysiert. Dabei steht die Erscheinung des Mikro-Doppler-Effekts, hervorgerufen durch das hohe Maß an kinematischen Freiheitsgraden der Objekte, im Vordergrund der Betrachtung. Die durch den Mikro-Doppler-Effekt entstehenden charakteristischen Radar-Signaturen erlauben eine detailliertere Perzeption der Objekte und können in direkten Zusammenhang zu ihren aktuellen Bewegungszuständen gesetzt werden. Es werden neuartige Methoden vorgestellt, die die geometrischen und kinematischen Ausdehnungen der Objekte berücksichtigen und echtzeitfähige Ansätze zur Klassifikation und Verhaltensindikation realisieren. Wird ein ausgedehntes Ziel (z.B. Radfahrer) von einem Radarsensor detektiert, können aus dessen Mikro-Doppler-Signatur wesentliche Eigenschaften bezüglich seines Bewegungszustandes innerhalb eines Messzyklus erfasst werden. Die Geschwindigkeitsverteilungen der sich drehenden Räder erlauben eine adaptive Eingrenzung der Tretbewegung, deren Verhalten essentielle Merkmale im Hinblick auf eine vorausschauende Unfallprädiktion aufweist. Ferner unterliegen ausgedehnte Radarziele einer Orientierungsabhängigkeit, die deren geometrischen und kinematischen Profile direkt beeinflusst. Dies kann sich sowohl negativ auf die Klassifikations-Performance als auch auf die Verwertbarkeit von Parametern auswirken, die eine Absichtsbekundung des Radarziels konstituieren. Am Beispiel des Radfahrers wird hierzu ein Verfahren vorgestellt, das die orientierungsabhängigen Parameter in Entfernung und Doppler normalisiert und die gemessenen Mehrdeutigkeiten kompensiert. Ferner wird in dieser Arbeit eine Methodik vorgestellt, die auf Grundlage des Mikro- Doppler-Profils eines Fußgängers dessen Beinbewegungen über die Zeit schätzt (Tracking) und wertvolle Objektinformationen hinsichtlich seines Bewegungsverhaltens offenbart. Dazu wird ein Bewegungsmodell entwickelt, das die nichtlineare Fortbewegung des Beins approximiert und dessen hohes Maß an biomechanischer Variabilität abbildet. Durch die Einbeziehung einer wahrscheinlichkeitsbasierten Datenassoziation werden die Radar-Detektionen ihren jeweils hervorrufenden Quellen (linkes und rechtes Bein) zugeordnet und eine Trennung der Gliedmaßen realisiert. Im Gegensatz zu bisherigen Tracking-Verfahren weist die vorgestellte Methodik eine Steigerung in der Genauigkeit der Objektinformationen auf und stellt damit einen entscheidenden Vorteil für zukünftige Fahrerassistenzsysteme dar, um deutlich schneller auf kritische Verkehrssituationen reagieren zu können.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 135Sensor-based detection of the near field in the context of highly automated driving is experiencing a noticeable trend in the integration of radar sensor technology. Advances in microelectronics allow the use of high-resolution radar sensors that continuously increase measurement accuracy through efficient processes in angle as well as distance and Doppler. This opens up novel possibilities in determining the geometric and kinematic nature of extended targets in the vehicle environment, which can be used for the specific development of automotive safety systems. In this work, vulnerable road users such as pedestrians and cyclists are analyzed using a high-resolution automotive radar. The focus is on the appearance of the micro-Doppler effect, caused by the objects’ high kinematic degree of freedom. The characteristic radar signatures produced by the micro-Doppler effect allow a clearer perception of the objects and can be directly related to their current state of motion. Novel methods are presented that consider the geometric and kinematic extents of the objects and realize real-time approaches to classification and behavioral indication. When a radar sensor detects an extended target (e.g., bicyclist), its motion state’s fundamental properties can be captured from its micro-Doppler signature within a measurement cycle. The spinning wheels’ velocity distributions allow an adaptive containment of the pedaling motion, whose behavior exhibits essential characteristics concerning predictive accident prediction. Furthermore, extended radar targets are subject to orientation dependence, directly affecting their geometric and kinematic profiles. This can negatively affect both the classification performance and the usability of parameters constituting the radar target’s intention statement. For this purpose, using the cyclist as an example, a method is presented that normalizes the orientation-dependent parameters in range and Doppler and compensates for the measured ambiguities. Furthermore, this paper presents a methodology that estimates a pedestrian’s leg motion over time (tracking) based on the pedestrian’s micro-Doppler profile and reveals valuable object information regarding his motion behavior. To this end, a motion model is developed that approximates the leg’s nonlinear locomotion and represents its high degree of biomechanical variability. By incorporating likelihood-based data association, radar detections are assigned to their respective evoking sources (left and right leg), and limb separation is realized. In contrast to previous tracking methods, the presented methodology shows an increase in the object information’s accuracy. It thus represents a decisive advantage for future driver assistance systems in order to be able to react significantly faster to critical traffic situations.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 13

Frequency Domain Decomposition of Digital Video Containing Multiple Moving Objects

Motion estimation has been dominated by time domain methods such as block matching and optical flow. However, these methods have problems with multiple moving objects in the video scene, moving backgrounds, noise, and fractional pixel/frame motion. This dissertation proposes a frequency domain method (FDM) that solves these problems. The methodology introduced here addresses multiple moving objects, with or without a moving background, 3-D frequency domain decomposition of digital video as the sum of locally translational (or, in the case of background, a globally translational motion), with high noise rejection. Additionally, via a version of the chirp-Z, fractional pixel/frame motion detection and quantification is accomplished. Furthermore, images of particular moving objects can be extracted and reconstructed from the frequency domain. Finally, this method can be integrated into a larger system to support motion analysis. The method presented here has been tested with synthetic data, realistic, high fidelity simulations, and actual data from established video archives to verify the claims made for the method, all presented here. In addition, a convincing comparison with an up-and-coming spatial domain method, incremental principal component pursuit (iPCP), is presented, where the FDM performs markedly better than its competition

Fast algorithms for nonuniform Chirp-Fourier transform

The Chirp-Fourier transform is one of the most important tools of the modern signal processing. It has been widely used in the fields of ultrasound imaging, parameter estimation, and so on. The key to its application lies in the sampling and fast algorithms. In practical applications, nonuniform sampling can be caused by sampling equipment and other reasons. For the nonuniform sampling, we utilized function approximation and interpolation theory to construct different approximation forms of Chirp-Fourier transform kernel function, and proposed three fast nonuniform Chirp-Fourier transform algorithms. By analyzing the approximation error and the computational complexity of these algorithms, the effectiveness of the proposed algorithms was proved

Computational imaging and automated identification for aqueous environments

Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution June 2011Sampling the vast volumes of the ocean requires tools capable of observing from a distance while retaining detail necessary for biology and ecology, ideal for optical methods. Algorithms that work with existing SeaBED AUV imagery are developed, including habitat classi fication with bag-of-words models and multi-stage boosting for rock sh detection. Methods for extracting images of sh from videos of longline operations are demonstrated. A prototype digital holographic imaging device is designed and tested for quantitative in situ microscale imaging. Theory to support the device is developed, including particle noise and the effects of motion. A Wigner-domain model provides optimal settings and optical limits for spherical and planar holographic references. Algorithms to extract the information from real-world digital holograms are created. Focus metrics are discussed, including a novel focus detector using local Zernike moments. Two methods for estimating lateral positions of objects in holograms without reconstruction are presented by extending a summation kernel to spherical references and using a local frequency signature from a Riesz transform. A new metric for quickly estimating object depths without reconstruction is proposed and tested. An example application, quantifying oil droplet size distributions in an underwater plume, demonstrates the efficacy of the prototype and algorithms.Funding was provided by NOAA Grant #5710002014, NOAA NMFS Grant #NA17RJ1223, NSF Grant #OCE-0925284, and NOAA Grant #NA10OAR417008

Modeling Backscattering Behavior of Vulnerable Road Users Based on High-Resolution Radar Measurements

Bei der Weiterentwicklung der Technologie des autonomen Fahrens (AD) ist die Beschaffung zuverlässiger dreidimensionaler Umgebungsinformationen eine unverzichtbare Aufgabe, um ein sicheres Fahren zu ermöglichen. Diese Herausforderung kann durch den Einsatz von Fahrzeugradaren zusammen mit optischen Sensoren, z. B. Kameras oder Lidars, bewältigt werden, sei es in der Simulation oder in konventionellen Tests auf der Straße. Das Betriebsverhalten von Fahrzeugradaren kann in einer Over-the-Air (OTA) Vehicle-in-the-Loop (ViL) Umgebung genau bewertet werden. Für eine umfassende experimentelle Verifizierung der Fahrzeugradare muss jedoch die Umgebung, insbesondere die gefährdeten Verkehrsteilnehmer (VRUs), möglichst realistisch modelliert werden. Moderne Radarsensoren sind in der Lage, hochaufgelöste Erkennungsinformationen von komplexen Verkehrszielen zu liefern, um diese zu verfolgen. Diese hochauflösenden Erkennungsdaten, die die reflektierten Signale von den Streupunkten (SPs) der VRUs enthalten, können zur Erzeugung von Rückstreumodelle genutzt werden. Darüber hinaus kann ein realistischeres Rückstreumodell der VRUs, insbesondere von Menschen als Fußgänger oder Radfahrer, durch die Modellierung der Bewegung ihrer Extremitäten in Verkehrsszenarien erreicht werden. Die Voraussetzung für die Erstellung eines solchen detaillierten Modells in verschiedenen Situationen sind der Radarquerschnitt (RCS) und die Doppler-Signaturen, die sich aus den menschlichen Extremitäten in einer bewegten Situation ergeben. Diese Daten können durch die gesammelten Radardaten aus hochauflösenden RCS-Messungen im Radial- und Winkelbereich gewonnen werden, was durch die Analyse der Range-Doppler-Spezifikation der menschlichen Extremitäten in verschiedenen Bewegungen möglich ist. Die entwickelten realistischen Radarmodelle können bei der Wellenausbreitung im Radarkanal, bei der Zielerkennung und -klassifizierung sowie bei Datentrainingsalgorithmen zur Validierung und Verifizierung der Kfz-Radarfunktionen eingesetzt werden. Anschließend kann mit dieser Bewertung die Sicherheit von fortschrittlichen Fahrerassistenzsystemen (ADAS) beurteilt werden. Daher wird in dieser Arbeit ein hochauflösendes RCS-Messverfahren vorgeschlagen, um die relevanten SPs verschiedener VRUs mit hoher radialer und winkelmäßiger Auflösung zu bestimmen. Eine Gruppe unterschiedliche VRUs wird in statischen Situationen gemessen, und die notwendigen Signalverarbeitungsschritte, um die relevanten SPs mit den entsprechenden RCS-Werten zu extrahieren, werden im Detail beschrieben. Während der Analyse der gemessenen Daten wird ein Algorithmus entwickelt, um die physischen Größen der gemessenen Testpersonen aus dem extrahierten Rückstreumodell zu schätzen und sie anhand ihrer Größe und Statur zu klassifizieren. Zusätzlich wird ein Dummy-Mensch vermessen, der eine vergleichbare Größe wie die vermessenen Probanden hat. Das extrahierte Rückstreuverhalten einer beispielhaften VRU-Gruppe wird für ihre verschiedenen Typen ausgewertet, um die Übereinstimmung zwischen virtuellen Validierungen und der Realität aufzuzeigen und den Genauigkeitsgrad der Modelle sicherzustellen. In einem weiteren Schritt wird diese hochauflösende RCS-Messtechnik mit der Motion Capture Technologie kombiniert, um die Reflektivität der SPs von den menschlichen Körperregionen in verschiedenen Bewegungen zu erfassen und die Radarsignaturen der menschlichen Extremitäten genau zu schätzen. Spezielle Signalverarbeitungsschritte werden eingesetzt, um die Radarsignaturen aus den Messergebnissen des sich bewegenden Menschen zu extrahieren. Diese nachbearbeiteten Daten ermöglichen es der Technik, die zeitlich variierenden SPs an den Extremitäten des menschlichen Körpers mit den entsprechenden RCS-Werten und Dopplersignaturen einzuführen. Das extrahierte Rückstreumodell der VRUs enthält eine Vielzahl von SPs. Daher wird ein Clustering-Algorithmus entwickelt, um die Berechnungskomplexität bei Radarkanalsimulationen durch die Einführung einiger virtueller Streuzentren (SCs) zu minimieren. Jedes entwickelte virtuelle SCs hat seine eigene spezifische Streueigenschaft

Rf sensing and processing methods for noninvasive health monitoring

Vulnerable populations include groups of people with a higher risk of poor health as a result of the limitations due to illness or disability. The health issues of vulnerable populations include three categories: physical, psychological, and social. The people with physical issues include high-risk mothers and infants, older adults and others with chronic illnesses and people with disabilities. The psychological issues of vulnerable populations include chronic mental conditions, such as bipolar disorder, major depression, and hyperactivity disorder, as well as substance abuse and those who are suicidal. The social issues in vulnerable populations include those living in abusive families, the homeless, etc. This dissertation concentrates on methods for helping two groups of vulnerable populations, namely, frail older adults and psychiatric hospital patients, to monitor their activity level, respiration rate, sleeping quality, and restless time in bed. In the first part of our work, we investigate a contactless monitoring system for psychiatric patients in a naturalistic hospital setting that can track their motion in bed, estimate the breathing rate of patients during their peaceful sleeping periods, and can be used to estimate a patient's restless time and sleep quality. Specifically, the contactless monitoring system uses a Vayyar Radar system with a carrier frequency of 6.014 GHz to capture all reflections by the FMCW (frequency modulation continuous waveform) signal. The Vayyar Radar system has been installed in a Psychiatric Center to capture 12 nights with over 135 hours of data from 7 patients. A depth camera and a thermal camera have also been installed and are used as the ground truth. The goal is to classify in bed and out of bed classes, quantify restlessness in bed, and determine the breathing rate while patients are lying in bed. We have simulated the psychiatric hospital set-up in the lab, where a respiration belt is used for ground truth, and tested the system with body postures of patients observed in the psychiatric hospital. We estimated respiration rate with different sleep postures, with the aim of investigating a contactless monitoring system for psychiatric patients in the hospital that can estimate the breathing rate of patients during typical sleeping postures, and find the torso area when the patients use other postures, such as reading books in bed or reversing the body on the bed. In the second part of our work, we investigate two methods for learning the room structure via radio wave reflections for longitudinal health monitoring of older adults in a naturalistic home setting. The goal is to use these data as part of a monitoring system that can be easily installed in a home with minimal configuration, for the purpose of detecting very early signs of illness and functional decline. Two studies are conducted using RF (radio frequency) sensing. The first method learns the structure from the RF clutter patterns and uses the beat frequency of the maximum peak in each chirp to calculate the wall position. The second method learns the room structure from active movement patterns and uses the open space between the clusters of active movement patterns to estimate the possible wall locations. Comparing the two results from these methods provides a more robust wall location. In addition, a background filter is designed based on the wall position, and the activity level of people in different rooms is estimated using a fuzzy rule system applied to the RF motion data

Computational imaging and automated identification for aqueous environments

Thesis (Ph. D.)--Joint Program in Oceanography/Applied Ocean Science and Engineering (Massachusetts Institute of Technology, Dept. of Mechanical Engineering; and the Woods Hole Oceanographic Institution), 2011."June 2011." Cataloged from PDF version of thesis.Includes bibliographical references (p. 253-293).Sampling the vast volumes of the ocean requires tools capable of observing from a distance while retaining detail necessary for biology and ecology, ideal for optical methods. Algorithms that work with existing SeaBED AUV imagery are developed, including habitat classification with bag-of-words models and multi-stage boosting for rock sh detection. Methods for extracting images of sh from videos of long-line operations are demonstrated. A prototype digital holographic imaging device is designed and tested for quantitative in situ microscale imaging. Theory to support the device is developed, including particle noise and the effects of motion. A Wigner-domain model provides optimal settings and optical limits for spherical and planar holographic references. Algorithms to extract the information from real-world digital holograms are created. Focus metrics are discussed, including a novel focus detector using local Zernike moments. Two methods for estimating lateral positions of objects in holograms without reconstruction are presented by extending a summation kernel to spherical references and using a local frequency signature from a Riesz transform. A new metric for quickly estimating object depths without reconstruction is proposed and tested. An example application, quantifying oil droplet size distributions in an underwater plume, demonstrates the efficacy of the prototype and algorithms.by Nicholas C. Loomis.Ph.D