Radar based on automotive pedestrian detection using the micro Doppler effects

Orientador: Prof. Dr. Alessandro ZimmerDissertação (mestrado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa : Curitiba, 29/08/2018Inclui referências: p.74-78Resumo: O desenvolvimento do carro autônomo é hoje em dia uma prática comum entre as maiores indústrias automotivas, e também em indústrias tecnológicas, como o Google e a Apple. Ao adicionar mais sensores, o veículo é capaz de se movimentar sozinho, identificar a trajetória correta, a distância para outros carros, e também a presença de objetos e seres vivos. Entretanto, existem muitos aspectos bloqueando o lançamento do carro autônomo. Como exemplo aspectos técnicos, como o caso do reconhecimento de pedestres. Embora, esse tópico seja abundantemente estudado para o uso de câmeras digitais, as mesmas não possuem confiabilidade nas medições de velocidade e distância, e ainda apresentam péssimos resultados quando há variação ou a falta de luz no ambiente. Baseado no que foi mencionado anteriormente, o foco dessa dissertação é de desenvolver e discutir a eficiência de um sistema de rápida identificação de pedestres, utilizando um novo radar de 79GHz de frequência. O principal objetivo é reconhecer o pedestre o mais rápido possível utilizando os efeitos micro Doppler do movimento humano em situações muito próximas de um acidente, junto com o método de classificação support vector machine (SVM). Objetivando essa meta algumas técnicas são usadas ao longo do trabalho. Primeiramente, a resolução de velocidade é melhorada com técnicas de otimização multiobjetivos, como algoritmos genéticos e random search para extrair o micro efeito Doppler. Então as informações de velocidade e distância são medidas pelo radar. Em sequência, um método de extração de características chamado de video temporal gradiente é aplicado. O método de machine learning SVM classifica os objetos em pedestre e não pedestres, com quadro diferentes métodos de treinamento. Por fim, é possível ver as vantagens do método de otimização que consegue atingir uma resolução de velocidade de 0,12 m/s. A comparação dos modelos de SVM mostra que o quarto modelo, utilizando kernel polinomial, apresenta os melhores resultados com uma acurácia de 99,5%. Entretanto, o tempo de processamento não é bom o suficiente, levando 72 ms para a classificação de um objeto. Palavras-Chaves: Carro autônomo. Reconhecimento de pedestres. Micro Doppler. Otimização multiobjectivos. Support vector machine.Abstract: The development of the autonomous car is nowadays a common practice in all the greatest automotive factories in the world, also in companies outside the automotive market, like Google and Apple. By adding more sensors, the vehicle is now capable of moving alone, identifying the correct path, the distance from another cars, also the presence of objects and people. However, there are still many issues blocking the autonomous car to be released. There are technical aspects to be solved, as the pedestrian recognition issues. Although, the recognition is widely studied and applied using cameras and digital images, there are issues to be improved. Like the distance and velocity reliability and the problems occurred because the lack of light in the environment. Based on the before mentioned, the focus in this presented work is to develop and discuss the efficiency of a pedestrian recognition system, using one automotive radar of 79 GHz. The main goal is to early detect the pedestrian using the micro Doppler characteristics of a human body in near to crash situations. Aiming this goal some techniques are used in the work. Firstly, the velocity resolution is improved, in order to extract the micro Doppler characteristics of the objects. The improvement of velocity resolution is reached by the use of multiobjective optimization techniques, as genetic algorithm and random search. The information about velocity and range is measured by the radar. In sequence a simple feature extraction method called video temporal gradient transform the data. The result is used in a machine learning technique called support vector machine (SVM). Which classifies the objects between pedestrians and non-pedestrians, with four different approaches. Concluding the work, it is possible to see the advantages of the multiobjective optimization in order to extract the micro Doppler effects. The optimization reached the velocity resolution of 0,12 m/s. The SVM comparison show that the fourth model with a polynomial kernel presented better result with accuracy 99,5%. However, the processing time of the system was not good enough taking 72 ms to identify an object. Keywords: Autonomous car. Pedestrian recognition. Micro Doppler. Multiobjective optimization. Support vector machine

Micro-Doppler-Coded Drone Identification

The forthcoming era of massive drone delivery deployment in urban environments raises a need to develop reliable control and monitoring systems. While active solutions, i.e., wireless sharing of a real-time location between air traffic participants and control units, are of use, developing additional security layers is appealing. Among various surveillance systems, radars offer distinct advantages by operating effectively in harsh weather conditions and providing high-resolution reliable detection over extended ranges. However, contrary to traditional airborne targets, small drones and copters pose a significant problem for radar systems due to their relatively small radar cross-sections. Here, we propose an efficient approach to label drones by attaching passive resonant scatterers to their rotor blades. While blades themselves generate micro-Doppler rotor-specific signatures, those are typically hard to capture at large distances owing to small signal-to-noise ratios in radar echoes. Furthermore, drones from the same vendor are indistinguishable by their micro-Doppler signatures. Here we demonstrate that equipping the blades with multiple resonant scatterers not only extends the drone detection range but also assigns it a unique micro-Doppler encoded identifier. By extrapolating the results of our laboratory and outdoor experiments to real high-grade radar surveillance systems, we estimate that the clear-sky identification range for a small drone is approximately 3-5 kilometers, whereas it would be barely detectable at 1000 meters if not labeled. This performance places the proposed passive system on par with its active counterparts, offering the clear benefits of reliability and resistance to jamming

Radar-Based Multi-Target Classification Using Deep Learning

Real-time, radar-based human activity and target recognition has several applications in various fields. Examples include hand gesture recognition, border and home surveillance, pedestrian recognition for automotive safety and fall detection for assisted living. This dissertation sought to improve the speed and accuracy of a previously developed model classifying human activity and targets using radar data for outdoor surveillance purposes. An improvement in accuracy and speed of classification helps surveillance systems to provide reliable results on time. For example, the results can be used to intercept trespassers, poachers or smugglers. To achieve these objectives, radar data was collected using a C-band pulse-Doppler radar and converted to spectrograms using the Short-time Fourier transform (STFT) algorithm. Spectrograms of the following classes were utilised in classification: one human walking, two humans walking, one human running, moving vehicles, a swinging sphere and clutter/noise. A seven-layer residual network was proposed, which utilised batch normalisation (BN), global average pooling (GAP), and residual connections to achieve a classification accuracy of 92.90% and 87.72% on the validation and test data, respectively. Compared to the previously proposed model, this represented a 10% improvement in accuracy on the validation data and a 3% improvement on the test data. Applying model quantisation provided up to 3.8 times speedup in inference, with a less than 0.4% accuracy drop on both the validation and test data. The quantised model could support a range of up to 89.91 kilometres in real-time, allowing it to be used in radars that operate within this range

Aprendizagem automática aplicada à deteção de pessoas baseada em radar

The present dissertation describes the development and implementation of a radar-based system with the purpose of being able to detect people amidst other objects that are moving in an indoor scenario. The detection methods implemented exploit radar data that is processed by a system that includes the data acquisition, the pre-processing of the data, the feature extraction, and the application of these data to machine learning models specifically designed to attain the objective of target classification. Beyond the basic theoretical research necessary for its sucessful development, the work contamplates an important component of software development and experimental tests. Among others, the following topics were covered in this dissertation: the study of radar working principles and hardware; radar signal processing; techniques of clutter removal, feature exctraction, and data clustering applied to radar signals; implementation and hyperparameter tuning of machine learning classification systems; study of multi-target detection and tracking methods. The people detection application was tested in different indoor scenarios that include a static radar and a radar dynamically deployed by a mobile robot. This application can be executed in real time and perform multiple target detection and classification using basic clustering and tracking algorithms. A study of the effects of the detection of multiple targets in the performance of the application, as well as an assessment of the efficiency of the different classification methods is presented. The envisaged applications of the proposed detection system include intrusion detection in indoor environments and acquisition of anonymized data for people tracking and counting in public spaces such as hospitals and schools.A presente dissertação descreve o desenvolvimento e implementação de um sistema baseado em radar que tem como objetivo detetar e distinguir pessoas de outros objetos que se movem num ambiente interior. Os métodos de deteção e distinção exploram os dados de radar que são processados por um sistema que abrange a aquisição e pré-processamento dos dados, a extração de características, e a aplicação desses dados a modelos de aprendizagem automática especificamente desenhados para atingir o objetivo de classificação de alvos. Além do estudo da teoria básica de radar para o desenvolvimento bem sucedido desta dissertação, este trabalho contempla uma componente importante de desenvolvimento de software e testes experimentais. Entre outros, os seguintes tópicos foram abordados nesta dissertação: o estudo dos princípios básicos do funcionamento do radar e do seu equipamento; processamento de sinal do radar; técnicas de remoção de ruído, extração de características, e segmentação de dados aplicada ao sinal de radar; implementação e calibração de hiper-parâmetros dos modelos de aprendizagem automática para sistemas de classificação; estudo de métodos de deteção e seguimento de múltiplos alvos. A aplicação para deteção de pessoas foi testada em diferentes cenários interiores que incluem o radar estático ou transportado por um robot móvel. Esta aplicação pode ser executada em tempo real e realizar deteção e classificação de múltiplos alvos usando algoritmos básicos de segmentação e seguimento. O estudo do impacto da deteção de múltiplos alvos no funcionamento da aplicação é apresentado, bem como a avaliação da eficiência dos diferentes métodos de classificação usados. As possíveis aplicações do sistema de deteção proposto incluem a deteção de intrusão em ambientes interiores e aquisição de dados anónimos para seguimento e contagem de pessoas em espaços públicos tais como hospitais ou escolas.Mestrado em Engenharia de Computadores e Telemátic

Radar-based Application of Pedestrian and Cyclist Micro-Doppler Signatures for Automotive Safety Systems

Die sensorbasierte Erfassung des Nahfeldes im Kontext des hochautomatisierten Fahrens erfährt einen spürbaren Trend bei der Integration von Radarsensorik. Fortschritte in der Mikroelektronik erlauben den Einsatz von hochauflösenden Radarsensoren, die durch effiziente Verfahren sowohl im Winkel als auch in der Entfernung und im Doppler die Messgenauigkeit kontinuierlich ansteigen lassen. Dadurch ergeben sich neuartige Möglichkeiten bei der Bestimmung der geometrischen und kinematischen Beschaffenheit ausgedehnter Ziele im Fahrzeugumfeld, die zur gezielten Entwicklung von automotiven Sicherheitssystemen herangezogen werden können. Im Rahmen dieser Arbeit werden ungeschützte Verkehrsteilnehmer wie Fußgänger und Radfahrer mittels eines hochauflösenden Automotive-Radars analysiert. Dabei steht die Erscheinung des Mikro-Doppler-Effekts, hervorgerufen durch das hohe Maß an kinematischen Freiheitsgraden der Objekte, im Vordergrund der Betrachtung. Die durch den Mikro-Doppler-Effekt entstehenden charakteristischen Radar-Signaturen erlauben eine detailliertere Perzeption der Objekte und können in direkten Zusammenhang zu ihren aktuellen Bewegungszuständen gesetzt werden. Es werden neuartige Methoden vorgestellt, die die geometrischen und kinematischen Ausdehnungen der Objekte berücksichtigen und echtzeitfähige Ansätze zur Klassifikation und Verhaltensindikation realisieren. Wird ein ausgedehntes Ziel (z.B. Radfahrer) von einem Radarsensor detektiert, können aus dessen Mikro-Doppler-Signatur wesentliche Eigenschaften bezüglich seines Bewegungszustandes innerhalb eines Messzyklus erfasst werden. Die Geschwindigkeitsverteilungen der sich drehenden Räder erlauben eine adaptive Eingrenzung der Tretbewegung, deren Verhalten essentielle Merkmale im Hinblick auf eine vorausschauende Unfallprädiktion aufweist. Ferner unterliegen ausgedehnte Radarziele einer Orientierungsabhängigkeit, die deren geometrischen und kinematischen Profile direkt beeinflusst. Dies kann sich sowohl negativ auf die Klassifikations-Performance als auch auf die Verwertbarkeit von Parametern auswirken, die eine Absichtsbekundung des Radarziels konstituieren. Am Beispiel des Radfahrers wird hierzu ein Verfahren vorgestellt, das die orientierungsabhängigen Parameter in Entfernung und Doppler normalisiert und die gemessenen Mehrdeutigkeiten kompensiert. Ferner wird in dieser Arbeit eine Methodik vorgestellt, die auf Grundlage des Mikro- Doppler-Profils eines Fußgängers dessen Beinbewegungen über die Zeit schätzt (Tracking) und wertvolle Objektinformationen hinsichtlich seines Bewegungsverhaltens offenbart. Dazu wird ein Bewegungsmodell entwickelt, das die nichtlineare Fortbewegung des Beins approximiert und dessen hohes Maß an biomechanischer Variabilität abbildet. Durch die Einbeziehung einer wahrscheinlichkeitsbasierten Datenassoziation werden die Radar-Detektionen ihren jeweils hervorrufenden Quellen (linkes und rechtes Bein) zugeordnet und eine Trennung der Gliedmaßen realisiert. Im Gegensatz zu bisherigen Tracking-Verfahren weist die vorgestellte Methodik eine Steigerung in der Genauigkeit der Objektinformationen auf und stellt damit einen entscheidenden Vorteil für zukünftige Fahrerassistenzsysteme dar, um deutlich schneller auf kritische Verkehrssituationen reagieren zu können.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 135Sensor-based detection of the near field in the context of highly automated driving is experiencing a noticeable trend in the integration of radar sensor technology. Advances in microelectronics allow the use of high-resolution radar sensors that continuously increase measurement accuracy through efficient processes in angle as well as distance and Doppler. This opens up novel possibilities in determining the geometric and kinematic nature of extended targets in the vehicle environment, which can be used for the specific development of automotive safety systems. In this work, vulnerable road users such as pedestrians and cyclists are analyzed using a high-resolution automotive radar. The focus is on the appearance of the micro-Doppler effect, caused by the objects’ high kinematic degree of freedom. The characteristic radar signatures produced by the micro-Doppler effect allow a clearer perception of the objects and can be directly related to their current state of motion. Novel methods are presented that consider the geometric and kinematic extents of the objects and realize real-time approaches to classification and behavioral indication. When a radar sensor detects an extended target (e.g., bicyclist), its motion state’s fundamental properties can be captured from its micro-Doppler signature within a measurement cycle. The spinning wheels’ velocity distributions allow an adaptive containment of the pedaling motion, whose behavior exhibits essential characteristics concerning predictive accident prediction. Furthermore, extended radar targets are subject to orientation dependence, directly affecting their geometric and kinematic profiles. This can negatively affect both the classification performance and the usability of parameters constituting the radar target’s intention statement. For this purpose, using the cyclist as an example, a method is presented that normalizes the orientation-dependent parameters in range and Doppler and compensates for the measured ambiguities. Furthermore, this paper presents a methodology that estimates a pedestrian’s leg motion over time (tracking) based on the pedestrian’s micro-Doppler profile and reveals valuable object information regarding his motion behavior. To this end, a motion model is developed that approximates the leg’s nonlinear locomotion and represents its high degree of biomechanical variability. By incorporating likelihood-based data association, radar detections are assigned to their respective evoking sources (left and right leg), and limb separation is realized. In contrast to previous tracking methods, the presented methodology shows an increase in the object information’s accuracy. It thus represents a decisive advantage for future driver assistance systems in order to be able to react significantly faster to critical traffic situations.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 13

Machine learning applied to radar data: classification and semantic instance segmentation of moving road users

Classification and semantic instance segmentation applications are rarely considered for automotive radar sensors. In current implementations, objects have to be tracked over time before some semantic information can be extracted. In this thesis, data from a network of 77 GHz automotive radar sensors is used to construct, train and evaluate machine learning algorithms for the classification of moving road users. The classification step is deliberately performed early in the process chain so that a subsequent tracking algorithm can benefit from this extra information. For this purpose, a large data set with real-world scenarios from about 5 h of driving was recorded and annotated. Given that the point clouds measured by the radar sensors are both sparse and noisy, the proposed methods have to be sensitive to those features that discern the individual classes from each other and at the same time, they have to be robust to outliers and measurement errors. Two groups of applications are considered: classi- fication of clustered data and semantic (instance) segmentation of whole scenes. In the first category, specifically designed density-based clustering algorithms are used to group individual measurements to objects. These objects are then used either as input to a manual feature extraction step or as input to a neural network, which operates directly on the bare input points. Different classifiers are trained and evaluated on these input data. For the algorithms of the second category, the measurements of a whole scene are used as input, so that the clustering step becomes obsolete. A newly designed recurrent neural network for instance segmentation of point clouds is utilized. This approach outperforms all of the other proposed methods and exceeds the baseline score by about ten percentage points. In additional experiments, the performance of human test candidates on the same task is analyzed. This study shows that temporal correlations in the data are of great use for the test candidates, who are nevertheless outrun by the recurrent network

Edge Artificial Intelligence for Real-Time Target Monitoring

The key enabling technology for the exponentially growing cellular communications sector is location-based services. The need for location-aware services has increased along with the number of wireless and mobile devices. Estimation problems, and particularly parameter estimation, have drawn a lot of interest because of its relevance and engineers' ongoing need for higher performance. As applications expanded, a lot of interest was generated in the accurate assessment of temporal and spatial properties. In the thesis, two different approaches to subject monitoring are thoroughly addressed. For military applications, medical tracking, industrial workers, and providing location-based services to the mobile user community, which is always growing, this kind of activity is crucial. In-depth consideration is given to the viability of applying the Angle of Arrival (AoA) and Receiver Signal Strength Indication (RSSI) localization algorithms in real-world situations. We presented two prospective systems, discussed them, and presented specific assessments and tests. These systems were put to the test in diverse contexts (e.g., indoor, outdoor, in water...). The findings showed the localization capability, but because of the low-cost antenna we employed, this method is only practical up to a distance of roughly 150 meters. Consequently, depending on the use-case, this method may or may not be advantageous. An estimation algorithm that enhances the performance of the AoA technique was implemented on an edge device. Another approach was also considered. Radar sensors have shown to be durable in inclement weather and bad lighting conditions. Frequency Modulated Continuous Wave (FMCW) radars are the most frequently employed among the several sorts of radar technologies for these kinds of applications. Actually, this is because they are low-cost and can simultaneously provide range and Doppler data. In comparison to pulse and Ultra Wide Band (UWB) radar sensors, they also need a lower sample rate and a lower peak to average ratio. The system employs a cutting-edge surveillance method based on widely available FMCW radar technology. The data processing approach is built on an ad hoc-chain of different blocks that transforms data, extract features, and make a classification decision before cancelling clutters and leakage using a frame subtraction technique, applying DL algorithms to Range-Doppler (RD) maps, and adding a peak to cluster assignment step before tracking targets. In conclusion, the FMCW radar and DL technique for the RD maps performed well together for indoor use-cases. The aforementioned tests used an edge device and Infineon Technologies' Position2Go FMCW radar tool-set

A New Form of Interlocking Developing Technology for Level Crossings and Depots with International Applications

There are multiple large rail infrastructure projects planned or currently being undertaken within the United Kingdom. Many of these projects aim to reduce the continual issue of limited or overcapacity service. These projects involve an expansion of Rail lines, introducing faster lines, improved stations in towns and cities and better communication networks. Some major projects like Control Period 6 (CP6) are being managed by Network Rail; where projects are initiated throughout Great Britain. Many projects are managed outside Great Britain e.g., Trans-European Transport Network Program, which is planning for expansion of Rail lines (almost double) for High-Speed Rails (category I and II). These projects will increase the number of junctions and Level Crossings. A Level Crossing is where a Rail Line is crossed by a road or a walkway without the use of a tunnel or bridge. The misuse from the road users account for nearly 90% of the fatalities and near misses at Level Crossings. During 2016/2017, the Rail Network recorded 6 fatalities, about 400 near-misses and more than 77 incidents of shock and trauma. Accidents at Level Crossings represent 8% of the total accidents from the whole Rail Network. Office of Rail and Road (ORR) suggested that among these accidents at Level Crossings 90% of them are pedestrians. Such high numbers of accidents, fatalities and high risk have alarmed authorities. These authorities found it necessary to invest time and utilise given resources to improve the safety system at a Level Crossing using the safer and reliable interlocking system. The interlocking system is a feature of a control system that makes the state of two functions mutually independent. The primary function of Interlocking is to ensure that trains are safe from collision and derailment. Considering the risk associated with the Level Crossing system, the new proposed interlocking system should utilise the sensing system available at a Level Crossing to significantly reduce implementation cost and comply with the given standards and Risk Assessments. The new proposed interlocking system is designed to meet the “Safety Integrity Level- SIL” and possibly use the “2oo2” approach for its application at a Level Crossing, where the operational cycle is automated or train driver is alarmed for risk situations. Importantly, the new proposed system should detect and classify small objects and provide a reasonable solution to the current risk associated with Level Crossing, which was impossible with the traditional sensing systems. The present work discusses the sensors and algorithms used and has the potential to detect and classify objects within a Level Crossing area. The review of existing solutions e.g Inductive Loops and other major sensors allows the reader to understand why RADAR and Video Cameras are preferable choices of a sensing system for a Level Crossing. Video data provides sufficient information for the proposed algorithm to detect and classify objects at Level Crossings without the need of a manual “operator”. The RADAR sensing system can provide information using micro-Doppler signatures, which are generated from small regular movements of an obstacle. The two sensors will make the system a two-layer resilient system. The processed information from these two sensing systems is used as the “2oo2” logic system for Interlocking for automating the operational cycle or alarm the train drive using effective communication e.g., GSM-R. These two sensors provide sufficient information for the proposed algorithm, which will allow the system to automatically make an “intelligent decision” and proceed with a safe Level Crossing operational cycle. Many existing traditional algorithms depend on pixels values, which are compared with background pixels. This approach cannot detect complex textures, adapt to a dynamic background or avoid detection of unnecessary harmless objects. To avoid these problems, the proposed work utilises “Deep Learning” technology integrated with the proposed Vision and RADAR system. The Deep Learning technology can learn representations from labelled pixels; hence it does not depend on background pixels. The Deep 3 | P a g e Learning technology can classify, detect and localise objects at a Level Crossing area. It can classify and differentiate between a child and a small inanimate object, which was impossible with traditional algorithms. The system can detect an object regardless of its position, orientation and scale without any additional training because it learns representation from the data and does not rely on background pixels. The proposed system e.g., Deep Learning technology is integrated with the existing Vision System and RADAR installed at a Level Crossing, hence implementation cost is significantly reduced as well. The proposed work address two main aspects of training a model using Deep Learning technology; training from scratch and training using Transfer Learning techniques. Results are demonstrated for Image Classification, Object Detection and micro-Doppler signals from RADAR. An architecture of Convolutional Neural Network from scratch is trained consisting of Input Layer, Convolution, Pooling and Dropout Layer. The model achieves an accuracy of about 66.78%. Different notable models are trained using Transfer Learning techniques and their results are mentioned along with the MobileNet model, which achieves the highest accuracy of 91.9%. The difference between Image Classification and Object Detection is discussed and results for Object Detection are mentioned as well, where the Loss metrics are used to evaluate the performance of the Object Detector. MobileNet achieves the smallest loss metric of about 0.092. These results clearly show the effectiveness and preferability of these models for their applicability at Level Crossings. Another Convolutional Neural Network is trained using micro-Doppler signatures from the Radar system. The model trained using the micro-Doppler signature achieved an accuracy of 92%. The present work also addresses the Risk Assessment associated with the installation and maintenance of the system using Deep Learning technology. RAMS (Reliability, Availability, Maintainability and Safety) management system is used to address the General and Specific Risks associated with the sensing system integrated with the Deep Learning technology. Finally, the work is concluded with the preferred choice, its application, results and associated Risk Assessment. Deep Learning is an evolving field with new improvements being introduced constantly. Any new challenges and problems should be monitored regularly. Some future work is discussed as well. To further improve the model's accuracy, the dataset from the same distribution should be gathered with the cooperation of relevant Railway authorities. Also, the RADAR dataset could be generated rather than simulated to further include diversity and avoid any biases in the dataset during the training process. Also, the proposed system can be implemented and used in different applications within the Rail Industry e.g., passenger census and classification of passengers at the platform as discussed in the work

SDR-GAIN: A High Real-Time Occluded Pedestrian Pose Completion Method for Autonomous Driving

To mitigate the challenges arising from partial occlusion in human pose keypoint based pedestrian detection methods , we present a novel pedestrian pose keypoint completion method called the separation and dimensionality reduction-based generative adversarial imputation networks (SDR-GAIN) . Firstly, we utilize OpenPose to estimate pedestrian poses in images. Then, we isolate the head and torso keypoints of pedestrians with incomplete keypoints due to occlusion or other factors and perform dimensionality reduction to enhance features and further unify feature distribution. Finally, we introduce two generative models based on the generative adversarial networks (GAN) framework, which incorporate Huber loss, residual structure, and L1 regularization to generate missing parts of the incomplete head and torso pose keypoints of partially occluded pedestrians, resulting in pose completion. Our experiments on MS COCO and JAAD datasets demonstrate that SDR-GAIN outperforms basic GAIN framework, interpolation methods PCHIP and MAkima, machine learning methods k-NN and MissForest in terms of pose completion task. In addition, the runtime of SDR-GAIN is approximately 0.4ms, displaying high real-time performance and significant application value in the field of autonomous driving