People tracking by cooperative fusion of RADAR and camera sensors

Accurate 3D tracking of objects from monocular camera poses challenges due to the loss of depth during projection. Although ranging by RADAR has proven effective in highway environments, people tracking remains beyond the capability of single sensor systems. In this paper, we propose a cooperative RADAR-camera fusion method for people tracking on the ground plane. Using average person height, joint detection likelihood is calculated by back-projecting detections from the camera onto the RADAR Range-Azimuth data. Peaks in the joint likelihood, representing candidate targets, are fed into a Particle Filter tracker. Depending on the association outcome, particles are updated using the associated detections (Tracking by Detection), or by sampling the raw likelihood itself (Tracking Before Detection). Utilizing the raw likelihood data has the advantage that lost targets are continuously tracked even if the camera or RADAR signal is below the detection threshold. We show that in single target, uncluttered environments, the proposed method entirely outperforms camera-only tracking. Experiments in a real-world urban environment also confirm that the cooperative fusion tracker produces significantly better estimates, even in difficult and ambiguous situations

Keep off the Grass: Permissible Driving Routes from Radar with Weak Audio Supervision

Reliable outdoor deployment of mobile robots requires the robust identification of permissible driving routes in a given environment. The performance of LiDAR and vision-based perception systems deteriorates significantly if certain environmental factors are present e.g. rain, fog, darkness. Perception systems based on FMCW scanning radar maintain full performance regardless of environmental conditions and with a longer range than alternative sensors. Learning to segment a radar scan based on driveability in a fully supervised manner is not feasible as labelling each radar scan on a bin-by-bin basis is both difficult and time-consuming to do by hand. We therefore weakly supervise the training of the radar-based classifier through an audio-based classifier that is able to predict the terrain type underneath the robot. By combining odometry, GPS and the terrain labels from the audio classifier, we are able to construct a terrain labelled trajectory of the robot in the environment which is then used to label the radar scans. Using a curriculum learning procedure, we then train a radar segmentation network to generalise beyond the initial labelling and to detect all permissible driving routes in the environment.Comment: accepted for publication at the IEEE Intelligent Transportation Systems Conference (ITSC) 202

Radar based on automotive pedestrian detection using the micro Doppler effects

Orientador: Prof. Dr. Alessandro ZimmerDissertação (mestrado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa : Curitiba, 29/08/2018Inclui referências: p.74-78Resumo: O desenvolvimento do carro autônomo é hoje em dia uma prática comum entre as maiores indústrias automotivas, e também em indústrias tecnológicas, como o Google e a Apple. Ao adicionar mais sensores, o veículo é capaz de se movimentar sozinho, identificar a trajetória correta, a distância para outros carros, e também a presença de objetos e seres vivos. Entretanto, existem muitos aspectos bloqueando o lançamento do carro autônomo. Como exemplo aspectos técnicos, como o caso do reconhecimento de pedestres. Embora, esse tópico seja abundantemente estudado para o uso de câmeras digitais, as mesmas não possuem confiabilidade nas medições de velocidade e distância, e ainda apresentam péssimos resultados quando há variação ou a falta de luz no ambiente. Baseado no que foi mencionado anteriormente, o foco dessa dissertação é de desenvolver e discutir a eficiência de um sistema de rápida identificação de pedestres, utilizando um novo radar de 79GHz de frequência. O principal objetivo é reconhecer o pedestre o mais rápido possível utilizando os efeitos micro Doppler do movimento humano em situações muito próximas de um acidente, junto com o método de classificação support vector machine (SVM). Objetivando essa meta algumas técnicas são usadas ao longo do trabalho. Primeiramente, a resolução de velocidade é melhorada com técnicas de otimização multiobjetivos, como algoritmos genéticos e random search para extrair o micro efeito Doppler. Então as informações de velocidade e distância são medidas pelo radar. Em sequência, um método de extração de características chamado de video temporal gradiente é aplicado. O método de machine learning SVM classifica os objetos em pedestre e não pedestres, com quadro diferentes métodos de treinamento. Por fim, é possível ver as vantagens do método de otimização que consegue atingir uma resolução de velocidade de 0,12 m/s. A comparação dos modelos de SVM mostra que o quarto modelo, utilizando kernel polinomial, apresenta os melhores resultados com uma acurácia de 99,5%. Entretanto, o tempo de processamento não é bom o suficiente, levando 72 ms para a classificação de um objeto. Palavras-Chaves: Carro autônomo. Reconhecimento de pedestres. Micro Doppler. Otimização multiobjectivos. Support vector machine.Abstract: The development of the autonomous car is nowadays a common practice in all the greatest automotive factories in the world, also in companies outside the automotive market, like Google and Apple. By adding more sensors, the vehicle is now capable of moving alone, identifying the correct path, the distance from another cars, also the presence of objects and people. However, there are still many issues blocking the autonomous car to be released. There are technical aspects to be solved, as the pedestrian recognition issues. Although, the recognition is widely studied and applied using cameras and digital images, there are issues to be improved. Like the distance and velocity reliability and the problems occurred because the lack of light in the environment. Based on the before mentioned, the focus in this presented work is to develop and discuss the efficiency of a pedestrian recognition system, using one automotive radar of 79 GHz. The main goal is to early detect the pedestrian using the micro Doppler characteristics of a human body in near to crash situations. Aiming this goal some techniques are used in the work. Firstly, the velocity resolution is improved, in order to extract the micro Doppler characteristics of the objects. The improvement of velocity resolution is reached by the use of multiobjective optimization techniques, as genetic algorithm and random search. The information about velocity and range is measured by the radar. In sequence a simple feature extraction method called video temporal gradient transform the data. The result is used in a machine learning technique called support vector machine (SVM). Which classifies the objects between pedestrians and non-pedestrians, with four different approaches. Concluding the work, it is possible to see the advantages of the multiobjective optimization in order to extract the micro Doppler effects. The optimization reached the velocity resolution of 0,12 m/s. The SVM comparison show that the fourth model with a polynomial kernel presented better result with accuracy 99,5%. However, the processing time of the system was not good enough taking 72 ms to identify an object. Keywords: Autonomous car. Pedestrian recognition. Micro Doppler. Multiobjective optimization. Support vector machine

Vision based environment perception system for next generation off-road ADAS : innovation report

Advanced Driver Assistance Systems (ADAS) aids the driver by providing information or automating the driving related tasks to improve driver comfort, reduce workload and improve safety. The vehicle senses its external environment using sensors, building a representation of the world used by the control systems. In on-road applications, the perception focuses on establishing the location of other road participants such as vehicles and pedestrians and identifying the road trajectory. Perception in the off-road environment is more complex, as the structure found in urban environments is absent. Off-road perception deals with the estimation of surface topography and surface type, which are the factors that will affect vehicle behaviour in unstructured environments. Off-road perception has seldom been explored in automotive context. For autonomous off-road driving, the perception solutions are primarily related to robotics and not directly applicable in the ADAS domain due to the different goals of unmanned autonomous systems, their complexity and the cost of employed sensors. Such applications consider only the impact of the terrain on the vehicle safety and progress but do not account for the driver comfort and assistance. This work addresses the problem of processing vision sensor data to extract the required information about the terrain. The main focus of this work is on the perception task with the constraints of automotive sensors and the requirements of the ADAS systems. By providing a semantic representation of the off-road environment including terrain attributes such as terrain type, description of the terrain topography and surface roughness, the perception system can cater for the requirements of the next generation of off-road ADAS proposed by Land Rover. Firstly, a novel and computationally efficient terrain recognition method was developed. The method facilitates recognition of low friction grass surfaces in real-time with high accuracy, by applying machine learning Support Vector Machine with illumination invariant normalised RGB colour descriptors. The proposed method was analysed and its performance was evaluated experimentally in off-road environments. Terrain recognition performance was evaluated on a variety of different surface types including grass, gravel and tarmac, showing high grass detection performance with accuracy of 97%. Secondly, a terrain geometry identification method was proposed which facilitates semantic representation of the terrain in terms of macro terrain features such as slopes, crest and ditches. The terrain geometry identification method processes 3D information reconstructed from stereo imagery and constructs a compact grid representation of the surface topography. This representation is further processed to extract object representation of slopes, ditches and crests. Thirdly, a novel method for surface roughness identification was proposed. The surface roughness descriptor is then further used to recommend a vehicle velocity, which will maintain passenger comfort. Surface roughness is described by the Power Spectral Density of the surface profile which correlates with the acceleration experienced by the vehicle. The surface roughness descriptor is then mapped onto vehicle speed recommendation so that the speed of the vehicle can be adapted in anticipation of the surface roughness. Terrain geometry and surface roughness identification performance were evaluated on a range of off-road courses with varying topology showing the capability of the system to correctly identify terrain features up to 20 m ahead of the vehicle and analyse surface roughness up to 15 m ahead of the vehicle. The speed was recommended correctly within +/- 5 kph. Further, the impact of the perception system on the speed adaptation was evaluated, showing the improvements in speed adaptation allowing for greater passenger comfort. The developed perception components facilitated the development of new off-road ADAS systems and were successfully applied in prototype vehicles. The proposed off-road ADAS are planned to be introduced in future generations of Land Rover products. The benefits of this research also included new Intellectual Property generated for Jaguar Land Rover. In the wider context, the enhanced off-road perception capability may facilitate further development of off-road automated driving and off-road autonomy within the constraints of the automotive platfor