Predictive Techniques for Scene Understanding by using Deep Learning in Autonomous Driving

La conducción autónoma es considerada uno de los más grandes retos tecnológicos de la actualidad. Cuando los coches autónomos conquisten nuestras carreteras, los accidentes se reducirán notablemente, hasta casi desaparecer, ya que la tecnología estará testada y no incumplirá las normas de conducción, entre otros beneficios sociales y económicos. Uno de los aspectos más críticos a la hora de desarrollar un vehículo autónomo es percibir y entender la escena que le rodea. Esta tarea debe ser tan precisa y eficiente como sea posible para posteriormente predecir el futuro de esta misma y ayudar a la toma de decisiones. De esta forma, las acciones tomadas por el vehículo garantizarán tanto la seguridad del vehículo en sí mismo y sus ocupantes, como la de los obstáculos circundantes, tales como viandantes, otros vehículos o infraestructura de la carretera. En ese sentido, esta tesis doctoral se centra en el estudio y desarrollo de distintas técnicas predictivas para el entendimiento de la escena en el contexto de la conducción autónoma. Durante la tesis, se observa una incorporación progresiva de técnicas de aprendizaje profundo en los distintos algoritmos propuestos para mejorar el razonamiento sobre qué está ocurriendo en el escenario de tráfico, así como para modelar las complejas interacciones entre la información social (distintos participantes o agentes del escenario, tales como vehículos, ciclistas o peatones) y física (es decir, la información geométrica, semántica y topológica del mapa de alta definición) presente en la escena. La capa de percepción de un vehículo autónomo se divide modularmente en tres etapas: Detección, Seguimiento (Tracking), y Predicción. Para iniciar el estudio de las etapas de seguimiento y predicción, se propone un algoritmo de Multi-Object Tracking basado en técnicas clásicas de estimación de movimiento y asociación validado en el dataset KITTI, el cual obtiene métricas del estado del arte. Por otra parte, se propone el uso de un filtro inteligente basado en información contextual de mapa, cuyo objetivo es monitorizar los agentes más relevantes de la escena en el tiempo, representando estos agentes filtrados la entrada preliminar para realizar predicciones unimodales basadas en un modelo cinemático. Para validar esta propuesta de filtro inteligente se usa CARLA (CAR Learning to Act), uno de los simuladores hiperrealistas para conducción autónoma más prometedores en la actualidad, comprobando cómo al usar información contextual de mapa se puede reducir notablemente el tiempo de inferencia de un algoritmo de tracking y predicción basados en métodos físicos, prestando atención a los agentes realmente relevantes del escenario de tráfico. Tras observar las limitaciones de un modelo de predicción basado en cinemática para la predicción a largo plazo de un agente, los distintos algoritmos de la tesis se centran en el módulo de predicción, usando los datasets Argoverse 1 y Argoverse 2, donde se asume que los agentes proporcionados en cada escenario de tráfico ya están monitorizados durante un cierto número de observaciones. En primer lugar, se introduce un modelo basado en redes neuronales recurrentes (particularmente redes LSTM, Long-Short Term Memory) y mecanismo de atención para codificar las trayectorias pasadas de los agentes, y una representación simplificada del mapa en forma de posiciones finales potenciales en la carretera para calcular las trayectorias futuras unimodales, todo envuelto en un marco GAN (Generative Adversarial Network), obteniendo métricas similares al estado del arte en el caso unimodal. Una vez validado el modelo anterior en Argoverse 1, se proponen distintos modelos base (sólo social, incorporando mapa, y una mejora final basada en Transformer encoder, redes convolucionales 1D y mecanismo de atención cruzada para la fusión de características) precisos y eficientes basados en el modelo de predicción anterior, introduciendo dos nuevos conceptos. Por un lado, el uso de redes neuronales gráficas (particularmente GCN, Graph Convolutional Network) para codificar de una forma potente las interacciones de los agentes. Por otro lado, se propone el preprocesamiento de trayectorias preliminares a partir de un mapa con un método heurístico. Gracias a estas entradas y una arquitectura más potente de codificación, los modelos base serán capaces de predecir distintas trayectorias futuras multimodales, es decir, cubriendo distintos posibles futuros para el agente de interés. Los modelos base propuestos obtienen métricas de regresión del estado del arte tanto en el caso multimodal como unimodal manteniendo un claro compromiso de eficiencia con respecto a otras propuestas. El modelo final de la tesis, inspirado en los modelos anteriores y validado en el más reciente dataset para algoritmos de predicción en conducción autónoma (Argoverse 2), introduce varias mejoras para entender mejor el escenario de tráfico y decodificar la información de una forma precisa y eficiente. Se propone incorporar información topológica y semántica de los carriles futuros preliminares con el método heurístico antes mencionado, codificación de mapa basada en aprendizaje profundo con redes GCN, ciclo de fusión de características físicas y sociales, estimación de posiciones finales en la carretera y agregación de su entorno circundante con aprendizaje profundo y finalmente módulo de refinado para mejorar la calidad de las predicciones multimodales finales de un modo elegante y eficiente. Comparado con el estado del arte, nuestro método logra métricas de predicción a la par con los métodos mejor posicionados en el Leaderboard de Argoverse 2, reduciendo de forma notable el número de parámetros y operaciones de coma flotante por segundo. Por último, el modelo final de la tesis ha sido validado en simulación en distintas aplicaciones de conducción autónoma. En primer lugar, se integra el modelo para proporcionar predicciones a un algoritmo de toma de decisiones basado en aprendizaje por refuerzo en el simulador SMARTS (Scalable Multi-Agent Reinforcement Learning Training School), observando en los estudios como el vehículo es capaz de tomar mejores decisiones si conoce el comportamiento futuro de la escena y no solo el estado actual o pasado de esta misma. En segundo lugar, se ha realizado un estudio de adaptación de dominio exitoso en el simulador hiperrealista CARLA en distintos escenarios desafiantes donde el entendimiento de la escena y predicción del entorno son muy necesarios, como una autopista o rotonda con gran densidad de tráfico o la aparición de un usuario vulnerable de la carretera de forma repentina. En ese sentido, el modelo de predicción ha sido integrado junto con el resto de capas de la arquitectura de navegación autónoma del grupo de investigación donde se desarrolla la tesis como paso previo a su implementación en un vehículo autónomo real

Automotive applications of high precision GNSS

This thesis aims to show that Global Navigation Satellite Systems (GNSS) positioning can play a significant role in the positioning systems of future automotive applications. This is through the adoption of state-of-the-art GNSS positioning technology and techniques, and the exploitation of the rapidly developing vehicle-to-vehicle concept. The merging together of these two developments creates greater performance than can be achieved separately. The original contribution of this thesis comes from this combination: Through the introduction of the Pseudo-VRS concept. Pseudo-VRS uses the princples of Network Real Time Kinematic (N-RTK) positioning to share GNSS information between vehicles, which enables absolute vehicle positioning. Pseudo-VRS is shown to improve the performance of high precision GNSS positioning for road vehicles, through the increased availability of GNSS correction messages and the rapid resolution of the N-RTK fixed solution. Positioning systems in the automotive sector are dominated by satellite-based solutions provided by GNSS. This has been the case since May 2001, when the United States Department of Defense switched off Selective Availability, enabling significantly improved positioning performance for civilian users. The average person most frequently encounters GNSS when using electronic personal navigation devices. The Sat Nav or GPS Navigator is ubiquitous in modern societies, where versions can be found on nomadic devices such as smartphones and dedicated personal navigation devices, or built in to the dashboards of vehicles. Such devices have been hugely successful due to their intrinsic ability to provide position information anywhere in the world with an accuracy of approximately 10 metres, which has proved ideal for general navigation applications. There are a few well known limitations of GNSS positioning, including anecdotal evidence of incorrect navigation advice for personal navigation devices, but these are minor compared to the overall positioning performance. Through steady development of GNSS positioning devices, including the integration of other low cost sensors (for instance, wheel speed or odometer sensors in vehicles), and the development of robust map matching algorithms, the performance of these devices for navigation applications is truly incredible. However, when tested for advanced automotive applications, the performance of GNSS positioning devices is found to be inadequate. In particular, in the most advanced fields of research such as autonomous vehicle technology, GNSS positioning devices are relegated to a secondary role, or often not used at all. They are replaced by terrestrial sensors that provide greater situational awareness, such as radar and lidar. This is due to the high performance demand of such applications, including high positioning accuracy (sub-decimetre), high availability and continuity of solutions (100%), and high integrity of the position information. Low-cost GNSS receivers generally do not meet such requirements. This could be considered an enormous oversight, as modern GNSS positioning technology and techniques have significantly improved satellite-based positioning performance. Other non-GNSS techniques also have their limitations that GNSS devices can minimise or eliminate. For instance, systems that rely on situational awareness require accurate digital maps of their surroundings as a reference. GNSS positioning can help to gather this data, provide an input, and act as a fail-safe in the event of digital map errors. It is apparent that in order to deliver advanced automotive applications - such as semi- or fully-autonomous vehicles - there must be an element of absolute positioning capability. Positioning systems will work alongside situational awareness systems to enable the autonomous vehicles to navigate through the real world. A strong candidate for the positioning system is GNSS positioning. This thesis builds on work already started by researchers at the University of Nottingham, to show that N-RTK positioning is one such technique. N-RTK can provide sub-decimetre accuracy absolute positioning solutions, with high availability, continuity, and integrity. A key component of N-RTK is the availability of real-time GNSS correction data. This is typically delivered to the GNSS receiver via mobile internet (for a roving receiver). This can be a significant limitation, as it relies on the performance of the mobile communications network, which can suffer from performance degradation during dynamic operation. Mobile communications systems are expected to improve significantly over the next few years, as consumers demand faster download speeds and wider availability. Mobile communications coverage already covers a high percentage of the population, but this does not translate into a high percentage of a country's geography. Pockets of poor coverage, often referred to as notspots, are widespread. Many of these notspots include the transportation infrastructure. The vehicle-to-vehicle concept has made significant forward steps in the last few years. Traditionally promoted as a key component of future automotive safety applications, it is now driven primarily by increased demand for in-vehicle infotainment. The concept, which shares similarities with the Internet of Things and Mobile Ad-hoc Networks, relies on communication between road vehicles and other road agents (such as pedestrians and road infrastructure). N-RTK positioning can take advantage of this communication link to minimise its own communications-related limitations. Sharing GNSS information between local GNSS receivers enables better performance of GNSS positioning, based on the principles of differential GNSS and N-RTK positioning techniques. This advanced concept is introduced and tested in this thesis. The Pseudo VRS concept follows the protocols and format of sharing GNSS data used in N-RTK positioning. The technique utilises the latest GNSS receiver design, including multiple frequency measurements and high quality antennas

2nd Symposium on Management of Future motorway and urban Traffic Systems (MFTS 2018): Booklet of abstracts: Ispra, 11-12 June 2018

The Symposium focuses on future traffic management systems, covering the subjects of traffic control, estimation, and modelling of motorway and urban networks, with particular emphasis on the presence of advanced vehicle communication and automation technologies. As connectivity and automation are being progressively introduced in our transport and mobility systems, there is indeed a growing need to understand the implications and opportunities for an enhanced traffic management as well as to identify innovative ways and tools to optimise traffic efficiency. In particular the debate on centralised versus decentralised traffic management in the presence of connected and automated vehicles has started attracting the attention of the research community. In this context, the Symposium provides a remarkable opportunity to share novel ideas and discuss future research directions.JRC.C.4-Sustainable Transpor

Naturalistic Driver Intention and Path Prediction using Machine Learning

Autonomous vehicles are still yet to be available to the public. This is because there are a number of challenges that have not been overcome to ensure that autonomous vehicles can safely and efficiently drive on public roads. Accurate prediction of other vehicles is vital for safe driving, as interacting with other vehicles is unavoidable on public streets. This thesis explores reasons why this problem of scene understanding is still unsolved, and presents methods for driver intention and path prediction. The thesis focuses on intersections, as this is a very complex scenario in which to predict the actions of human drivers. There is very limited data available for intersection studies from the perspective of an autonomous vehicle. This thesis presents a very large dataset of over 23,000 vehicle trajectories, used to validate the algorithms presented in this thesis. This dataset was collected using a lidar based vehicle detection and tracking system onboard a vehicle. Analytics of this data is presented. To determine the intent of vehicle at an intersection, a method for manoeuvre classification through the use of recurrent neural networks is presented. This allows accurate predictions of which destination a vehicle will take at an unsignalised intersection, based on that vehicle's approach. The final contribution of this thesis presents a method for driver path prediction, based on recurrent neural networks. It produces a multi-modal prediction for the vehicle’s path with uncertainty assigned to each mode. The output modes are not hand labelled, but instead learned from the data. This results in there not being a fixed number of output modes. Whilst the application of this method is vehicle prediction, this method shows significant promise to be used in other areas of robotics

Infrastructure Design, Signalling and Security in Railway

Railway transportation has become one of the main technological advances of our society. Since the first railway used to carry coal from a mine in Shropshire (England, 1600), a lot of efforts have been made to improve this transportation concept. One of its milestones was the invention and development of the steam locomotive, but commercial rail travels became practical two hundred years later. From these first attempts, railway infrastructures, signalling and security have evolved and become more complex than those performed in its earlier stages. This book will provide readers a comprehensive technical guide, covering these topics and presenting a brief overview of selected railway systems in the world. The objective of the book is to serve as a valuable reference for students, educators, scientists, faculty members, researchers, and engineers

A prototype energy management system for a solar powered cycle

Bibliography: leaves 92-93

Automotive applications of high precision GNSS

This thesis aims to show that Global Navigation Satellite Systems (GNSS) positioning can play a significant role in the positioning systems of future automotive applications. This is through the adoption of state-of-the-art GNSS positioning technology and techniques, and the exploitation of the rapidly developing vehicle-to-vehicle concept. The merging together of these two developments creates greater performance than can be achieved separately. The original contribution of this thesis comes from this combination: Through the introduction of the Pseudo-VRS concept. Pseudo-VRS uses the princples of Network Real Time Kinematic (N-RTK) positioning to share GNSS information between vehicles, which enables absolute vehicle positioning. Pseudo-VRS is shown to improve the performance of high precision GNSS positioning for road vehicles, through the increased availability of GNSS correction messages and the rapid resolution of the N-RTK fixed solution. Positioning systems in the automotive sector are dominated by satellite-based solutions provided by GNSS. This has been the case since May 2001, when the United States Department of Defense switched off Selective Availability, enabling significantly improved positioning performance for civilian users. The average person most frequently encounters GNSS when using electronic personal navigation devices. The Sat Nav or GPS Navigator is ubiquitous in modern societies, where versions can be found on nomadic devices such as smartphones and dedicated personal navigation devices, or built in to the dashboards of vehicles. Such devices have been hugely successful due to their intrinsic ability to provide position information anywhere in the world with an accuracy of approximately 10 metres, which has proved ideal for general navigation applications. There are a few well known limitations of GNSS positioning, including anecdotal evidence of incorrect navigation advice for personal navigation devices, but these are minor compared to the overall positioning performance. Through steady development of GNSS positioning devices, including the integration of other low cost sensors (for instance, wheel speed or odometer sensors in vehicles), and the development of robust map matching algorithms, the performance of these devices for navigation applications is truly incredible. However, when tested for advanced automotive applications, the performance of GNSS positioning devices is found to be inadequate. In particular, in the most advanced fields of research such as autonomous vehicle technology, GNSS positioning devices are relegated to a secondary role, or often not used at all. They are replaced by terrestrial sensors that provide greater situational awareness, such as radar and lidar. This is due to the high performance demand of such applications, including high positioning accuracy (sub-decimetre), high availability and continuity of solutions (100%), and high integrity of the position information. Low-cost GNSS receivers generally do not meet such requirements. This could be considered an enormous oversight, as modern GNSS positioning technology and techniques have significantly improved satellite-based positioning performance. Other non-GNSS techniques also have their limitations that GNSS devices can minimise or eliminate. For instance, systems that rely on situational awareness require accurate digital maps of their surroundings as a reference. GNSS positioning can help to gather this data, provide an input, and act as a fail-safe in the event of digital map errors. It is apparent that in order to deliver advanced automotive applications - such as semi- or fully-autonomous vehicles - there must be an element of absolute positioning capability. Positioning systems will work alongside situational awareness systems to enable the autonomous vehicles to navigate through the real world. A strong candidate for the positioning system is GNSS positioning. This thesis builds on work already started by researchers at the University of Nottingham, to show that N-RTK positioning is one such technique. N-RTK can provide sub-decimetre accuracy absolute positioning solutions, with high availability, continuity, and integrity. A key component of N-RTK is the availability of real-time GNSS correction data. This is typically delivered to the GNSS receiver via mobile internet (for a roving receiver). This can be a significant limitation, as it relies on the performance of the mobile communications network, which can suffer from performance degradation during dynamic operation. Mobile communications systems are expected to improve significantly over the next few years, as consumers demand faster download speeds and wider availability. Mobile communications coverage already covers a high percentage of the population, but this does not translate into a high percentage of a country's geography. Pockets of poor coverage, often referred to as notspots, are widespread. Many of these notspots include the transportation infrastructure. The vehicle-to-vehicle concept has made significant forward steps in the last few years. Traditionally promoted as a key component of future automotive safety applications, it is now driven primarily by increased demand for in-vehicle infotainment. The concept, which shares similarities with the Internet of Things and Mobile Ad-hoc Networks, relies on communication between road vehicles and other road agents (such as pedestrians and road infrastructure). N-RTK positioning can take advantage of this communication link to minimise its own communications-related limitations. Sharing GNSS information between local GNSS receivers enables better performance of GNSS positioning, based on the principles of differential GNSS and N-RTK positioning techniques. This advanced concept is introduced and tested in this thesis. The Pseudo VRS concept follows the protocols and format of sharing GNSS data used in N-RTK positioning. The technique utilises the latest GNSS receiver design, including multiple frequency measurements and high quality antennas