Road Infrastructure Challenges Faced by Automated Driving: A Review

Automated driving can no longer be referred to as hype or science fiction but rather a technology that has been gradually introduced to the market. The recent activities of regulatory bodies and the market penetration of automated driving systems (ADS) demonstrate that society is exhibiting increasing interest in this field and gradually accepting new methods of transport. Automated driving, however, does not depend solely on the advances of onboard sensor technology or artificial intelligence (AI). One of the essential factors in achieving trust and safety in automated driving is road infrastructure, which requires careful consideration. Historically, the development of road infrastructure has been guided by human perception, but today we are at a turning point at which this perspective is not sufficient. In this study, we review the limitations and advances made in the state of the art of automated driving technology with respect to road infrastructure in order to identify gaps that are essential for bridging the transition from human control to self-driving. The main findings of this study are grouped into the following five clusters, characterised according to challenges that must be faced in order to cope with future mobility: international harmonisation of traffic signs and road markings, revision of the maintenance of the road infrastructure, review of common design patterns, digitalisation of road networks, and interdisciplinarity. The main contribution of this study is the provision of a clear and concise overview of the interaction between road infrastructure and ADS as well as the support of international activities to define the requirements of road infrastructure for the successful deployment of ADS

Unconstrained Road Sign Recognition

There are many types of road signs, each of which carries a different meaning and function: some signs regulate traffic, others indicate the state of the road or guide and warn drivers and pedestrians. Existent image-based road sign recognition systems work well under ideal conditions, but experience problems when the lighting conditions are poor or the signs are partially occluded. The aim of this research is to propose techniques to recognize road signs in a real outdoor environment, especially to deal with poor lighting and partially occluded road signs. To achieve this, hybrid segmentation and classification algorithms are proposed. In the first part of the thesis, we propose a hybrid dynamic threshold colour segmentation algorithm based on histogram analysis. A dynamic threshold is very important in road sign segmentation, since road sign colours may change throughout the day due to environmental conditions. In the second part, we propose a geometrical shape symmetry detection and reconstruction algorithm to detect and reconstruct the shape of the sign when it is partially occluded. This algorithm is robust to scale changes and rotations. The last part of this thesis deals with feature extraction and classification. We propose a hybrid feature vector based on histograms of oriented gradients, local binary patterns, and the scale-invariant feature transform. This vector is fed into a classifier that combines a Support Vector Machine (SVM) using a Random Forest and a hybrid SVM k-Nearest Neighbours (kNN) classifier. The overall method proposed in this thesis shows a high accuracy rate of 99.4% in ideal conditions, 98.6% in noisy and fading conditions, 98.4% in poor lighting conditions, and 92.5% for partially occluded road signs on the GRAMUAH traffic signs dataset

Astigmatism and Pseudoaccommodation in Pseudophakic Eyes

noAdvanced IOLs with circumferential zones of different power provide pseudoaccommodation. We investigated the potential for power variation with meridian, namely astigmatism, to provide pseudo-accommodation. With appropriate power and axis orientations, acceptable pseudo-accommodation can be achieved

Automatic segmentation and reconstruction of traffic accident scenarios from mobile laser scanning data

Virtual reconstruction of historic sites, planning of restorations and attachments of new building parts, as well as forest inventory are few examples of fields that benefit from the application of 3D surveying data. Originally using 2D photo based documentation and manual distance measurements, the 3D information obtained from multi camera and laser scanning systems realizes a noticeable improvement regarding the surveying times and the amount of generated 3D information. The 3D data allows a detailed post processing and better visualization of all relevant spatial information. Yet, for the extraction of the required information from the raw scan data and for the generation of useable visual output, time-consuming, complex user-based data processing is still required, using the commercially available 3D software tools. In this context, the automatic object recognition from 3D point cloud and depth data has been discussed in many different works. The developed tools and methods however, usually only focus on a certain kind of object or the detection of learned invariant surface shapes. Although the resulting methods are applicable for certain practices of data segmentation, they are not necessarily suitable for arbitrary tasks due to the varying requirements of the different fields of research. This thesis presents a more widespread solution for automatic scene reconstruction from 3D point clouds, targeting street scenarios, specifically for the task of traffic accident scene analysis and documentation. The data, obtained by sampling the scene using a mobile scanning system is evaluated, segmented, and finally used to generate detailed 3D information of the scanned environment. To realize this aim, this work adapts and validates various existing approaches on laser scan segmentation regarding the application on accident relevant scene information, including road surfaces and markings, vehicles, walls, trees and other salient objects. The approaches are therefore evaluated regarding their suitability and limitations for the given tasks, as well as for possibilities concerning the combined application together with other procedures. The obtained knowledge is used for the development of new algorithms and procedures to allow a satisfying segmentation and reconstruction of the scene, corresponding to the available sampling densities and precisions. Besides the segmentation of the point cloud data, this thesis presents different visualization and reconstruction methods to achieve a wider range of possible applications of the developed system for data export and utilization in different third party software tools

Monitoring the driver's activity using 3D information

Driver supervision is crucial in safety systems for the driver. It is important to monitor the driver to understand his necessities, patterns of movements and behaviour under determined circumstances. The availability of an accurate tool to supervise the driver’s behaviour allows multiple objectives to be achieved such as the detection of drowsiness (analysing the head movements and blinking pattern) and distraction (estimating where the driver is looking by studying the head and eyes position). Once the misbehaviour is detected in both cases an alarm, of the correct type according to the situation, could be triggered to correct the driver’s behaviour. This application distinguishes itself form other driving assistance systems due to the fact that it is oriented to analyse the inside of the vehicle instead of the outside. It is important to notice that inside supervising applications are as important as the outside supervising applications because if the driver falls asleep, a pedestrian detection algorithm can do only limited actions to prevent the accident. All this under the best and predetermined circumstances. The application has the potential to be used to estimate if the driver is looking at certain area where another application detected that an obstacle is present (inert object, animal or pedestrian). Although the market has already available technologies, able to provide automatic driver monitoring, the associated cost of the sensors to accomplish this task is very high as it is not a popular product (compared to other home or entertaining devices) nor there is a market with a high demand and supply for this sensors. Many of these technologies require external and invasive devices (attach one or a set of sensors to the body) which may interfere the driving movements proper of the nature of the driver under no supervised conditions. Current applications based on computer vision take advantage of the latest development of information technologies and the increase in computational power to create applications that fit to the criteria of a non-invasive method for driving monitoring application. Technologies such as stereo and time of flight cameras are able to overcome some of the difficulties related to computer vision applications such as extreme lighting conditions (too dark or too bright) saturation of the colour sensors and lack of depth information. It is true that the combination of different sensors can overcome this problems by performing multiple scans from different areas or by combining the information obtained from different devices but this requires an additional step of calibration, positioning and it involves a dependability factor of the application on not one but as many sensors included in the task to perform the supervision because if one of them fails, the results may not be correct. Some of the recent gaming sensors available in the market, such as the Kinect sensor bar form Microsoft, are providing a new set of previously-expensive sensors embedded in a low cost device, thus providing 3D information together with some additional features and without the need for complex sets of handcrafted system that can fail as previously mentioned. The proposed solution in this thesis monitors the driver by using the different data from the Kinect sensor (depth information, infrared and colour image). The fusion of the information from the different sources allows the usage of 2D and 3D algorithms in order to provide a reliable face detection, accurate pose estimation and trustable detection of facial features such as the eyes and nose. The system will compare, with an average speed over 10Hz, the initial face capture with the next frames, it will compare by an iterative algorithm previously configured with the compromise of accuracy and speed. In order to determine the reliability and accuracy of the proposed system, several tests were performed for the head-pose orientation algorithm with an Inertial Measurement Unit (IMU) attached to the back of the head of the collaborative subjects. The inertial measurements provided by the IMU were used as a ground truth for three degrees of freedom (3DoF) tests (yaw, pitch and roll). Finally, the tests results were compared with those available in current literature to check the performance of the algorithm presented. Estimating the head orientation is the main function of this proposal as it is the one that delivers more information to estimate the behaviour of the driver. Whether it is to have a first estimation if the driver is looking to the front or if it is presenting signs of fatigue when nodding. Supporting this tool, is another that is in charge of the analysis of the colour image that will deal with the study of the eyes of the driver. From this study, it will be possible to estimate where the driver is looking at by estimating the gaze orientation through the position of the pupil. The gaze orientation would help, along with the head orientation, to have a more accurate guess regarding where the driver is looking. The gaze orientation is then a support tool that complements the head orientation. Another way to estimate a hazardous situation is with the analysis of the opening of the eyes. It can be estimated if the driver is tired through the study of the driver’s blinking pattern during a determined time. If it is so, the driver increases the chance to cause an accident due to drowsiness. The part of the whole solution that deals with solving this problem will analyse one eye of the driver to estimate if it is closed or open according to the analysis of dark regions in the image. Once the state of the eye is determined, an analysis during a determined period of time will be done in order to know if the eye was most of the time closed or open and thus estimate in a more accurate way if the driver is falling asleep or not. This 2 modules, drowsiness detector and gaze estimator, will complement the estimation of the head orientation with the goal of getting more certainty regarding the driver’s status and, when possible, to prevent an accident due to misbehaviours. It is worth to mention that the Kinect sensor is built specifically for indoor use and connected to a video console, not for the outside. Therefore, it is inevitable that some limitations arise when performing monitoring under real driving conditions. They will be discussed in this proposal. However, the algorithm presented can be used with any point-cloud based sensor (stereo cameras, time of flight cameras, laser scanners etc...); more expensive, but less sensitive compared to the former. Future works are described at the end in order to show the scalability of this proposal.La supervisión del conductor es crucial en los sistemas de asistencia a la conducción. Resulta importante monitorizarle para entender sus necesidades, patrones de movimiento y comportamiento bajo determinadas circunstancias. La disponibilidad de una herramienta precisa que supervise el comportamiento del conductor permite que varios objetivos sean alcanzados como la detección de somnolencia (analizando los movimientos de la cabeza y parpadeo) y distracción (estimando hacia donde está mirando por medio del estudio de la posición tanto de la cabeza como de los ojos). En ambos casos, una vez detectado el mal comportamiento, se podría activar una alarma del tipo adecuado según la situación que le corresponde con el objetivo de corregir su comportamiento del conductor Esta aplicación se distingue de otros sistemas avanzados de asistencia la conducción debido al hecho de que está orientada al análisis interior del vehículo en lugar del exterior. Es importante notar que las aplicaciones de supervisión interna son tan importantes como las del exterior debido a que si el conductor se duerme, un sistema de detección de peatones o vehículos sólo podrá hacer ciertas maniobras para evitar un accidente. Todo esto bajo las condiciones idóneas y circunstancias predeterminadas. Esta aplicación tiene el potencial para estimar si quien conduce está mirando hacia una zona específica que otra aplicación que detecta objetos, animales y peatones ha remarcado como importante. Aunque en el mercado existen tecnologías disponibles capaces de supervisar al conductor, estas tienen un coste prohibitivo para cierto grupo de clientela debido a que no es un producto popular (comparado con otros dispositivos para el hogar o de entretenimiento) ni existe un mercado con alta oferta y demanda de dichos dispositivos. Muchas de estas tecnologías requieren de dispositivos externos e invasivos (colocarle al conductor uno o más sensores en el cuerpo) que podrían interferir con la naturaleza de los movimientos propios de la conducción bajo condiciones sin supervisar. Las aplicaciones actuales basadas en visión por computador toman ventaja de los últimos desarrollos de la tecnología informática y el incremento en poder computacional para crear aplicaciones que se ajustan al criterio de un método no invasivo para aplicarlo a la supervisión del conductor. Tecnologías como cámaras estéreo y del tipo “tiempo de vuelo” son capaces de sobrepasar algunas de las dificultades relacionadas a las aplicaciones de visión por computador como condiciones extremas de iluminación (diurna y nocturna), saturación de los sensores de color y la falta de información de profundidad. Es cierto que la combinación y fusión de sensores puede resolver este problema por medio de múltiples escaneos de diferentes zonas o combinando la información obtenida de diversos dispositivos pero esto requeriría un paso adicional de calibración, posicionamiento e involucra un factor de dependencia de la aplicación hacia no uno sino los múltiples sensores involucrados ya que si uno de ellos falla, los resultados podrían no ser correctos. Recientemente han aparecido en el mercado de los videojuego algunos sensores, como es el caso de la barra de sensores Kinect de Microsoft, dispositivo de bajo coste, que ofrece información 3D junto con otras características adicionales y sin la necesidad de sistemas complejos de sistemas manufacturados que pueden fallar como se ha mencionado anteriormente. La solución propuesta en esta tesis supervisa al conductor por medio del uso de información diversa del sensor Kinect (información de profundidad, imágenes de color en espectro visible y en espectro infrarrojo). La fusión de información de diversas fuentes permite el uso de algoritmos en 2D y 3D con el objetivo de proveer una detección facial confiable, estimación de postura precisa y detección de características faciales como los ojos y la nariz. El sistema comparará, con una velocidad promedio superior a 10Hz, la captura inicial de la cara con el resto de las imágenes de video, la comparación la hará por medio de un algoritmo iterativo previamente configurado comprometido con el balance entre velocidad y precisión. Con tal de determinar la fiabilidad y precisión del sistema propuesto, diversas pruebas fueron realizadas para el algoritmo de estimación de postura de la cabeza con una unidad de medidas inerciales (IMU por sus siglas en inglés) situada en la parte trasera de la cabeza de los sujetos que participaron en los ensayos. Las medidas inerciales provistas por la IMU fueron usadas como punto de referencia para las pruebas de los tres grados de libertad de movimiento. Finalmente, los resultados de las pruebas fueron comparados con aquellos disponibles en la literatura actual para comprobar el rendimiento del algoritmo aquí presentado. Estimar la orientación de la cabeza es la función principal de esta propuesta ya que es la que más aporta información para la estimación del comportamiento del conductor. Sea para tener una primera estimación si ve hacia el frente o si presenta señales de fatiga al cabecear hacia abajo. Acompañando a esta herramienta, está el análisis de la imagen a color que se encargará del estudio de los ojos. A partir de dicho estudio, se podrá estimar hacia donde está viendo el conductor según la posición de la pupila. La orientación de la mirada ayudaría, junto con la orientación de la cabeza, a saber hacia dónde ve el conductor. La estimación de la orientación de la mirada es una herramienta de soporte que complementa la orientación de la cabeza. Otra forma de determinar una situación de riesgo es con el análisis de la apertura de los ojos. A través del estudio del patrón de parpadeo en el conductor durante un determinado tiempo se puede estimar si se encuentra cansado. De ser así, el conductor aumenta las posibilidades de causar un accidente debido a la somnolencia. La parte de la solución que se encarga de resolver este problema analizará un ojo del conductor para estimar si se encuentra cerrado o abierto de acuerdo al análisis de regiones de interés en la imagen. Una vez determinado el estado del ojo, se procederá a hacer un análisis durante un determinado tiempo para saber si el ojo ha estado mayormente cerrado o abierto y estimar de forma más acertada si se está quedando dormido o no. Estos 2 módulos, el detector de somnolencia y el análisis de la mirada complementarán la estimación de la orientación de la cabeza con el objetivo de brindar mayor certeza acerca del estado del conductor y, de ser posible, prevenir un accidente debido a malos comportamientos. Es importante mencionar que el sensor Kinect está construido específicamente para el uso dentro de una habitación y conectado a una videoconsola, no para el exterior. Por lo tanto, es inevitable que algunas limitaciones salgan a luz cuando se realice la monitorización bajo condiciones reales de conducción. Dichos problemas serán mencionados en esta propuesta. Sin embargo, el algoritmo presentado es generalizable a cualquier sensor basado en nubes de puntos (cámaras estéreo, cámaras del tipo “time of flight”, escáneres láseres etc...); más caros pero menos sensibles a estos inconvenientes previamente descritos. Se mencionan también trabajos futuros al final con el objetivo de enseñar la escalabilidad de esta propuesta.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Andrés Iborra García.- Secretario: Francisco José Rodríguez Urbano.- Vocal: José Manuel Pastor Garcí

Detection and Recognition of Traffic Signs Inside the Attentional Visual Field of Drivers

Traffic sign detection and recognition systems are essential components of Advanced Driver Assistance Systems and self-driving vehicles. In this contribution we present a vision-based framework which detects and recognizes traffic signs inside the attentional visual field of drivers. This technique takes advantage of the driver\u27s 3D absolute gaze point obtained through the combined use of a front-view stereo imaging system and a non-contact 3D gaze tracker. We used a linear Support Vector Machine as a classifier and a Histogram of Oriented Gradient as features for detection. Recognition is performed by using Scale Invariant Feature Transforms and color information. Our technique detects and recognizes signs which are in the field of view of the driver and also provides indication when one or more signs have been missed by the driver

Pedestrian detection in far infrared images

Detection of people in images is a relatively new field of research, but has been widely accepted. The applications are multiple, such as self-labeling of large databases, security systems and pedestrian detection in intelligent transportation systems. Within the latter, the purpose of a pedestrian detector from a moving vehicle is to detect the presence of people in the path of the vehicle. The ultimate goal is to avoid a collision between the two. This thesis is framed with the advanced driver assistance systems, passive safety systems that warn the driver of conditions that may be adverse. An advanced driving assistance system module, aimed to warn the driver about the presence of pedestrians, using computer vision in thermal images, is presented in this thesis. Such sensors are particularly useful under conditions of low illumination.The document is divided following the usual parts of a pedestrian detection system: development of descriptors that define the appearance of people in these kind of images, the application of these descriptors to full-sized images and temporal tracking of pedestrians found. As part of the work developed in this thesis, database of pedestrians in the far infrared spectrum is presented. This database has been used in developing an evaluation of pedestrian detection systems as well as for the development of new descriptors. These descriptors use techniques for the systematic description of the shape of the pedestrian as well as methods to achieve invariance to contrast, illumination or ambient temperature. The descriptors are analyzed and modified to improve their performance in a detection problem, where potential candidates are searched for in full size images. Finally, a method for tracking the detected pedestrians is proposed to reduce the number of miss-detections that occurred at earlier stages of the algorithm. --La detección de personas en imágenes es un campo de investigación relativamente nuevo, pero que ha tenido una amplia acogida. Las aplicaciones son múltiples, tales como auto-etiquetado de grandes bases de datos, sistemas de seguridad y detección de peatones en sistemas inteligentes de transporte. Dentro de este último, la detección de peatones desde un vehículo móvil tiene como objetivo detectar la presencia de personas en la trayectoria del vehículo. EL fin último es evitar una colisión entre ambos. Esta tesis se enmarca en los sistemas avanzados de ayuda a la conducción; sistemas de seguridad pasivos, que advierten al conductor de condiciones que pueden ser adversas. En esta tesis se presenta un módulo de ayuda a la conducción destinado a advertir de la presencia de peatones, mediante el uso de visión por computador en imágenes térmicas. Este tipo de sensores resultan especialmente útiles en condiciones de baja iluminación. El documento se divide siguiendo las partes habituales de una sistema de detección de peatones: desarrollo de descriptores que defina la apariencia de las personas en este tipo de imágenes, la aplicación de estos en imágenes de tamano completo y el seguimiento temporal de los peatones encontrados. Como parte del trabajo desarrollado en esta tesis se presenta una base de datos de peatones en el espectro infrarrojo lejano. Esta base de datos ha sido utilizada para desarrollar una evaluación de sistemas de detección de peatones, así como para el desarrollo de nuevos descriptores. Estos integran técnicas para la descripción sistemática de la forma del peatón, así como métodos para la invariancia al contraste, la iluminación o la temperatura externa. Los descriptores son analizados y modificados para mejorar su rendimiento en un problema de detección, donde se buscan posibles candidatos en una imagen de tamano completo. Finalmente, se propone una método de seguimiento de los peatones detectados para reducir el número de fallos que se hayan producido etapas anteriores del algoritmo