A high speed Tri-Vision system for automotive applications

Purpose: Cameras are excellent ways of non-invasively monitoring the interior and exterior of vehicles. In particular, high speed stereovision and multivision systems are important for transport applications such as driver eye tracking or collision avoidance. This paper addresses the synchronisation problem which arises when multivision camera systems are used to capture the high speed motion common in such applications. Methods: An experimental, high-speed tri-vision camera system intended for real-time driver eye-blink and saccade measurement was designed, developed, implemented and tested using prototype, ultra-high dynamic range, automotive-grade image sensors specifically developed by E2V (formerly Atmel) Grenoble SA as part of the European FP6 project – sensation (advanced sensor development for attention stress, vigilance and sleep/wakefulness monitoring). Results : The developed system can sustain frame rates of 59.8 Hz at the full stereovision resolution of 1280 × 480 but this can reach 750 Hz when a 10 k pixel Region of Interest (ROI) is used, with a maximum global shutter speed of 1/48000 s and a shutter efficiency of 99.7%. The data can be reliably transmitted uncompressed over standard copper Camera-Link® cables over 5 metres. The synchronisation error between the left and right stereo images is less than 100 ps and this has been verified both electrically and optically. Synchronisation is automatically established at boot-up and maintained during resolution changes. A third camera in the set can be configured independently. The dynamic range of the 10bit sensors exceeds 123 dB with a spectral sensitivity extending well into the infra-red range. Conclusion: The system was subjected to a comprehensive testing protocol, which confirms that the salient requirements for the driver monitoring application are adequately met and in some respects, exceeded. The synchronisation technique presented may also benefit several other automotive stereovision applications including near and far-field obstacle detection and collision avoidance, road condition monitoring and others.Partially funded by the EU FP6 through the IST-507231 SENSATION project.peer-reviewe

Detecting Distracted Driving with Deep Learning

© Springer International Publishing AG 2017Driver distraction is the leading factor in most car crashes and near-crashes. This paper discusses the types, causes and impacts of distracted driving. A deep learning approach is then presented for the detection of such driving behaviors using images of the driver, where an enhancement has been made to a standard convolutional neural network (CNN). Experimental results on Kaggle challenge dataset have confirmed the capability of a convolutional neural network (CNN) in this complicated computer vision task and illustrated the contribution of the CNN enhancement to a better pattern recognition accuracy.Peer reviewe

Single-imager occupant detection based on surface reconstruction

This thesis introduces a novel framework for a real-time occupant detection system capable of extracting both two- and three-dimensional information using a single imager with active illumination. The primary objective of this thesis is to demonstrate the feasibility of such a low-cost classification system with comparable performance to multi-camera based stereo vision systems. Severe illumination conditions characterised by a frequent and wide illumination fluctuation are also challenging problems addressed in this work. The proposed system is designed to solve a problem of classifying three occupant classes being an adult, a forward-facing child seat, and a rear-facing child seat. DoubleFlash is employed to eliminate the influence of ambient illumination and to compress the optical dynamic range of target scenes. The idea underlying this technique is to subtract images flashed by different illumination power levels. The extension of this active illumination technique leads to the development of a novel shadow removal technique, called ShadowFlash. By simulating an artificial infinite illuminating plane over the field of view, the technique produces a shadowless scene without losing image details by composing multiple images illuminated from different directions. The ShadowFlash technique is then extended to the temporal domain by employing the sliding n-tuple strategy, which is introduced to avoid the reduction of the original frame rate. A modified active contour model, facilitated by morphological operations, extracts the boundary of the target object from the shadow-free scenes produced by the ShadowFlash. Based on the brightness information of the image triplet generated by the DoubleFlash, the orientations of the object surface at pixel points are estimated by the photometric stereo method and integrated into the 3D surface by means of global minimisation. The boundary information is used to specify the region of interest to reconstruct. Investigating both the two- and three-dimensional properties of vehicle occupants, 29 features are defined for the training of a neural network. The system is tested on a database of over 84,000 frames collected from a wide range of objects in various illumination conditions. A classification accuracy of 98.9% was achieved within the decision-time limit of three seconds

Quantitative Performance Assessment of LiDAR-based Vehicle Contour Estimation Algorithms for Integrated Vehicle Safety Applications

Many nations and organizations are committing to achieving the goal of `Vision Zero\u27 and eliminate road traffic related deaths around the world. Industry continues to develop integrated safety systems to make vehicles safer, smarter and more capable in safety critical scenarios. Passive safety systems are now focusing on pre-crash deployment of restraint systems to better protect vehicle passengers. Current commonly used bounding box methods for shape estimation of crash partners lack the fidelity required for edge case collision detection and advanced crash modeling. This research presents a novel algorithm for robust and accurate contour estimation of opposing vehicles. The presented method is evaluated via a developed framework for key performance metrics and compared to alternative algorithms found in literature

Head motion tracking in 3D space for drivers

Ce travail présente un système de vision par ordinateur capable de faire un suivi du mouvement en 3D de la tête d’une personne dans le cadre de la conduite automobile. Ce système de vision par ordinateur a été conçu pour faire partie d'un système intégré d’analyse du comportement des conducteurs tout en remplaçant des équipements et des accessoires coûteux, qui sont utilisés pour faire le suivi du mouvement de la tête, mais sont souvent encombrants pour le conducteur. Le fonctionnement du système est divisé en quatre étapes : l'acquisition d'images, la détection de la tête, l’extraction des traits faciaux, la détection de ces traits faciaux et la reconstruction 3D des traits faciaux qui sont suivis. Premièrement, dans l'étape d'acquisition d'images, deux caméras monochromes synchronisées sont employées pour former un système stéréoscopique qui facilitera plus tard la reconstruction 3D de la tête. Deuxièmement, la tête du conducteur est détectée pour diminuer la dimension de l’espace de recherche. Troisièmement, après avoir obtenu une paire d’images de deux caméras, l'étape d'extraction des traits faciaux suit tout en combinant les algorithmes de traitement d'images et la géométrie épipolaire pour effectuer le suivi des traits faciaux qui, dans notre cas, sont les deux yeux et le bout du nez du conducteur. Quatrièmement, dans une étape de détection des traits faciaux, les résultats 2D du suivi sont consolidés par la combinaison d'algorithmes de réseau de neurones et la géométrie du visage humain dans le but de filtrer les mauvais résultats. Enfin, dans la dernière étape, le modèle 3D de la tête est reconstruit grâce aux résultats 2D du suivi et ceux du calibrage stéréoscopique des caméras. En outre, on détermine les mesures 3D selon les six axes de mouvement connus sous le nom de degrés de liberté de la tête (longitudinal, vertical, latéral, roulis, tangage et lacet). La validation des résultats est effectuée en exécutant nos algorithmes sur des vidéos préenregistrés des conducteurs utilisant un simulateur de conduite afin d'obtenir des mesures 3D avec notre système et par la suite, à les comparer et les valider plus tard avec des mesures 3D fournies par un dispositif pour le suivi de mouvement installé sur la tête du conducteur.This work presents a computer vision module capable of tracking the head motion in 3D space for drivers. This computer vision module was designed to be part of an integrated system to analyze the behaviour of the drivers by replacing costly equipments and accessories that track the head of a driver but are often cumbersome for the user. The vision module operates in five stages: image acquisition, head detection, facial features extraction, facial features detection, and 3D reconstruction of the facial features that are being tracked. Firstly, in the image acquisition stage, two synchronized monochromatic cameras are used to set up a stereoscopic system that will later make the 3D reconstruction of the head simpler. Secondly the driver’s head is detected to reduce the size of the search space for finding facial features. Thirdly, after obtaining a pair of images from the two cameras, the facial features extraction stage follows by combining image processing algorithms and epipolar geometry to track the chosen features that, in our case, consist of the two eyes and the tip of the nose. Fourthly, in a detection stage, the 2D tracking results are consolidated by combining a neural network algorithm and the geometry of the human face to discriminate erroneous results. Finally, in the last stage, the 3D model of the head is reconstructed from the 2D tracking results (e.g. tracking performed in each image independently) and calibration of the stereo pair. In addition 3D measurements according to the six axes of motion known as degrees of freedom of the head (longitudinal, vertical and lateral, roll, pitch and yaw) are obtained. The validation of the results is carried out by running our algorithms on pre-recorded video sequences of drivers using a driving simulator in order to obtain 3D measurements to be compared later with the 3D measurements provided by a motion tracking device installed on the driver’s head

Investigation of low-cost infrared sensing for intelligent deployment of occupant restraints

In automotive transport, airbags and seatbelts are effective at restraining the driver and passenger in the event of a crash, with statistics showing a dramatic reduction in the number of casualties from road crashes. However, statistics also show that a small number of these people have been injured or even killed from striking the airbag, and that the elderly and small children are especially at risk of airbag-related injury. This is the result of the fact that in-car restraint systems were designed for the average male at an average speed of 50 km/hr, and people outside these norms are at risk. Therefore one of the future safety goals of the car manufacturers is to deploy sensors that would gain more information about the driver or passenger of their cars in order to tailor the safety systems specifically for that person, and this is the goal of this project. This thesis describes a novel approach to occupant detection, position measurement and monitoring using a low-cost thermal imaging based system, which is a departure from traditional video camera-based systems, and at an affordable price. Experiments were carried out using a specially designed test rig and a car driving simulator with members of the public. Results have shown that the thermal imager can detect a human in a car cabin mock up and provide crucial real-time position data, which could be used to support intelligent restraint deployment. Other valuable information has been detected such as whether the driver is smoking, drinking a hot or cold drink, using a mobile phone, which can help to infer the level of driver attentiveness or engagement

Real time occupant detection in high dynamic range environments

The aim of this thesis is to explore strategies for real-time image segmentation of non-rigid objects in a spatio-temporal domain with a stationary camera within an optical high dynamic range environment. Camera, illumination and segmentation techniques are discussed for image processing in environments which are characterized by large intensity fluctuations and hence a high optical dynamic range (HDR), in particular for vehicle interior surveillance. Since the introduction of the airbag in 1981 numberless lives were saved and bad injuries were avoided. But in recent years the airbag has frequently been in the headlines due to the increasing number of injuries caused by it. To avoid these injuries a new generation of ’smart airbags’ has been designed which shows the ability to inflate in multiple steps and with different volumes. In order to determine the optimal inflation mode for a crash it is necessary to consider information about the interior situation and the occupants of the vehicle. This thesis presents a real-time visual occupant detection and classification system for advanced airbag deployment, utilizing a custom CMOS camera and motion based image segmentation algorithms for embedded systems under adverse illumination conditions. A novel illumination method is presented which combines a set of images flashed with different radiant intensities, which significantly simplifies image segmentation in HDR environments. With a constant exposure time for the imager a single image can be produced with a compressed dynamic range and a simultaneously reduced offset. This makes it possible to capture a vehicle interior under adverse light conditions without using high dynamic range cameras and without losing image detail. The expansion of this active illumination experiment leads to a novel shadow detection and removal technique that produces a shadow-free scene by simulating an artificial infinite illuminant plane over the held of view. Finally a shadowless image without loss of texture details is obtained without any region extraction phase. Furthermore, a texture based segmentation approach for stationary cam-eras is presented which is neither effected by sudden illumination changes nor by shadow effects

Towards a Common Software/Hardware Methodology for Future Advanced Driver Assistance Systems

The European research project DESERVE (DEvelopment platform for Safe and Efficient dRiVE, 2012-2015) had the aim of designing and developing a platform tool to cope with the continuously increasing complexity and the simultaneous need to reduce cost for future embedded Advanced Driver Assistance Systems (ADAS). For this purpose, the DESERVE platform profits from cross-domain software reuse, standardization of automotive software component interfaces, and easy but safety-compliant integration of heterogeneous modules. This enables the development of a new generation of ADAS applications, which challengingly combine different functions, sensors, actuators, hardware platforms, and Human Machine Interfaces (HMI). This book presents the different results of the DESERVE project concerning the ADAS development platform, test case functions, and validation and evaluation of different approaches. The reader is invited to substantiate the content of this book with the deliverables published during the DESERVE project. Technical topics discussed in this book include:Modern ADAS development platforms;Design space exploration;Driving modelling;Video-based and Radar-based ADAS functions;HMI for ADAS;Vehicle-hardware-in-the-loop validation system

Real Time Driver Safety System

The technology for driver safety has been developed in many fields such as airbag system, Anti-lock Braking System or ABS, ultrasonic warning system, and others. Recently, some of the automobile companies have introduced a new feature of driver safety systems. This new system is to make the car slower if it finds a driver’s drowsy eyes. For instance, Toyota Motor Corporation announced that it has given its pre-crash safety system the ability to determine whether a driver’s eyes are properly open with an eye monitor. This paper is focusing on finding a driver’s drowsy eyes by using face detection technology. The human face is a dynamic object and has a high degree of variability; that is why face detection is considered a difficult problem in computer vision. Even with the difficulty of this problem, scientists and computer programmers have developed and improved the face detection technologies. This paper also introduces some algorithms to find faces or eyes and compares algorithm’s characteristics. Once we find a face in a sequence of images, the matter is to find drowsy eyes in the driver safety system. This system can slow a car or alert the user not to sleep; that is the purpose of the pre-crash safety system. This paper introduces the VeriLook SDK, which is used for finding a driver’s face in the real time driver safety system. With several experiments, this paper also introduces a new way to find drowsy eyes by AOI,Area of Interest. This algorithm improves the speed of finding drowsy eyes and the consumption of memory use without using any object classification methods or matching eye templates. Moreover, this system has a higher accuracy of classification than others