Deep Learning-based Driver Behavior Modeling and Analysis

Driving safety continues receiving widespread attention from car designers, safety regulators, and automotive research community as driving accidents due to driver distraction or fatigue have increased drastically over the years. In the past decades, there has been a remarkable push towards designing and developing new driver assistance systems with much better recognition and prediction capabilities. Equipped with various sensory systems, these Advanced Driver Assistance Systems (ADAS) are able to accurately perceive information on road conditions, predict traffic situations, estimate driving risks, and provide drivers with imminent warnings and visual assistance. In this thesis, we focus on two main aspects of driver behavior modeling in the design of new generation of ADAS. We first aim at improving the generalization ability of driver distraction recognition systems to diverse driving scenarios using the latest tools of machine learning and connectionist modeling, namely deep learning. To this end, we collect a large dataset of images on various driving situations of drivers from the Internet. Then we introduce Generative Adversarial Networks (GANs) as a data augmentation tool to enhance detection accuracy. A novel driver monitoring system is also introduced. This monitoring system combines multi-information resources, including a driver distraction recognition system, to assess the danger levels of driving situations. Moreover, this thesis proposes a multi-modal system for distraction recognition under various lighting conditions and presents a new Convolutional Neural Network (CNN) architecture, which can operate real-time on a resources-limited computational platform. The new CNN is built upon a novel network bottleneck of Depthwise Separable Convolution layers. The second part of this thesis focuses on driver maneuver prediction, which infers the direction a driver will turn to before a green traffic light is on and predicts accurately whether or not he/she will change the current driving lane. Here, a new method to label driving maneuver records is proposed, by which driving feature sequences for the training of prediction systems are more closely related to their labels. To this end, a new prediction system, which is based on Quasi-Recurrent Neural Networks, is introduced. In addition, and as an application of maneuver prediction, a novel driving proficiency assessment method is proposed. This method exploits the generalization abilities of different maneuver prediction systems to estimate drivers' driving abilities, and it demonstrates several advantages against existing assessment methods. In conjunction with the theoretical contribution, a series of comprehensive experiments are conducted, and the proposed methods are assessed against state-of-the-art works. The analysis of experimental results shows the improvement of results as compared with existing techniques

A Review of Driver Gaze Estimation and Application in Gaze Behavior Understanding

Driver gaze plays an important role in different gaze-based applications such as driver attentiveness detection, visual distraction detection, gaze behavior understanding, and building driver assistance system. The main objective of this study is to perform a comprehensive summary of driver gaze fundamentals, methods to estimate driver gaze, and it's applications in real world driving scenarios. We first discuss the fundamentals related to driver gaze, involving head-mounted and remote setup based gaze estimation and the terminologies used for each of these data collection methods. Next, we list out the existing benchmark driver gaze datasets, highlighting the collection methodology and the equipment used for such data collection. This is followed by a discussion of the algorithms used for driver gaze estimation, which primarily involves traditional machine learning and deep learning based techniques. The estimated driver gaze is then used for understanding gaze behavior while maneuvering through intersections, on-ramps, off-ramps, lane changing, and determining the effect of roadside advertising structures. Finally, we have discussed the limitations in the existing literature, challenges, and the future scope in driver gaze estimation and gaze-based applications

Driver activity recognition for intelligent vehicles: a deep learning approach

Driver decisions and behaviors are essential factors that can affect the driving safety. To understand the driver behaviors, a driver activities recognition system is designed based on the deep convolutional neural networks (CNN) in this study. Specifically, seven common driving activities are identified, which are the normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle radio device, texting, and answering the mobile phone, respectively. Among these activities, the first four are regarded as normal driving tasks, while the rest three are classified into the distraction group. The experimental images are collected using a low-cost camera, and ten drivers are involved in the naturalistic data collection. The raw images are segmented using the Gaussian mixture model (GMM) to extract the driver body from the background before training the behavior recognition CNN model. To reduce the training cost, transfer learning method is applied to fine tune the pre-trained CNN models. Three different pre-trained CNN models, namely, AlexNet, GoogLeNet, and ResNet50 are adopted and evaluated. The detection results for the seven tasks achieved an average of 81.6% accuracy using the AlexNet, 78.6% and 74.9% accuracy using the GoogLeNet and ResNet50, respectively. Then, the CNN models are trained for the binary classification task and identify whether the driver is being distracted or not. The binary detection rate achieved 91.4% accuracy, which shows the advantages of using the proposed deep learning approach. Finally, the real-world application are analysed and discussed

The identification of non-driving activities with associated implication on the take-over process

In conditionally automated driving, the engagement of non-driving activities (NDAs) can be regarded as the main factor that affects the driver’s take-over performance, the investigation of which is of great importance to the design of an intelligent human–machine interface for a safe and smooth control transition. This paper introduces a 3D convolutional neural network-based system to recognize six types of driver behaviour (four types of NDAs and two types of driving activities) through two video feeds based on head and hand movement. Based on the interaction of driver and object, the selected NDAs are divided into active mode and passive mode. The proposed recognition system achieves 85.87% accuracy for the classification of six activities. The impact of NDAs on the perspective of the driver’s situation awareness and take-over quality in terms of both activity type and interaction mode is further investigated. The results show that at a similar level of achieved maximum lateral error, the engagement of NDAs demands more time for drivers to accomplish the control transition, especially for the active mode NDAs engagement, which is more mentally demanding and reduces drivers’ sensitiveness to the driving situation change. Moreover, the haptic feedback torque from the steering wheel could help to reduce the time of the transition process, which can be regarded as a productive assistance system for the take-over process.Engineering and Physical Sciences Research Council (EPSRC) EP/R511511/

Estimation of Driver's Gaze Region from Head Position and Orientation using Probabilistic Confidence Regions

A smart vehicle should be able to understand human behavior and predict their actions to avoid hazardous situations. Specific traits in human behavior can be automatically predicted, which can help the vehicle make decisions, increasing safety. One of the most important aspects pertaining to the driving task is the driver's visual attention. Predicting the driver's visual attention can help a vehicle understand the awareness state of the driver, providing important contextual information. While estimating the exact gaze direction is difficult in the car environment, a coarse estimation of the visual attention can be obtained by tracking the position and orientation of the head. Since the relation between head pose and gaze direction is not one-to-one, this paper proposes a formulation based on probabilistic models to create salient regions describing the visual attention of the driver. The area of the predicted region is small when the model has high confidence on the prediction, which is directly learned from the data. We use Gaussian process regression (GPR) to implement the framework, comparing the performance with different regression formulations such as linear regression and neural network based methods. We evaluate these frameworks by studying the tradeoff between spatial resolution and accuracy of the probability map using naturalistic recordings collected with the UTDrive platform. We observe that the GPR method produces the best result creating accurate predictions with localized salient regions. For example, the 95% confidence region is defined by an area that covers 3.77% region of a sphere surrounding the driver.Comment: 13 Pages, 12 figures, 2 table

A Steering Wheel Mounted Grip Sensor: Design, Development and Evaluation

Department of Human Factors EngineeringDriving is a commonplace but safety critical daily activity for billions of people. It remains one of the leading causes of death worldwide, particularly in younger adults. In the last decades, a wide range of technologies, such as intelligent braking or speed regulating systems, have been integrated into vehicles to improve safetyannually decreasing death rates testify to their success. A recent research focus in this area has been in the development of systems that sense human states or activities during driving. This is valuable because human error remains a key reason underlying many vehicle accidents and incidents. Technologies that can intervene in response to information sensed about a driver may be able to detect, predict and ultimately prevent problems before they progress into accidents, thus avoiding the occurrence of critical situations rather than just mitigating their consequences. Commercial examples of this kind of technology include systems that monitor driver alertness or lane holding and prompt drivers who are sleepy or drifting off-lane. More exploratory research in this area has sought to capture emotional state or stress/workload levels via physiological measurements of Heart Rate Variability (HRV), Electrocardiogram (ECG) and Electroencephalogram (EEG), or behavioral measurements of eye gaze or face pose. Other research has monitored explicitly user actions, such as head pose or foot movements to infer intended actions (such as overtaking or lane change) and provide automatic assessments of the safety of these future behaviors ??? for example, providing a timely warning to a driver who is planning to overtake about a vehicle in his or her blind spot. Researchers have also explored how sensing hands on the wheel can be used to infer a driver???s presence, identity or emotional state. This thesis extends this body of work through the design, development and evaluation of a steering wheel sensor platform that can directly detect a driver???s hand pose all around a steering wheel. This thesis argues that full steering hand pose is a potentially rich source of information about a driver???s intended actions. For example, it proposes a link between hand posture on the wheel and subsequent turning or lane change behavior. To explore this idea, this thesis describes the construction of a touch sensor in the form of a steering wheel cover. This cover integrates 32 equidistantly spread touch sensing electrodes (11.250 inter-sensor spacing) in the form of conductive ribbons (0.2" wide and 0.03" thick). Data from each ribbons is captured separately via a set of capacitive touch sensor microcontrollers every 64 ms. We connected this hardware platform to an OpenDS, an open source driving simulator and ran two studies capturing hand pose during a sequential lane change task and a slalom task. We analyzed the data to determine whether hand pose is a useful predictor of future turning behavior. For this we classified a 5-lane road into 4 turn sizes and used machine-learning recognizers to predict the future turn size from the change in hand posture in terms of hand movement properties from the early driving data. Driving task scenario of the first experiment was not appropriately matched with the real life turning task therefore we modified the scenario with more appropriate task in the second experiments. Class-wise prediction of the turn sizes for both experiments didn???t show good accuracy, however prediction accuracy was improved when the classes were reduced into two classes from four classes. In the experiment 2 turn sizes were overlapped between themselves, which made it very difficult to distinguish them. Therefore, we did continuous prediction as well and the prediction accuracy was better than the class-wise prediction system for the both experiments. In summary, this thesis designed, developed and evaluated a combined hardware and software system that senses the steering behavior of a driver by capturing grip pose. We assessed the value of this information via two studies that explored the relationship between wheel grip and future turning behaviors. The ultimate outcome of this study can inform the development of in car sensing systems to support safer driving.ope

A refined non-driving activity classification using a two-stream convolutional neural network

It is of great importance to monitor the driver’s status to achieve an intelligent and safe take-over transition in the level 3 automated driving vehicle. We present a camera-based system to recognise the non-driving activities (NDAs) which may lead to different cognitive capabilities for take-over based on a fusion of spatial and temporal information. The region of interest (ROI) is automatically selected based on the extracted masks of the driver and the object/device interacting with. Then, the RGB image of the ROI (the spatial stream) and its associated current and historical optical flow frames (the temporal stream) are fed into a two-stream convolutional neural network (CNN) for the classification of NDAs. Such an approach is able to identify not only the object/device but also the interaction mode between the object and the driver, which enables a refined NDA classification. In this paper, we evaluated the performance of classifying 10 NDAs with two types of devices (tablet and phone) and 5 types of tasks (emailing, reading, watching videos, web-browsing and gaming) for 10 participants. Results show that the proposed system improves the averaged classification accuracy from 61.0% when using a single spatial stream to 90.5