Multimodal Polynomial Fusion for Detecting Driver Distraction

Distracted driving is deadly, claiming 3,477 lives in the U.S. in 2015 alone. Although there has been a considerable amount of research on modeling the distracted behavior of drivers under various conditions, accurate automatic detection using multiple modalities and especially the contribution of using the speech modality to improve accuracy has received little attention. This paper introduces a new multimodal dataset for distracted driving behavior and discusses automatic distraction detection using features from three modalities: facial expression, speech and car signals. Detailed multimodal feature analysis shows that adding more modalities monotonically increases the predictive accuracy of the model. Finally, a simple and effective multimodal fusion technique using a polynomial fusion layer shows superior distraction detection results compared to the baseline SVM and neural network models.Comment: INTERSPEECH 201

Facial attributes recognition using computer vision to detect drowsiness and distraction in drivers

Driving is an activity that requires a high degree of concentration on the part of the person who performs it, since the slightest negligence is sufficient to provoke an accident with the consequent material and/or human losses. According to the most recent study published by the World Health Organization (WHO) in 2013, it was estimated that 1.25 million people died as a result of traffic accidents, whereas between 20 and 50 million did not die but consequences resulted in chronic conditions. Many of these accidents are caused by what is known as inattention. This term encloses different conditions such as distraction and drowsiness, which are, precisely, the ones that cause more fatalities. Many publications and research have tried to set figures indicating the consequences of inattention (and its subtypes), but there is no exact number of the accidents caused by inattention since all these studies have been carried out in different places, different time frames and, therefore, under different conditions. Overall, it has been estimated that inattention causes between 25% and 75% of accidents and near-accidents. A study on drowsiness while driving in ten European countries found that fatigue risks increasing reaction time by 86% and it is the fourth leading cause of death on Spanish roads. Distraction is also a major contributor to fatal accidents in Spain. According to the Directorate General of Traffic (DGT), distraction is the first violation found in fatal accidents, 13.15% of the cases. Overall, considering both distraction and drowsiness, the latest statistics on inattentive driving in Spanish drivers are alarming, appearing as the leading cause of fatalities (36%), well above excessive speed (21%) or alcohol consumption (11%).The reason for this PhD thesis is the direct consequences of the abovementioned figures and its purpose is to provide mechanisms to help reduce driver inattention effects using computer vision techniques. The extraction of facial attributes can be used to detect inattention robustly.Specifically, research establishes a frame of reference to characterize distraction in drivers in order to provide solid foundations for future research [1]. Based on this research [1], an architecture based on the analysis of visual characteristics has been proposed, constructed and validated by using techniques of computer vision and automatic learning for the detection of both distraction and drowsiness [2], integrating several innovative elements in order to operate in a completely autonomous way for the robust detection of the main visual indicators characterizing the driver's both distraction and drowsiness: (1) a review of the role of computer vision technology applied to the development of monitoring systems to detect distraction [3]; (2) a face processing algorithm based on Local Binary Patterns (LBP) and Support Vector Machine (SVM) to detect facial attributes [4]; (3) detection unit for the presence/absence of the driver using both a marker and a machine learning algorithm [2]; (4) robust face tracking algorithm based on both the position of the camera and the face detection algorithm [2]; (5) a face alignment and normalization algorithm to improve the eyes state detection [3]; (6) driver drowsiness detection based on the eyes state detection over time [2]; (7) driver distraction detection based on the position of the head over time [2]. This architecture has been validated, firstly, with reference databases testing the different modules that compose it, and, secondly, with users in real environments, obtaining in both cases, excellent results with a suitable computational load for the embedded devices in vehicle environments [2]. In connection with the tests performed in real-world settings, 16 drivers were involved performing several activities imitating different signs of sleepiness and distraction. Overall, an accuracy of 93.11% is obtained considering all activities and all drivers [2].Additionally, other contributions of this thesis have been experimentally validated in controlled settings, but are expected to be included in the abovementioned architecture: (1) glasses detection algorithm prior to the detection of the eyes state [3] (the eyes state can not be accurately obtained if the driver is wearing glasses or sunglasses [1]); (2) face recognition and spoofing detection algorithm to identify the driver [5]; (3) physiological information (Heart Rate, Respiration Rate and Heart Rate Variability) are extracted from the users face [6] (using this information, cognitive load and stress can be obtained [1]); (4) a real-time big data architecture to process a large number of relatively small-sized images [7]. Therefore, future work will include these points to complete the architecture

Facial attributes recognition using computer vision to detect drowsiness and distraction in drivers

Driving is an activity that requires a high degree of concentration on the part of the person who performs it, since the slightest negligence is sufficient to provoke an accident with the consequent material and/or human losses. According to the most recent study published by the World Health Organization (WHO) in 2013, it was estimated that 1.25 million people died as a result of traffic accidents, whereas between 20 and 50 million did not die but consequences resulted in chronic conditions. Many of these accidents are caused by what is known as inattention. This term encloses different conditions such as distraction and drowsiness, which are, precisely, the ones that cause more fatalities. Many publications and research have tried to set figures indicating the consequences of inattention (and its subtypes), but there is no exact number of the accidents caused by inattention since all these studies have been carried out in different places, different time frames and, therefore, under different conditions. Overall, it has been estimated that inattention causes between 25% and 75% of accidents and near-accidents. A study on drowsiness while driving in ten European countries found that fatigue risks increasing reaction time by 86% and it is the fourth leading cause of death on Spanish roads. Distraction is also a major contributor to fatal accidents in Spain. According to the Directorate General of Traffic (DGT), distraction is the first violation found in fatal accidents, 13.15% of the cases. Overall, considering both distraction and drowsiness, the latest statistics on inattentive driving in Spanish drivers are alarming, appearing as the leading cause of fatalities (36%), well above excessive speed (21%) or alcohol consumption (11%).The reason for this PhD thesis is the direct consequences of the abovementioned figures and its purpose is to provide mechanisms to help reduce driver inattention effects using computer vision techniques. The extraction of facial attributes can be used to detect inattention robustly.Specifically, research establishes a frame of reference to characterize distraction in drivers in order to provide solid foundations for future research [1]. Based on this research [1], an architecture based on the analysis of visual characteristics has been proposed, constructed and validated by using techniques of computer vision and automatic learning for the detection of both distraction and drowsiness [2], integrating several innovative elements in order to operate in a completely autonomous way for the robust detection of the main visual indicators characterizing the driver’s both distraction and drowsiness: (1) a review of the role of computer vision technology applied to the development of monitoring systems to detect distraction [3]; (2) a face processing algorithm based on Local Binary Patterns (LBP) and Support Vector Machine (SVM) to detect facial attributes [4]; (3) detection unit for the presence/absence of the driver using both a marker and a machine learning algorithm [2]; (4) robust face tracking algorithm based on both the position of the camera and the face detection algorithm [2]; (5) a face alignment and normalization algorithm to improve the eyes state detection [3]; (6) driver drowsiness detection based on the eyes state detection over time [2]; (7) driver distraction detection based on the position of the head over time [2]. This architecture has been validated, firstly, with reference databases testing the different modules that compose it, and, secondly, with users in real environments, obtaining in both cases, excellent results with a suitable computational load for the embedded devices in vehicle environments [2]. In connection with the tests performed in real-world settings, 16 drivers were involved performing several activities imitating different signs of sleepiness and distraction. Overall, an accuracy of 93.11% is obtained considering all activities and all drivers [2].Additionally, other contributions of this thesis have been experimentally validated in controlled settings, but are expected to be included in the abovementioned architecture: (1) glasses detection algorithm prior to the detection of the eyes state [3] (the eyes state can not be accurately obtained if the driver is wearing glasses or sunglasses [1]); (2) face recognition and spoofing detection algorithm to identify the driver [5]; (3) physiological information (Heart Rate, Respiration Rate and Heart Rate Variability) are extracted from the users face [6] (using this information, cognitive load and stress can be obtained [1]); (4) a real-time big data architecture to process a large number of relatively small-sized images [7]. Therefore, future work will include these points to complete the architecture

Driver Distraction Identification with an Ensemble of Convolutional Neural Networks

The World Health Organization (WHO) reported 1.25 million deaths yearly due to road traffic accidents worldwide and the number has been continuously increasing over the last few years. Nearly fifth of these accidents are caused by distracted drivers. Existing work of distracted driver detection is concerned with a small set of distractions (mostly, cell phone usage). Unreliable ad-hoc methods are often used.In this paper, we present the first publicly available dataset for driver distraction identification with more distraction postures than existing alternatives. In addition, we propose a reliable deep learning-based solution that achieves a 90% accuracy. The system consists of a genetically-weighted ensemble of convolutional neural networks, we show that a weighted ensemble of classifiers using a genetic algorithm yields in a better classification confidence. We also study the effect of different visual elements in distraction detection by means of face and hand localizations, and skin segmentation. Finally, we present a thinned version of our ensemble that could achieve 84.64% classification accuracy and operate in a real-time environment.Comment: arXiv admin note: substantial text overlap with arXiv:1706.0949

Basic Emotion Recogniton using Automatic Facial Expression Analysis Software

Facial expression was proven to show a person's emotions, including 6 basic human emotions, namely happy, sad, surprise, disgusted, angry, and fear. Currently, the recognition of basic emotions is applied using the automatic facial expression analysis software. In fact, not all emotions are performed with the same expressions. This study aims to analyze whether the six basic human emotions can be recognized by software. Ten subjects were asked to spontaneously show the expressions of the six basic emotions sequentially. Subjects are not given instructions on how the standard expressions of each of the basic emotions are. The results show that only happy expressions can be consistently identified clearly by the software, while sad expressions are almost unrecognizable. On the other hand surprise expressions tend to be recognized as mixed emotions of surprise and happy. There are two emotions that are difficult to express by the subject, namely fear and anger. The subject interpretation of these two emotions varies widely and tends to be unrecognizable by software. The conclusion of this study is the way a person shows his emotions varies. Although there are some similarities in expression, it cannot be proven that all expressions of basic human emotions can be generalized. Further implication of this research needs further discussion

DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis

Vision is the richest and most cost-effective technology for Driver Monitoring Systems (DMS), especially after the recent success of Deep Learning (DL) methods. The lack of sufficiently large and comprehensive datasets is currently a bottleneck for the progress of DMS development, crucial for the transition of automated driving from SAE Level-2 to SAE Level-3. In this paper, we introduce the Driver Monitoring Dataset (DMD), an extensive dataset which includes real and simulated driving scenarios: distraction, gaze allocation, drowsiness, hands-wheel interaction and context data, in 41 hours of RGB, depth and IR videos from 3 cameras capturing face, body and hands of 37 drivers. A comparison with existing similar datasets is included, which shows the DMD is more extensive, diverse, and multi-purpose. The usage of the DMD is illustrated by extracting a subset of it, the dBehaviourMD dataset, containing 13 distraction activities, prepared to be used in DL training processes. Furthermore, we propose a robust and real-time driver behaviour recognition system targeting a real-world application that can run on cost-efficient CPU-only platforms, based on the dBehaviourMD. Its performance is evaluated with different types of fusion strategies, which all reach enhanced accuracy still providing real-time response.Comment: Accepted to ECCV 2020 workshop - Assistive Computer Vision and Robotic

Fuzzy System to Assess Dangerous Driving: A Multidisciplinary Approach

Dangerous driving can cause accidents, injuries and loss of life. An efficient assessment helps to identify the absence or degree of dangerous driving to take the appropriate decisions while driving. Previous studies assess dangerous driving through two approaches: (i) using electronic devices or sensors that provide objective variables (acceleration, turns and speed), and (ii) analyzing responses to questionnaires from behavioral science that provide subjective variables (driving thoughts, opinions and perceptions from the driver). However, we believe that a holistic and more realistic assessment requires a combination of both types of variables. Therefore, we propose a three-phase fuzzy system with a multidisciplinary (computer science and behavioral sciences) approach that draws on the strengths of sensors embedded in smartphones and questionnaires to evaluate driver behavior and social desirability. Our proposal combines objective and subjective variables while mitigating the weaknesses of the disciplines used (sensor reading errors and lack of honesty from respondents, respectively). The methods used are of proven reliability in each discipline, and their outputs feed a combined fuzzy system used to handle the vagueness of the input variables, obtaining a personalized result for each driver. The results obtained using the proposed system in a real scenario were efficient at 84.21%, and were validated with mobility experts’ opinions. The presented fuzzy system can support intelligent transportation systems, driving safety, or personnel selection

Methods to detect and reduce driver stress: a review

Automobiles are the most common modes of transportation in urban areas. An alert mind is a prerequisite while driving to avoid tragic accidents; however, driver stress can lead to faulty decision-making and cause severe injuries. Therefore, numerous techniques and systems have been proposed and implemented to subdue negative emotions and improve the driving experience. Studies show that conditions such as the road, state of the vehicle, weather, as well as the driver’s personality, and presence of passengers can affect driver stress. All the above-mentioned factors significantly influence a driver’s attention. This paper presents a detailed review of techniques proposed to reduce and recover from driving stress. These technologies can be divided into three categories: notification alert, driver assistance systems, and environmental soothing. Notification alert systems enhance the driving experience by strengthening the driver’s awareness of his/her physiological condition, and thereby aid in avoiding accidents. Driver assistance systems assist and provide the driver with directions during difficult driving circumstances. The environmental soothing technique helps in relieving driver stress caused by changes in the environment. Furthermore, driving maneuvers, driver stress detection, driver stress, and its factors are discussed and reviewed to facilitate a better understanding of the topic

Leveraging driver vehicle and environment interaction: Machine learning using driver monitoring cameras to detect drunk driving

Excessive alcohol consumption causes disability and death. Digital interventions are promising means to promote behavioral change and thus prevent alcohol-related harm, especially in critical moments such as driving. This requires real-time information on a person's blood alcohol concentration (BAC). Here, we develop an in-vehicle machine learning system to predict critical BAC levels. Our system leverages driver monitoring cameras mandated in numerous countries worldwide. We evaluate our system with n=30 participants in an interventional simulator study. Our system reliably detects driving under any alcohol influence (area under the receiver operating characteristic curve [AUROC] 0.88) and driving above the WHO recommended limit of 0.05g/dL BAC (AUROC 0.79). Model inspection reveals reliance on pathophysiological effects associated with alcohol consumption. To our knowledge, we are the first to rigorously evaluate the use of driver monitoring cameras for detecting drunk driving. Our results highlight the potential of driver monitoring cameras and enable next-generation drunk driver interaction preventing alcohol-related harm