Low-cost vehicle driver assistance system for fatigue and distraction detection

In recent years, the automotive industry is equipping vehicles with sophisticated, and often, expensive systems for driving assistance. However, this vehicular technology is more focused on facilitating the driving and not in monitoring the driver. This paper presents a low-cost vehicle driver assistance system for monitoring the drivers activity that intends to prevent an accident. The system consists of 4 sensors that monitor physical parameters and driver position. From these values, the system generates a series of acoustic signals to alert the vehicle driver and avoiding an accident. Finally the system is tested to verify its proper operation.This work has been partially supported by the “Programa para la Formación de Personal Investigador–(FPI-2015-S2-884)” by the “Universitat Politècnica de València”.Sendra, S.; García-García, L.; Jimenez, JM.; Lloret, J. (2017). Low-cost vehicle driver assistance system for fatigue and distraction detection. En Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer Verlag. 69-78. doi:10.1007/978-3-319-51207-5_7S6978Mapfre Foundation. (Online Article) Seguridad activa y pasiva. www.seguridadvialenlaempresa.com/seguridad-empresas/actualidad/noticias/seguridad-vial-activa-y-pasiva-2.jsp . Accessed 25 Aug 2016Dirección general de tráfico, Ministerio del Interior, Spanish Government. (Online Article) Las principales cifras de la siniestralidad vial. España 2014, p. 21 (2014). http://www.dgt.es/es/seguridad-vial/estadisticas-e-indicadores/publicaciones/ . Accessed 25 Aug 2016Fukuhara, H.: Vehicle collision alert system. US Patent 5355118 A, 11 Oct 1994Dirección general de tráfico, Ministerio del Interior, Spanish Government. (Online Article) Anuario estadístico de accidentes 2014, p. 10 (2014). http://www.dgt.es/es/seguridad-vial/estadisticas-e-indicadores/publicaciones/anuario-estadistico-general/ . Accessed 25 Aug 2016Dirección general de tráfico, Ministerio del Interior, Spanish Government. (Online Article) Otros factores de riesgo: La fatiga. http://www.dgt.es/PEVI/documentos/catalogo_recursos/didacticos/did_adultas/fatiga.pdf . Accessed 25 Aug 2016Seeing machines web page. https://www.seeingmachines.com/ . Accessed 25 Aug 2016Sigari, M.H., Pourshahabi, M.R., Soryani, M., Fathy, M.: A review on driver face monitoring systems for fatigue and distraction detection. Int. J. Adv. Sci. Technol. 64, 73–100 (2014). http://dx.doi.org/10.14257/ijast.2014.64.07Kutila, M., Jokela, M., Markkula, G., Romera Rue, M.: Driver distraction detection with a camera vision system. In: 14th IEEE International Conference on Image Processing (ICIP 2007), San Antonio, TX, USA, 16–19 September 2007. doi: 10.1109/ICIP.2007.4379556Rezaei, M., Klette, R.: 3D cascade of classifiers for open and closed eye detection in driver distraction monitoring. In: Real, P., Diaz-Pernil, D., Molina-Abril, H., Berciano, A., Kropatsch, W. (eds.) CAIP 2011. LNCS, vol. 6855, pp. 171–179. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23678-5_19Mbouna, R.O., Kong, S.G., Chun, M.G.: Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Trans. Intell. Transp. Syst. 14(3), 1462–1469 (2013). doi: 10.1109/TITS.2013.2262098Wahlstrom, E., Masoud, O., Papanikolopoulos, N.: Vision-based methods for driver monitoring. In: Proceedings of the International Conference on Intelligent Transportation Systems, vol. 2, pp. 903–908 (2003)Cherrat, L., Ezziyyani, M., El Mouden, A., Hassar, M.: Security and surveillance system for drivers based on user profile and learning systems for face recognition. Netw. Protoc. Algorithms 7(1), 98–118 (2015). doi: http://dx.doi.org/10.5296/npa.v7i1.7151Dong, Y., Hu, Z., Uchimura, K., Murayama, N.: Driver inattention monitoring system for intelligent vehicles: a review. IEEE Trans. Intell. Transp. Syst. 12(2), 596–614 (2011). doi: 10.1109/TITS.2010.2092770Force Sensitive Resistor features. http://www.trossenrobotics.com/productdocs/2010-10-26-DataSheet-FSR402-Layout2.pdf . Accessed 25 Aug 2016Louiza, M., Samira, M.: A new framework for request-driven data harvesting in vehicular sensor networks. Netw. Protoc. Algorithms 5(4), 1–18 (2013)Yao, H., Si, P., Yang, R., Zhang, Y.: Dynamic spectrum management with movement prediction in vehicular ad hoc networks. Ad Hoc Sens. Wirel. Netw. 32(1), 79–97 (2016

Evaluation and Validation of Distraction Detection Algorithms on Multiple Data Sources

This study aims to evaluate algorithms designed to detect distracted driving. This includes the comparison of how efficiently they detect the state of distraction and likelihood of a crash. Four algorithms that utilize measures of cumulative glance, past glance behavior, and glance eccentricity were used to understand the distracted state of the driver and were validated on two separate data sources (i.e., simulator and naturalistic data). Additionally, an independent method for distraction detection was designed using data mining methods. This approach utilized measures like steering degree, lane offset, lateral and longitudinal velocity, and acceleration. The results showed a higher likelihood of distracted events when cumulative glances were considered. However, the state of distraction was observed to be higher when glance eccentricity was added. Additionally, it was observed that glance behavior using the four legacy algorithms were better detectors of the state of distraction as compared to the data mining method that used vehicular measures. This research has implications in understanding the state of distraction, predicting the power of different methods, and comparing approaches in different contexts (naturalistic vs simulator). These findings provide the fundamental building blocks towards designing advanced mitigation systems that give drivers feedback in instances of high crash likelihood

The cockpit for the 21st century

Interactive surfaces are a growing trend in many domains. As one possible manifestation of Mark Weiser’s vision of ubiquitous and disappearing computers in everywhere objects, we see touchsensitive screens in many kinds of devices, such as smartphones, tablet computers and interactive tabletops. More advanced concepts of these have been an active research topic for many years. This has also influenced automotive cockpit development: concept cars and recent market releases show integrated touchscreens, growing in size. To meet the increasing information and interaction needs, interactive surfaces offer context-dependent functionality in combination with a direct input paradigm. However, interfaces in the car need to be operable while driving. Distraction, especially visual distraction from the driving task, can lead to critical situations if the sum of attentional demand emerging from both primary and secondary task overextends the available resources. So far, a touchscreen requires a lot of visual attention since its flat surface does not provide any haptic feedback. There have been approaches to make direct touch interaction accessible while driving for simple tasks. Outside the automotive domain, for example in office environments, concepts for sophisticated handling of large displays have already been introduced. Moreover, technological advances lead to new characteristics for interactive surfaces by enabling arbitrary surface shapes. In cars, two main characteristics for upcoming interactive surfaces are largeness and shape. On the one hand, spatial extension is not only increasing through larger displays, but also by taking objects in the surrounding into account for interaction. On the other hand, the flatness inherent in current screens can be overcome by upcoming technologies, and interactive surfaces can therefore provide haptically distinguishable surfaces. This thesis describes the systematic exploration of large and shaped interactive surfaces and analyzes their potential for interaction while driving. Therefore, different prototypes for each characteristic have been developed and evaluated in test settings suitable for their maturity level. Those prototypes were used to obtain subjective user feedback and objective data, to investigate effects on driving and glance behavior as well as usability and user experience. As a contribution, this thesis provides an analysis of the development of interactive surfaces in the car. Two characteristics, largeness and shape, are identified that can improve the interaction compared to conventional touchscreens. The presented studies show that large interactive surfaces can provide new and improved ways of interaction both in driver-only and driver-passenger situations. Furthermore, studies indicate a positive effect on visual distraction when additional static haptic feedback is provided by shaped interactive surfaces. Overall, various, non-exclusively applicable, interaction concepts prove the potential of interactive surfaces for the use in automotive cockpits, which is expected to be beneficial also in further environments where visual attention needs to be focused on additional tasks.Der Einsatz von interaktiven Oberflächen weitet sich mehr und mehr auf die unterschiedlichsten Lebensbereiche aus. Damit sind sie eine mögliche Ausprägung von Mark Weisers Vision der allgegenwärtigen Computer, die aus unserer direkten Wahrnehmung verschwinden. Bei einer Vielzahl von technischen Geräten des täglichen Lebens, wie Smartphones, Tablets oder interaktiven Tischen, sind berührungsempfindliche Oberflächen bereits heute in Benutzung. Schon seit vielen Jahren arbeiten Forscher an einer Weiterentwicklung der Technik, um ihre Vorteile auch in anderen Bereichen, wie beispielsweise der Interaktion zwischen Mensch und Automobil, nutzbar zu machen. Und das mit Erfolg: Interaktive Benutzeroberflächen werden mittlerweile serienmäßig in vielen Fahrzeugen eingesetzt. Der Einbau von immer größeren, in das Cockpit integrierten Touchscreens in Konzeptfahrzeuge zeigt, dass sich diese Entwicklung weiter in vollem Gange befindet. Interaktive Oberflächen ermöglichen das flexible Anzeigen von kontextsensitiven Inhalten und machen eine direkte Interaktion mit den Bildschirminhalten möglich. Auf diese Weise erfüllen sie die sich wandelnden Informations- und Interaktionsbedürfnisse in besonderem Maße. Beim Einsatz von Bedienschnittstellen im Fahrzeug ist die gefahrlose Benutzbarkeit während der Fahrt von besonderer Bedeutung. Insbesondere visuelle Ablenkung von der Fahraufgabe kann zu kritischen Situationen führen, wenn Primär- und Sekundäraufgaben mehr als die insgesamt verfügbare Aufmerksamkeit des Fahrers beanspruchen. Herkömmliche Touchscreens stellen dem Fahrer bisher lediglich eine flache Oberfläche bereit, die keinerlei haptische Rückmeldung bietet, weshalb deren Bedienung besonders viel visuelle Aufmerksamkeit erfordert. Verschiedene Ansätze ermöglichen dem Fahrer, direkte Touchinteraktion für einfache Aufgaben während der Fahrt zu nutzen. Außerhalb der Automobilindustrie, zum Beispiel für Büroarbeitsplätze, wurden bereits verschiedene Konzepte für eine komplexere Bedienung großer Bildschirme vorgestellt. Darüber hinaus führt der technologische Fortschritt zu neuen möglichen Ausprägungen interaktiver Oberflächen und erlaubt, diese beliebig zu formen. Für die nächste Generation von interaktiven Oberflächen im Fahrzeug wird vor allem an der Modifikation der Kategorien Größe und Form gearbeitet. Die Bedienschnittstelle wird nicht nur durch größere Bildschirme erweitert, sondern auch dadurch, dass Objekte wie Dekorleisten in die Interaktion einbezogen werden können. Andererseits heben aktuelle Technologieentwicklungen die Restriktion auf flache Oberflächen auf, so dass Touchscreens künftig ertastbare Strukturen aufweisen können. Diese Dissertation beschreibt die systematische Untersuchung großer und nicht-flacher interaktiver Oberflächen und analysiert ihr Potential für die Interaktion während der Fahrt. Dazu wurden für jede Charakteristik verschiedene Prototypen entwickelt und in Testumgebungen entsprechend ihres Reifegrads evaluiert. Auf diese Weise konnten subjektives Nutzerfeedback und objektive Daten erhoben, und die Effekte auf Fahr- und Blickverhalten sowie Nutzbarkeit untersucht werden. Diese Dissertation leistet den Beitrag einer Analyse der Entwicklung von interaktiven Oberflächen im Automobilbereich. Weiterhin werden die Aspekte Größe und Form untersucht, um mit ihrer Hilfe die Interaktion im Vergleich zu herkömmlichen Touchscreens zu verbessern. Die durchgeführten Studien belegen, dass große Flächen neue und verbesserte Bedienmöglichkeiten bieten können. Außerdem zeigt sich ein positiver Effekt auf die visuelle Ablenkung, wenn zusätzliches statisches, haptisches Feedback durch nicht-flache Oberflächen bereitgestellt wird. Zusammenfassend zeigen verschiedene, untereinander kombinierbare Interaktionskonzepte das Potential interaktiver Oberflächen für den automotiven Einsatz. Zudem können die Ergebnisse auch in anderen Bereichen Anwendung finden, in denen visuelle Aufmerksamkeit für andere Aufgaben benötigt wird

On driver behavior recognition for increased safety:A roadmap

Advanced Driver-Assistance Systems (ADASs) are used for increasing safety in the automotive domain, yet current ADASs notably operate without taking into account drivers’ states, e.g., whether she/he is emotionally apt to drive. In this paper, we first review the state-of-the-art of emotional and cognitive analysis for ADAS: we consider psychological models, the sensors needed for capturing physiological signals, and the typical algorithms used for human emotion classification. Our investigation highlights a lack of advanced Driver Monitoring Systems (DMSs) for ADASs, which could increase driving quality and security for both drivers and passengers. We then provide our view on a novel perception architecture for driver monitoring, built around the concept of Driver Complex State (DCS). DCS relies on multiple non-obtrusive sensors and Artificial Intelligence (AI) for uncovering the driver state and uses it to implement innovative Human–Machine Interface (HMI) functionalities. This concept will be implemented and validated in the recently EU-funded NextPerception project, which is briefly introduced

Context-Aware Driver Distraction Severity Classification using LSTM Network

Advanced Driving Assistance Systems (ADAS) has been a critical component in vehicles and vital to the safety of vehicle drivers and public road transportation systems. In this paper, we present a deep learning technique that classifies drivers’ distraction behaviour using three contextual awareness parameters: speed, manoeuver and event type. Using a video coding taxonomy, we study drivers’ distractions based on events information from Regions of Interest (RoI) such as hand gestures, facial orientation and eye gaze estimation. Furthermore, a novel probabilistic (Bayesian) model based on the Long shortterm memory (LSTM) network is developed for classifying driver’s distraction severity. This paper also proposes the use of frame-based contextual data from the multi-view TeleFOT naturalistic driving study (NDS) data monitoring to classify the severity of driver distractions. Our proposed methodology entails recurrent deep neural network layers trained to predict driver distraction severity from time series data

Employing Emerging Technologies to Develop and Evaluate In-Vehicle Intelligent Systems for Driver Support: Infotainment AR HUD Case Study

The plurality of current infotainment devices within the in-vehicle space produces an unprecedented volume of incoming data that overwhelm the typical driver, leading to higher collision probability. This work presents an investigation to an alternative option which aims to manage the incoming information while offering an uncluttered and timely manner of presenting and interacting with the incoming data safely. The latter is achieved through the use of an augmented reality (AR) head-up display (HUD) system, which projects the information within the driver’s field of view. An uncluttered gesture recognition interface provides the interaction with the AR visuals. For the assessment of the system’s effectiveness, we developed a full-scale virtual reality driving simulator which immerses the drivers in challenging, collision-prone, scenarios. The scenarios unfold within a digital twin model of the surrounding motorways of the city of Glasgow. The proposed system was evaluated in contrast to a typical head-down display (HDD) interface system by 30 users, showing promising results that are discussed in detail

Employing Emerging Technologies to Develop and Evaluate In-Vehicle Intelligent Systems for Driver Support: Infotainment AR HUD Case Study

The plurality of current infotainment devices within the in-vehicle space produces an unprecedented volume of incoming data that overwhelm the typical driver, leading to higher collision probability. This work presents an investigation to an alternative option which aims to manage the incoming information while offering an uncluttered and timely manner of presenting and interacting with the incoming data safely. The latter is achieved through the use of an augmented reality (AR) head-up display (HUD) system, which projects the information within the driver’s field of view. An uncluttered gesture recognition interface provides the interaction with the AR visuals. For the assessment of the system’s effectiveness, we developed a full-scale virtual reality driving simulator which immerses the drivers in challenging, collision-prone, scenarios. The scenarios unfold within a digital twin model of the surrounding motorways of the city of Glasgow. The proposed system was evaluated in contrast to a typical head-down display (HDD) interface system by 30 users, showing promising results that are discussed in detail

Vision-based Detection of Mobile Device Use While Driving

The aim of this study was to explore the feasibility of an automatic vision-based solution to detect drivers using mobile devices while operating their vehicles. The proposed system comprises of modules for vehicle license plate localisation, driver’s face detection and mobile phone interaction. The system were then implemented and systematically evaluated using suitable image datasets. The strengths and weaknesses of individual modules were analysed and further recommendations made to improve the overall system’s performance

A Context Aware Classification System for Monitoring Driver’s Distraction Levels

Understanding the safety measures regarding developing self-driving futuristic cars is a concern for decision-makers, civil society, consumer groups, and manufacturers. The researchers are trying to thoroughly test and simulate various driving contexts to make these cars fully secure for road users. Including the vehicle’ surroundings offer an ideal way to monitor context-aware situations and incorporate the various hazards. In this regard, different studies have analysed drivers’ behaviour under different case scenarios and scrutinised the external environment to obtain a holistic view of vehicles and the environment. Studies showed that the primary cause of road accidents is driver distraction, and there is a thin line that separates the transition from careless to dangerous. While there has been a significant improvement in advanced driver assistance systems, the current measures neither detect the severity of the distraction levels nor the context-aware, which can aid in preventing accidents. Also, no compact study provides a complete model for transitioning control from the driver to the vehicle when a high degree of distraction is detected. The current study proposes a context-aware severity model to detect safety issues related to driver’s distractions, considering the physiological attributes, the activities, and context-aware situations such as environment and vehicle. Thereby, a novel three-phase Fast Recurrent Convolutional Neural Network (Fast-RCNN) architecture addresses the physiological attributes. Secondly, a novel two-tier FRCNN-LSTM framework is devised to classify the severity of driver distraction. Thirdly, a Dynamic Bayesian Network (DBN) for the prediction of driver distraction. The study further proposes the Multiclass Driver Distraction Risk Assessment (MDDRA) model, which can be adopted in a context-aware driving distraction scenario. Finally, a 3-way hybrid CNN-DBN-LSTM multiclass degree of driver distraction according to severity level is developed. In addition, a Hidden Markov Driver Distraction Severity Model (HMDDSM) for the transitioning of control from the driver to the vehicle when a high degree of distraction is detected. This work tests and evaluates the proposed models using the multi-view TeleFOT naturalistic driving study data and the American University of Cairo dataset (AUCD). The evaluation of the developed models was performed using cross-correlation, hybrid cross-correlations, K-Folds validation. The results show that the technique effectively learns and adopts safety measures related to the severity of driver distraction. In addition, the results also show that while a driver is in a dangerous distraction state, the control can be shifted from driver to vehicle in a systematic manner