Personalized driver workload inference by learning from vehicle related measurements

Adapting in-vehicle systems (e.g. Advanced Driver Assistance Systems, In-Vehicle Information Systems) to individual drivers’ workload can enhance safety and convenience. To make this possible, it is a prerequisite to infer driver workload so that adaptive aiding can be provided to the driver at the right time and in a proper manner. Rather than developing an average model for all drivers, a Personalized Driver Workload Inference (PDWI) system considering individual drivers’ driving characteristics is developed using machine learning techniques via easily accessed Vehicle Related Measurements (VRMs). The proposed PDWI system comprises two stages. In offline training, individual drivers’ workload is first automatically splitted into different categories according to its inherent data characteristics using Fuzzy C means clustering. Then an implicit mapping between VRMs and different levels of workload is constructed via classification algorithms. In online implementation, VRMs samples are classified into different clusters, consequently driver workload can be successfully inferred. A recently collected dataset from real-world naturalistic driving experiments is drawn to validate the proposed PDWI system. Comparative experimental results indicate that the proposed framework integrating Fuzzy C-means clustering and Support Vector Machine classifier provides a promising workload recognition performance in terms of accuracy, precision, recall, F1-score and prediction time. The inter-individual differences in term of workload are also identified and can be accommodated by the proposed framework due to its adaptiveness

Personalized Driver Workload Inference by Learning From Vehicle Related Measurements

Adapting in-vehicle systems (e.g., advanced driver assistance systems and in-vehicle information systems) to individual drivers' workload can enhance both safety and convenience. To make this possible, it is a prerequisite to infer driver workload so that adaptive aiding can be provided to the driver at the right time and in an appropriate manner. Rather than developing an average model for all drivers, a personalized driver workload inference (PDWI) system considering individual drivers driving characteristics is developed using machine learning techniques via easily accessed vehicle related measurements (VRMs). The proposed PDWI system comprises two stages. In offline training, individual drivers workload is first automatically splitted into different categories according to its inherent data characteristics using fuzzy C-means (FCM) clustering. Then an implicit mapping between VRMs and different levels of workload is constructed via classification algorithms. In online implementation, VRMs samples are classified into different clusters, consequently driver workload type can be successfully inferred. A recently collected dataset from real-world naturalistic driving experiments is drawn to validate the proposed PDWI system. Comparative experimental results indicate that the proposed framework integrating FCM clustering and support vector machine classifier provides a promising workload recognition performance in terms of accuracy, precision, recall, F 1 -score, and prediction time. The interindividual differences in term of workload are also identified and can be accommodated by the proposed framework due to its adaptiveness

Implicit personalization in driving assistance: State-of-the-art and open issues

In recent decades, driving assistance systems have been evolving towards personalization for adapting to different drivers. With the consideration of driving preferences and driver characteristics, these systems become more acceptable and trustworthy. This article presents a survey on recent advances in implicit personalized driving assistance. We classify the collection of work into three main categories: 1) personalized Safe Driving Systems (SDS), 2) personalized Driver Monitoring Systems (DMS), and 3) personalized In-vehicle Information Systems (IVIS). For each category, we provide a comprehensive review of current applications and related techniques along with the discussion of industry status, benefits of personalization, application prospects, and future focal points. Both relevant driving datasets and open issues about personalized driving assistance are discussed to facilitate future research. By creating an organized categorization of the field, we hope that this survey could not only support future research and the development of new technologies for personalized driving assistance but also facilitate the application of these techniques within the driving automation community</h2

A machine learning based personalized system for driving state recognition

Reliable driving state recognition (e.g. normal, drowsy, and aggressive) plays a significant role in improving road safety, driving experience and fuel efficiency. It lays the foundation for a number of advanced functions such as driver safety monitoring systems and adaptive driving assistance systems. In these applications, state recognition accuracy is of paramount importance to guarantee user acceptance. This paper is mainly focused on developing a personalized driving state recognition system by learning from non-intrusive, easily accessible vehicle related measurements and its validation using real-world driving data. Compared to conventional approaches, this paper first highlights the necessities of adopting a personalized system by analysing feature distribution of individual driver’s data and all drivers’ data via advanced data visualization and statistical analysis. If significant differences are identified, a dedicated personalized model is learnt to predict the driver’s driving state. Spearman distance is also drawn to evaluate the differences between individual driver’s data and all drivers’ data in a quantitative manner. In addition, five categories of classifiers are tested and compared to identify a suitable one for classification, where random forest with Bayesian parameter optimization outperforms others and therefore is adopted in this paper. A recently collected dataset from real-world driving experiments is adopted to evaluate the proposed system. Comparative experimental results indicate that the personalized learning system with road information significantly outperforms conventional approaches without considering personalized characteristics or road information, where the overall accuracy increases from 81.3% to 91.6%. It is believed that the newly developed personalized learning system can find a wide range of applications where diverse behaviours exist

Driver Profiling and Bayesian Workload Estimation Using Naturalistic Peripheral Detection Study Data

Monitoring drivers' mental workload facilitates initiating and maintaining safe interactions with in-vehicle information systems, and thus delivers adaptive human machine interaction with reduced impact on the primary task of driving. In this paper, we tackle the problem of workload estimation from driving performance data. First, we present a novel on-road study for collecting subjective workload data via a modified peripheral detection task in naturalistic settings. Key environmental factors that induce a high mental workload are identified via video analysis, e.g. junctions and behaviour of vehicle in front. Second, a supervised learning framework using state-of-the-art time series classifiers (e.g. convolutional neural network and transform techniques) is introduced to profile drivers based on the average workload they experience during a journey. A Bayesian filtering approach is then proposed for sequentially estimating, in (near) real-time, the driver's instantaneous workload. This computationally efficient and flexible method can be easily personalised to a driver (e.g. incorporate their inferred average workload profile), adapted to driving/environmental contexts (e.g. road type) and extended with data streams from new sources. The efficacy of the presented profiling and instantaneous workload estimation approaches are demonstrated using the on-road study data, showing $F_{1}$ scores of up to 92% and 81%, respectively.Comment: Accepted for IEEE Transactions on Intelligent Vehicle

New Driver Workload Prediction Using Clustering-Aided Approaches

Awareness of driver workload (DW) plays a paramount role in enhancing driving safety and convenience for intelligent vehicles. The DW prediction systems proposed so far learn either from individual driver's data (termed personalized system) or existing drivers' data indiscriminately (termed average system). As a result, they either do not work or lead to a limited performance for new drivers without labeled data. To this end, we develop clustering-aided approaches exploiting group characteristics of the existing drivers' data. Two clustering aided predictors are proposed. The first is clustering-aided regression (CAR) model, where the regression model for the cluster with the highest likelihood is adopted. The second is clustering-aided multiple model regression model, where the concept of multiple models is further augmented to CAR. A recent dataset from real-world driving experiments is adopted to validate the algorithms. Comparative results against the conventional average system demonstrate that by incorporating clustering information, both the proposed approaches significantly improve workload prediction performance

Real-time Human Workload Estimation and Its Application in Adaptive Haptic Shared Control

Automated vehicles (AVs) are promising to have the potential to reduce driving-related injuries and deaths. However, autonomous driving technology is currently limited in its scope and reliability, giving rise to the semi-autonomous driving model, where the autonomy and the human share the control of the vehicle. Workload, despite being an important human factor, has not yet been considered when designing adaptive shared control. Recently, researchers have started to apply machine learning techniques to classify mental workload into different levels. However, most of these studies have adopted either a single-model-single-feature approach or a single-model-all-features approach. However different machine learning models are suitable for different features, how to leverage different models for different features is critical. To address these shortcomings and research gaps, the goals of this dissertation were to (1) examine whether and to what extent haptic shared control performance can be improved by incorporating operators' workload; (2) develop a computational model for workload estimation, and the model should be able to leverage different machine learning models that work best for different features; and (3) investigate the generalizability of the workload estimation model. To address these research goals, this dissertation was composed of four research phases with two pilot studies and four human subject experiments. (1) Collaborating with Yifan Weng, Dr. Tulga Ersal, and Prof. Jeffrey Stein from the Department of Mechanical Engineering at the University of Michigan, we developed a teleoperated dual-task shared control simulation platform where the human shared control of a ground vehicle with autonomy while performing a surveillance task simultaneously. In addition, we developed a real-time eye-tracking system based on Tobii Pro Glasses 2 to measure the human gaze points in a world frame and pupil sizes. (2) We proposed a workload-adaptive haptic shared control scheme together with our collaborators. We conducted two human subject experiments during this phase. The results indicated that the proposed workload-adaptive haptic shared control scheme can reduce human workload, increase human trust in the system, increase driving performance, and reduce human effort without sacrificing surveillance task performance. (3) We proposed a Bayesian inference model for workload estimation that can leverage the different machine learning models that work best for different features. Specifically, we used support-vector machines (SVMs) for pupil size change, the Hidden Markov Model (HMM) for gaze trajectory, SVMs for fixation feature, and Gaussian Mixture Models (GMMs) for fixation trajectory. The empirical results indicated that our proposed model achieved a 0.82 F1 score for workload imposed by varying surveillance task urgency. (4) We investigated the generalizability of our proposed Bayesian inference model for workload estimation by conducting two human subject experiments with 24 participants and using different factors to impose human workload, i.e., obstacle headway and driving speed. The results indicated that our proposed model achieved a 0.68 F1 score for the workload imposed by obstacle avoidance and the personalized version of our proposed model can distinguish the workload imposed by different driving speeds under high surveillance task urgency.PHDRoboticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169623/1/ruikunl_1.pd

Tune in to your emotions: a robust personalized affective music player

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application

On driver behavior recognition for increased safety:A roadmap

Advanced Driver-Assistance Systems (ADASs) are used for increasing safety in the automotive domain, yet current ADASs notably operate without taking into account drivers’ states, e.g., whether she/he is emotionally apt to drive. In this paper, we first review the state-of-the-art of emotional and cognitive analysis for ADAS: we consider psychological models, the sensors needed for capturing physiological signals, and the typical algorithms used for human emotion classification. Our investigation highlights a lack of advanced Driver Monitoring Systems (DMSs) for ADASs, which could increase driving quality and security for both drivers and passengers. We then provide our view on a novel perception architecture for driver monitoring, built around the concept of Driver Complex State (DCS). DCS relies on multiple non-obtrusive sensors and Artificial Intelligence (AI) for uncovering the driver state and uses it to implement innovative Human–Machine Interface (HMI) functionalities. This concept will be implemented and validated in the recently EU-funded NextPerception project, which is briefly introduced