The interplay between task difficulty and microsaccade rate: Evidence for the critical role of visual load

In previous research, microsaccades have been suggested as psychophysiological indicators of task load. So far, it is still under debate how different types of task demands are influencing microsaccade rate. This piece of research examines the relation between visual load, mental load and microsaccade rate. Fourteen participants carried out a continuous performance task (n-back), in which visual (letters vs. abstract figures) and mental task load (1-back to 4-back) were manipulated as within-subjects variables. Eye tracking data, performance data as well as subjective workload were recorded. Data analysis revealed an increased level of microsaccade rate for stimuli of high visual demand (i.e. abstract figures), while mental demand (n-back-level) did not modulate microsaccade rate. In conclusion, the present results suggest that microsaccade rate reflects visual load of a task rather than its mental load

Classification of Drivers' Workload Using Physiological Signals in Conditional Automation

The use of automation in cars is increasing. In future vehicles, drivers will no longer be in charge of the main driving task and may be allowed to perform a secondary task. However, they might be requested to regain control of the car if a hazardous situation occurs (i.e., conditionally automated driving). Performing a secondary task might increase drivers' mental workload and consequently decrease the takeover performance if the workload level exceeds a certain threshold. Knowledge about the driver's mental state might hence be useful for increasing safety in conditionally automated vehicles. Measuring drivers' workload continuously is essential to support the driver and hence limit the number of accidents in takeover situations. This goal can be achieved using machine learning techniques to evaluate and classify the drivers' workload in real-time. To evaluate the usefulness of physiological data as an indicator for workload in conditionally automated driving, three physiological signals from 90 subjects were collected during 25 min of automated driving in a fixed-base simulator. Half of the participants performed a verbal cognitive task to induce mental workload while the other half only had to monitor the environment of the car. Three classifiers, sensor fusion and levels of data segmentation were compared. Results show that the best model was able to successfully classify the condition of the driver with an accuracy of 95%. In some cases, the model benefited from sensors' fusion. Increasing the segmentation level (e.g., size of the time window to compute physiological indicators) increased the performance of the model for windows smaller than 4 min, but decreased for windows larger than 4 min. In conclusion, the study showed that a high level of drivers' mental workload can be accurately detected while driving in conditional automation based on 4-min recordings of respiration and skin conductance

Relevant Physiological Indicators for Assessing Workload in Conditionally Automated Driving, Through Three-Class Classification and Regression

In future conditionally automated driving, drivers may be asked to take over control of the car while it is driving autonomously. Performing a non-driving-related task could degrade their takeover performance, which could be detected by continuous assessment of drivers' mental load. In this regard, three physiological signals from 80 subjects were collected during 1 h of conditionally automated driving in a simulator. Participants were asked to perform a non-driving cognitive task (N-back) for 90 s, 15 times during driving. The modality and difficulty of the task were experimentally manipulated. The experiment yielded a dataset of drivers' physiological indicators during the task sequences, which was used to predict drivers' workload. This was done by classifying task difficulty (three classes) and regressing participants' reported level of subjective workload after each task (on a 0–20 scale). Classification of task modality was also studied. For each task, the effect of sensor fusion and task performance were studied. The implemented pipeline consisted of a repeated cross validation approach with grid search applied to three machine learning algorithms. The results showed that three different levels of mental load could be classified with a f1-score of 0.713 using the skin conductance and respiration signals as inputs of a random forest classifier. The best regression model predicted the subjective level of workload with a mean absolute error of 3.195 using the three signals. The accuracy of the model increased with participants' task performance. However, classification of task modality (visual or auditory) was not successful. Some physiological indicators such as estimates of respiratory sinus arrhythmia, respiratory amplitude, and temporal indices of heart rate variability were found to be relevant measures of mental workload. Their use should be preferred for ongoing assessment of driver workload in automated driving

Effect of Obstacle Type and Cognitive Task on Situation Awareness and Takeover Performance in Conditionally Automated Driving

In conditionally automated driving, several factors can affect the driver’s situation awareness and ability to take over control. To better understand the influence of some of these factors, 88 participants spent 20 minutes in a conditionally automated driving simulator. They had to react to four obstacles that varied in danger and movement. Half of the participants were required to engage in a verbal cognitive non-driving-related task. Situation awareness, takeover performance and physiological responses were measured for each situation. The results suggest that obstacle movement influences obstacle danger perception, situation awareness, and response time, while the latter is also influenced by obstacle danger. The cognitive verbal task also had an effect on the takeover response time. These results imply that the driver’s cognitive state and the driving situation (e.g. the movement/danger of entities present around the vehicle) must be considered when conveying information to drivers through in-vehicle interfaces

A dataset on the physiological state and behavior of drivers in conditionally automated driving

This dataset contains data of 346 drivers collected during six experiments conducted in a fixed-base driving simulator. Five studies simulated conditionally automated driving (L3 SAE), and the other one simulated manual driving (L0-SAE). The dataset includes physiological data (electrocardiogram (ECG), electrodermal activity (EDA), and respiration (RESP)), driving and behavioral data (reaction time, steering wheel angle, …), performance data of non-driving-related tasks, and questionnaire responses. Among them, measures from standardized questionnaires were collected, either to control the experimental manipulation of the driver's state, or to measure constructs related to human factors and driving safety (drowsiness, mental workload, affective state, situation awareness, situational trust, user experience). In the provided dataset, some raw data have been processed, notably physiological data from which physiological indicators (or features) have been calculated. The latter can be used as input for machine learning models to predict various states (sleep deprivation, high mental workload, ...) that may be critical for driver safety. Subjective self-reported measures can also be used as ground truth to apply regression techniques. Besides that, statistical analyses can be performed using the dataset, in particular to analyze the situational awareness or the takeover quality of drivers, in different states and different driving scenarios. Overall, this dataset contributes to better understanding and consideration of the driver's state and behavior in conditionally automated driving. In addition, this dataset stimulates and inspires research in the fields of physiological/affective computing and human factors in transportation, and allows companies from the automotive industry to better design adapted human-vehicle interfaces for safe use of automated vehicles on the roads

Tâche secondaire et conscience de l'environnement, une application mobile pour véhicule semi-autonome

National audienceAutonomous vehicles are developing rapidly and will lead to a significant change in the driver's role: he/she will have to move from the role of actor to the role of supervisor. Indeed, the driver will soon be able to perform a secondary task but he/she must be able to take over control in the event of a critical situation that is not managed by the autonomous system. This implies that the role of new interfaces and interactions within the vehicle is important to take into account. This article describes the design of an application that provides the driver with information about the environment perceived by his/her vehicle in the form of modules. This application is displayed as split screen on a tablet by which a secondary task can be performed. Initial tests were carried out with this application in a driving simulator. They made it possible to test the acceptance of the application and the clarity of the information transmitted. The results generally showed that the participants correctly identified some of the factors limiting the proper functioning of the autonomous pilot while performing a secondary task on a tablet.Les véhicules autonomes se développent rapidement et entraîneront un changement de rôle important chez le conducteur: ce dernier sera amené à passer du rôle d'acteur à celui de superviseur. En effet, le conducteur sera bientôt en mesure d'effectuer une tâche secondaire mais devra toutefois être capable de reprendre le contrôle dans le cas d'une situation critique non gérée par le système autonome. Ceci implique que le rôle des nouvelles interfaces et interactions au sein du véhicule est important à prendre en compte. Cet article décrit la conception d'une application transmettant au conducteur des informations relatives à l'environnement perçu par son véhicule sous forme de modules. Cette application s'affiche en partage d'écran sur une tablette grâce à laquelle une tâche secondaire peut être effectuée. De premiers tests ont été effectués avec cette application dans un simulateur de conduite. Ils ont permis de tester l'acceptation de l'application et la clarté des informations transmises. Les résultats ont globalement montré que les participants ont correctement identifié certains facteurs limitant le bon fonctionnement du pilote autonome tout en réalisant une tâche secondaire sur tablette

Model of the driver's physiological state in conditionally automated driving

Road accidents are still one of the main causes of death in the world, despite all the technological advances made on cars since its invention. In particular, the driver's state is often the cause of these road accidents. To assist the driver, the level of automation of cars has increased in recent years, including advanced driver assistance systems. The next level of automation should be conditionally automated driving, where the driver is no longer responsible for the main driving task. In theory, this should reduce the number of accidents. But the fact that the driver can engage in non-driving related tasks could be very dangerous if the car suddenly requires him or her to take control. In addition, long periods of automated driving can also reduce drivers' alertness and ability to take over control in critical situations, up to causing an accident. In this regard, this thesis aims at proposing an approach to assess the driver's state continuously in the specific context of conditionally automated driving. This would allow to know if he or she is able to take over control when the car asks him or her. To achieve that goal, machine learning techniques and physiological signals were used to assess the driver's state. In particular, the prediction of four predictive risk factors was done as they are critical at this level of automation: fatigue, mental workload, affective state and situation awareness. The main contribution of this thesis is the design, implementation and validation of a model that assesses continuously the driver's state using physiological signals in conditionally automated driving (L3-SAE). This main contribution encompasses the realisation of sub-contributions to address several formulated research questions: the collection of a physiological dataset in the specific context of conditionally automated driving, the creation of a pipeline to train machine learning models able to predict the selected risk factors from the collected data, and a system to measure breathing non-intrusively. The results showed that the different risk factors could be predicted with an accuracy ranging from 73 to 99%. The fusion of physiological signals generally increased the accuracy, as did the segmentation of the signals. The physiological signals (or features) and the optimal time window to use to predict each risk factor are proposed from the results obtained with machine learning. The continuous predictions of the model in the final experiment were globally consistent and are encouraging for the use of this kind of model in our future cars. Furthermore, the sensor developed to measure breathing in a non-intrusive way proved that the breathing rate could be measured with an error of more or less one breath per second on average compared to a reference sensor. The work proposed in this thesis suggests that machine learning models can accurately predict various risk factors related to conditionally automated driving. This work can serve as a basis for improving driver condition assessment for automotive manufacturers. It can also be used in academic research, especially to further optimize the prediction of the different risk factors selected in this thesis.Les accidents de la route restent l'une des principales causes de décès dans le monde, malgré tous les progrès technologiques réalisés sur la voiture depuis son invention. En particulier, l'état du conducteur est souvent la cause de ces accidents de la route. Pour aider le conducteur, le niveau d'automatisation des voitures a augmenté ces dernières années, notamment avec les systèmes avancés d'aide à la conduite. Le prochain niveau d'automatisation devrait être la conduite conditionnellement automatisée, où le conducteur n'est plus responsable de la tâche principale de conduite. En théorie, cela devrait réduire le nombre d'accidents. Mais le fait que le conducteur puisse s'engager dans des tâches non liées à la conduite pourrait s'avérer très dangereux si la voiture lui demande soudainement de prendre le contrôle. En outre, de longues périodes de conduite automatisée peuvent également réduire la vigilance du conducteur et sa capacité à reprendre le contrôle dans des situations critiques, jusqu'à provoquer un accident. A cet égard, cette thèse vise à proposer une approche permettant d'évaluer l'état du conducteur en continu dans le contexte spécifique de la conduite automatisée conditionnelle. Pour atteindre cet objectif, des techniques d'apprentissage automatique et des signaux physiologiques ont été utilisés pour évaluer l'état du conducteur. En particulier, la prédiction de quatre facteurs de risque a été effectuée, étant jugés critiques à ce niveau d'automatisation : la fatigue, la charge mentale, l'état affectif et la conscience de situation. La principale contribution de cette thèse est la conception, l'implémentation et la validation d'un modèle qui évalue en continu l'état du conducteur à l'aide de signaux physiologiques en conduite conditionnellement automatisée (L3-SAE). Cette contribution principale comprend la réalisation de sous-contributions pour répondre à plusieurs questions de recherche formulées : la collecte d'un ensemble de données physiologiques dans le contexte spécifique de la conduite conditionnellement automatisée, la création d'un pipeline pour entraîner des modèles d'apprentissage automatique capables de prédire les facteurs de risque sélectionnés à partir des données collectées, et un système pour mesurer la respiration de manière non-intrusive. Les résultats ont montré que les différents facteurs de risque pouvaient être prédits avec une précision allant de 73 à 99%. La fusion des signaux physiologiques a généralement augmenté la précision, tout comme la segmentation des signaux. Les signaux physiologiques (ou les caractéristiques) et la fenêtre temporelle optimale à utiliser pour prédire chaque facteur de risque sont proposés à partir des résultats obtenus avec l'apprentissage automatique. Les prédictions continues du modèle dans l'expérience finale étaient globalement cohérentes et sont encourageantes pour l'utilisation de ce type de modèle dans nos futures voitures. En outre, le capteur développé pour mesurer la respiration de manière non intrusive a prouvé que le rythme respiratoire pouvait être mesuré avec une erreur de plus ou moins une respiration par seconde en moyenne par rapport à un capteur de référence. Le travail proposé dans cette thèse suggère que les modèles d'apprentissage automatique peuvent prédire avec précision divers facteurs de risque liés à la conduite conditionnellement automatisée. Ce travail peut servir de base pour améliorer l'évaluation de l'état du conducteur pour les constructeurs automobiles. Il peut également être utilisé dans la recherche académique, notamment pour optimiser davantage la prédiction des différents facteurs de risque sélectionnés dans cette thèse

A recommender system for increasing energy efficiency of solar-powered smart homes

Photovoltaic installations can be environmentally beneficial to a greater or lesser extent, depending on the conditions. If the energy produced is not used, it is redirected to the grid, otherwise a battery with a high ecological footprint is needed to store it. To alleviate this problem, an innovative recommender system is proposed for residents of smart homes equipped with battery-free solar panels to optimise the energy produced. Using artificial intelligence, the system is designed to predict the energy produced and consumed for the day ahead using three data sources: sensor logs from the home automation solution, data collected by the solar inverter, and weather data. Based on these predictions, recommendations are then generated and ranked by relevance. Data collected over 76 days were used to train two variants of the system, considering or without considering energy consumption. Recommendations selected by the system over 14 days were randomly picked to be evaluated for relevance, ranking, and diversity by 11 people. The results show that it is difficult to predict residents’ consumption based solely on sensor logs. On average, respondents reported that 74% of the recommendations were relevant, while the values contained in them (i.e., accuracy of times of day and kW energy) were accurate in 66% (variant 1) and 77% of cases (variant 2). Also, the ranking of the recommendations was considered logical in 91% and 88% of cases. Overall, residents of such solar-powered smart homes might be willing to use such a system to optimise the energy produced. However, further research should be conducted to improve the accuracy of the values contained in the recommendations

Workshop on Tangible xAI

The goal of this workshop is to discuss the potential benefits and the open challenges of giving a physical form to Artificial Intelligence (AI), towards the definition of tangible, or graspable AI. In the workshop we will focus on the use-case of a convolutional Neural Network for image analysis and we will carry out a hands-on paper prototyping activity to imagine tangible interactive AI-powered systems where critical parameters of the Neural Network are physicalized. Such systems have the potential to lower the barrier for AI education and for making AI systems more trustable and explainabl

Workshop on tangible xAI

The goal of this workshop is to discuss the potential benefits and the open challenges of giving a physical form to Artificial Intelligence (AI), towards the definition of tangible, or graspable AI. In the workshop we will focus on the use-case of a convolutional Neural Network for image analysis and we will carry out a hands-on paper prototyping activity to imagine tangible interactive AI-powered systems where critical parameters of the Neural Network are physicalized. Such systems have the potential to lower the barrier for AI education and for making AI systems more trustable and explainable