Effects of cognitive tasks on car drivers’ behaviors and physiological responses

The effects of drivers’ engagement in cognitive tasks (i.e., non-visual, cognitively loading activities unrelated to the task of driving) are debated and unclear. Numerous experiments show impaired driver behaviors, yet naturalistic studies typically do not find an increased crash risk. In the future, autonomous driving (AD) is expected to improve traffic safety while allowing safe engagement in cognitive (and other) tasks. Having the opportunity to perform non-driving related tasks while traveling may then motivate drivers to use AD, provided they can actually engage in the tasks. Unfortunately, research on drivers’ engagement in cognitive tasks suffers severe methodological limitations since reliable and unintrusive measures of cognitive load are lacking.The aim of this thesis is therefore to advance the understanding of task-induced cognitive load in the context of traffic safety. This aim is split into two objectives: A) to better understand how drivers’ involvement in cognitive tasks can affect safety-relevant driver behaviors and decisions and B) to provide methodological guidance about assessing cognitive load in drivers using physiological measures.To accomplish Objective A, effects of cognitive tasks on driver behaviors were studied during routine driving and in a safety-critical event in a driving simulator. Also, drivers’ ability to engage in a non-driving related task while using AD in real traffic was explored. In line with the cognitive control hypothesis (Engstr\uf6m et al., 2017), it was found that cognitive tasks negatively affected driver behaviors in situations where cognitive control was needed, for example in intersections—but not in a lead vehicle braking scenario where responses were triggered automatically by visual looming. It was also found that although the number of off-path glances decreased during cognitive load, the timing of the remaining glances was unaffected. Clearly, cognitive load has different effects on different mechanisms. When using AD, drivers were indeed capable of engaging in a non-driving related task—suggesting that AD will be able to fulfill drivers’ desire to perform such tasks while traveling, which may motivate AD usage and thus improve traffic safety (given that AD is truly safer than manual driving). Finally, a simulator study addressing Objective B showed that the measurability of cognitive load was greatly improved by recognizing that multiple coexisting mental responses give rise to different physiological responses. This approach can provide less context-dependent measurements and allows for a better, more detailed understanding of the effects of cognitive tasks.These findings can help improve traffic safety—both by being used in system development, and as part of the systems themselves

Psychophysiological responses to takeover requests in conditionally automated driving

In SAE Level 3 automated driving, taking over control from automation raises significant safety concerns because drivers out of the vehicle control loop have difficulty negotiating takeover transitions. Existing studies on takeover transitions have focused on drivers' behavioral responses to takeover requests (TORs). As a complement, this exploratory study aimed to examine drivers' psychophysiological responses to TORs as a result of varying non-driving-related tasks (NDRTs), traffic density and TOR lead time. A total number of 102 drivers were recruited and each of them experienced 8 takeover events in a high fidelity fixed-base driving simulator. Drivers' gaze behaviors, heart rate (HR) activities, galvanic skin responses (GSRs), and facial expressions were recorded and analyzed during two stages. First, during the automated driving stage, we found that drivers had lower heart rate variability, narrower horizontal gaze dispersion, and shorter eyes-on-road time when they had a high level of cognitive load relative to a low level of cognitive load. Second, during the takeover transition stage, 4s lead time led to inhibited blink numbers and larger maximum and mean GSR phasic activation compared to 7s lead time, whilst heavy traffic density resulted in increased HR acceleration patterns than light traffic density. Our results showed that psychophysiological measures can indicate specific internal states of drivers, including their workload, emotions, attention, and situation awareness in a continuous, non-invasive and real-time manner. The findings provide additional support for the value of using psychophysiological measures in automated driving and for future applications in driver monitoring systems and adaptive alert systems.University of Michigan McityPeer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/162593/1/AAP_physiological_responses_HF_template.pdfSEL

Systematic Review of Driver Distraction in the Context of Advanced Driver Assistance Systems (ADAS) & Automated Driving Systems (ADS)

Advanced Vehicle Systems promise improved safety and comfort for drivers. Steady advancements in technology are resulting in increasing levels of vehicle automation capabilities, furthering safety benefits. In fact, some of these vehicle automation systems are already deployed and available, but with promised benefits, such systems can potentially change driving behaviors. There is evidence that drivers have increased secondary task engagements while driving with automated vehicle systems, but there is a need for a clearer scientific understanding of any potential correlations between the use of automated vehicle systems and potentially negative driver behaviors. Therefore, this thesis aims to understand the state of knowledge on automated vehicle systems and their possible impact on drivers’ distraction behaviors. I have conducted two systematic literature reviews to examine this question. This thesis reports these reviews and examines the effects of secondary task engagement on driving behaviors such as take-over times, visual attention, trust, and workload, and discusses the implications on driver safety

Dynamic scan paths investigations under manual and highly automated driving

Active visual scanning of the scene is a key task-element in all forms of human locomotion. In the field of driving, steering (lateral control) and speed adjustments (longitudinal control) models are largely based on drivers’ visual inputs. Despite knowledge gained on gaze behaviour behind the wheel, our understanding of the sequential aspects of the gaze strategies that actively sample that input remains restricted. Here, we apply scan path analysis to investigate sequences of visual scanning in manual and highly automated simulated driving. Five stereotypical visual sequences were identified under manual driving: forward polling (i.e. far road explorations), guidance, backwards polling (i.e. near road explorations), scenery and speed monitoring scan paths. Previously undocumented backwards polling scan paths were the most frequent. Under highly automated driving backwards polling scan paths relative frequency decreased, guidance scan paths relative frequency increased, and automation supervision specific scan paths appeared. The results shed new light on the gaze patterns engaged while driving. Methodological and empirical questions for future studies are discussed.Active visual scanning of the scene is a key task-element in all forms of human locomotion. In the field of driving, steering (lateral control) and speed adjustments (longitudinal control) models are largely based on drivers’ visual inputs. Despite knowledge gained on gaze behaviour behind the wheel, our understanding of the sequential aspects of the gaze strategies that actively sample that input remains restricted. Here, we apply scan path analysis to investigate sequences of visual scanning in manual and highly automated simulated driving. Five stereotypical visual sequences were identified under manual driving: forward polling (i.e. far road explorations), guidance, backwards polling (i.e. near road explorations), scenery and speed monitoring scan paths. Previously undocumented backwards polling scan paths were the most frequent. Under highly automated driving backwards polling scan paths relative frequency decreased, guidance scan paths relative frequency increased, and automation supervision specific scan paths appeared. The results shed new light on the gaze patterns engaged while driving. Methodological and empirical questions for future studies are discussed.Peer reviewe

DRIVERS’ HAZARD AVOIDANCE DURING VEHICLE AUTOMATION: IMPACT OF MENTAL MODELS AND IMPLICATIONS FOR TRAINING

Advanced Driver Assistance Systems (ADAS) are vehicle automation systems that have become more accessible and prevalent in vehicles in recent years. But the introduction of such technologies introduces new human factors challenges. Past literature suggests that users of vehicle automation lack the necessary and appropriate knowledge about their automation system. This may play a negative role in their hazard avoidance abilities when driving with automation features. Improving mental models and knowledge could generally lead to safer interactions with vehicle automation systems, but any effort to develop hazard avoidance skills when driving with vehicle automation is impeded by the lack of literature regarding the subject. Moreover, it is possible hazard avoidance for vehicle automation may actually differ from that for traditional driving. For vehicle automation, system-related changes occurring internally inside one’s vehicle also impact how the system responds and controls the vehicle. Failure to recognize certain critical system changes may have disastrous consequences. Hence, it is imperative that a new framework for hazard avoidance in the new context of vehicle automation, especially for ADAS features, is conceptualized. Initially, the research focused on realizing exactly this by proposing a conceptual framework for hazard avoidance in the context of vehicle automation by making use of past literary sources on hazard avoidance for traditional driving. Next, the relationship between mental models, training, and hazard avoidance was mapped and each new behavioral construct of hazard avoidance focusing on awareness, detection, and responses based on internal events was assigned potential outcome measure. Next, an observational study was conducted with ten experienced users of Adaptive Cruise Control (ACC). Among them, five were assigned to an eye movements group and five others to a verbal responses group. The eye movement observations gave us insights into how experienced users detect and respond to hazards and how these affect their interactions and responses using their ACC systems. The verbal group also provided insights about the participants’ awareness during the drive which featured several edge-case and normal events. These observations imply that hazard avoidance behaviors actually differ in the context of ADAS compared to traditional driving. The findings from the observational study were leveraged when designing and developing a new training program where drivers would receive an immersive and realistic training experience through a Virtual Reality (VR) headset. The main objective of the training program was to improve the user’s mental models about ACC and also equip them with the necessary skills to avoid hazard during edge case events of ACC. Finally, an evaluation study was conducted with 36 novice ACC users on a driving simulator capable of simulating ACC operations. The participants were equally and randomly assigned to one of three group – the VR group that received the newly designed VR training program; the SD group that received training material with state diagram visualization of ACC and other information derived from owner’s manuals; or the BI group that received basic textual information about ACC. The participants’ mental models before and after training were measured using a mental models survey, and the simulator drive was designed to collect valuable data about the participants interactions with ACC and their hazard avoidance behaviors. Findings revealed that although the VR training program had some impact on the participants\u27 mental models and hazard avoidance behaviors, the impact was not statistically significant. However, the VR training did show significantly positive influences on the participants’ internal glance activities that detect and assess system states, during edge case events. This finding is important since one of the modules of the VR training program was carefully curated to improve driver’s glance behavior when encountering edge case events of ACC. The results also establish the relationships between training and mental models although no significant correlations were found between the participants’ mental models and their hazard avoidance behaviors. However, this does fill a major gap in literature about our understanding about hazard avoidance in the context of vehicle automation and ADAS and could be extended for ADAS features other than ACC or even higher levels of automation. The VR training program can be built upon to include more ADAS features as well leading to better training practices in a rapidly developing world where vehicle automation has become a mainstay

IS THE MAGIC IN THE HANDS OR EYES? STUDYING THE EFFECTS OF DRIVER MONITORING STRATEGIES ON SITUATION AWARENESS, MIND WANDERING, AND CHANGE DETECTION BEHAVIOR IN LOW AND MEDIUM FIDELITY SEMI-AUTOMATED DRIVING ENVIRONMENTS

Advanced Driver Assist Systems (ADAS) are SAE level 2 features that require active driver control while engaged. Although drivers can have their feet off the pedals and hands off the steering wheel, they must maintain supervisory control of the vehicle. However, when these features are in use, drivers may become less aware of their surroundings, increasing the risk of accidents. To counter this problem, vehicle manufacturers use driver monitoring strategies to ensure drivers remain attentive while ADAS features are active. These monitoring strategies employ instructions to ensure drivers are engaged in the driving task. These instructions are broadly classified into hands-on-the-wheel and hands-free (eyes-on-the-road). The hands-on-the-wheel strategy measures driver engagement by examining steering wheel torque, while the hands-free strategy tracks the driver\u27s eyes to ensure they remain on the road. Although both strategies are commonly used in vehicles with SAE level 2 automation, there is a lack of publicly available data on their effectiveness and impact on takeover performance. In this dissertation, three studies were conducted to measure the effects of the hands-on-the-wheel and eyes-on-the-road driver monitoring strategies on situation awareness, change detection, mind-wandering, and gaze behavior. Study 1 was exploratory and utilized a low-fidelity semi-automated driving task to examine the effects of the two engagement strategies on driver attention during level 2 ADAS driving. Study 2 was an extension of Study 1 and moved to more naturalistic automation-related change detection in addition to a SAGAT freeze-probe protocol and comfort, fatigue, engagement, and takeover readiness measures in addition to the ones measured in Study 1. Study 3 extended Study 2 in a medium-fidelity driving simulator to investigate the effects of the two driver engagement strategies on driving performance variables and driver attention. Study 1 found that the hands-on-the-wheel strategy promoted less mind wandering during level 2 automated driving. Study 2 found that while the hands-on-the-wheel strategy also promoted less mind wandering, it promoted higher situation awareness, more perceived engagement with automated driving, less self-reported fatigue, and faster response to takeover requests. On the contrary, Study 3 found that the eyes-on-the-road strategy exhibited higher SA, faster responses to takeover requests, and less steering wheel variability but closer following distances post-takeover. Although the three studies have mixed findings, the hands-on-the-wheel strategy appears more promising because it engages drivers physically with the driving task, potentially leading to safer driving behaviors. This work has broader implications for SAE level 2 and 3 ADAS features, reinforcing the need for an engagement strategy with driver monitoring systems. Even as level 3 and higher technologies are developed, the results here inform strategies for automation-level step-downs as the drivers are brought back into actively controlling the vehicle

The driver response process in assisted and automated driving

Background: Safe assisted and automated driving can be achieved through a detailed understanding of the driver response process (the timing and quality of driver actions and visual behavior) triggered by an event such as a take-over request or a safety-relevant event. Importantly, most current evidence on driver response process in vehicle automation, and on automation effects (unsafe response process) is based on driving-simulator studies, whose results may not generalize to the real world. Objectives: To improve our understanding of the driver response process 1) in automated driving, which takes full responsibility for the driving task but assumes the driver is available to resume manual control upon request and 2) assisted driving, which supports the driver with longitudinal and lateral control but assumes the driver is responsible for safe driving at all times. Method: Data was collected in four experiments on a test track and public roads using the Wizard-of-Oz approach to simulate vehicle automation (assisted or automated). Results: The safety of the driver responses was found to depend on the type of vehicle automation. While a notable number of drivers crashed with a conflict object after experiencing highly reliable assisted driving, an automated driving function that issued a take-over request prior to the same event reduced the crash rate to zero. All participants who experienced automated driving were able to respond to the take-over requests and to potential safety-relevant events that occurred after automation deactivation. The responses to the take-over requests consisted of actions such as looking toward the instrument cluster, placing the hands on the steering wheel, deactivating automation, and moving the feet to the pedals. The order and timing of these actions varied among participants. Importantly, it was observed that the driver response process after receiving a take-over request included several off-path glances; in fact, drivers showed reduced visual attention to the forward road (compared to manual driving) for up to 15 s after the take-over request. Discussion: Overall, the work in this thesis could not confirm the presence of severe automation effects in terms of delayed response and a degraded intervention performance in safety-relevant events previously observed in driving simulators after automated driving. These differing findings likely stem from a combination of differences in the test environments and in the assumptions about the capabilities of the automated driving system. Conclusions: Assisted driving and automated driving should be designed separately: what is unsafe for assisted driving is not necessarily unsafe for automated driving and vice versa. While supervising drivers may crash in safety-relevant events without prior notification during highly reliable assisted driving, a clear and timely take-over request in automated driving ensures that drivers understand their responsibilities of acting in events when back in manual driving. In addition, when take-over requests are issued prior to the event onset, drivers generally perform similar manual driving and intervention performance as in a baseline. However, both before and just after the take-over requests, several drivers directed their gaze mainly off-road. Therefore, it is essential to consider the effect of take-over request designs not only on the time needed to deactivate automation, but also on drivers’ visual behavior. Overall, by reporting the results of tests of a future automated driving system (which is in line with future vehicle regulations and insurance company definitions) in realistic environments, this thesis provides novel findings that enhance the picture of automation effects that, before this thesis, seemed more severe