Identifying how automation can lose its intended benefit along the development process : a research plan

Doctoral Consortium Presentation © The Authors 2009Automation is usually considered to improve performance in virtually any domain. However it can fail to deliver the target benefit as intended by those managers and designers advocating the introduction of the tool. In safety critical domains this problem is of significance not only because the unexpected effects of automation might prevent its widespread usage but also because they might turn out to be a contributor to incident and accidents. Research on failures of automation to deliver the intended benefit has focused mainly on human automation interaction. This paper presents a PhD research plan that aims at characterizing decisions for those involved in development process of automation for safety critical domains, taken under productive pressure, to identify where and when the initial intention the automation is supposed to deliver can be lost along the development process. We tentatively call such decisions as drift and the final objective is to develop principles that will allow to identify and compensate for possible sources of drift in the development of new automation. The research is based on case studies and is currently entering Year 2

The automation effect: Investigating factors that influence the driver response process in a safety-relevant event during assisted driving and after unsupervised automation

Introduction: Safe vehicle automation can be achieved through a detailed understanding of drivers’ ability to respond to a safety-relevant event after a period of automated driving. For instance, there is a need to understand in which scenarios automation effects are present (e.g. delayed response, degraded driving performance, crashing). Further, there is a need to identify specific factors (e.g. test environment, system-prompts, hands-on-wheel requirement, automation duration) that contribute to or prevent these automation effects. Objectives: The aim of this thesis is to investigate factors that influence: (a) automation effects in a non-prompted (i.e. absence of warning/notification) safety-relevant event during assisted driving and (b) automation aftereffects (i.e. automation effects specifically occurring after automation has been deactivated) in a prompted safety-relevant event during unsupervised automation. Method: Two Wizard-of-Oz test-track experiments were performed in order to investigate the driver response process in safety-relevant events. In experiment 1, the drivers were required to supervise (with or without a hands-on-wheel requirement) an assisted driving system, and then respond to a safety-relevant event that was not prompted by the system. In experiment 2, the drivers drove manually (baseline) and with an unsupervised automation system (a short and a long duration) before encountering a safety-relevant event. The automation system prompted (issued a take-over request) the driver to resume manual driving shortly before the safety-relevant event became visible. Results: In experiment 1, one third of the drivers responded late, or did not act at all, and crashed in the non-prompted safety-relevant event. In fact, the drivers crashed to the same extent and responded similarly independent of if they supervised the assisted driving system with or without hands on the wheel. In experiment 2, all drivers resumed manual control and did not collide in the safety-relevant event, both after a short and a long automation duration. All drivers showed a similar response and driving performance in the safety-relevant event for both long and short automation duration as well as in the manual baseline. Discussion: A hands-on-wheel requirement was not found to prevent late response or crashing in a non-prompted safety-relevant event encountered during assisted driving. More work is needed to understand the potential safety-benefits of a hands-on-wheel requirement in other types of conflicts and for driver distractions. The finding of minor automation aftereffects in experiment 2 contrasts to previous driving simulator studies. The reason may be the different test environments but is more likely due to different timings for prompting the drivers to resume manual control in relation to when the safety-relevant event became visible. Conclusions: Safe vehicle automation, including both assisted and unsupervised automation, can be achieved in a realistic environment (test track) for most drivers. However, assisted driving in combination with a non-prompted safety-relevant event, can be detrimental for safety, since some drivers may not understand the need to respond to avoid a crash. In fact, a hands-on-wheel requirement did not result in earlier steering responses nor did it prevent drivers from crashing. Thus, more work is needed to understand how to make sure drivers understand the need to respond in non-prompted safety-relevant events during assisted driving. In fact, it seems that when automation has matured to a level when it can prompt the drivers (i.e. unsupervised automation that can issue a take-over request) prior to a safety-relevant event becomes visible, drivers are able to safely resume manual control and perform similar as after an extended period of manual driving. Such safe driving performance seems to be independent of automation durations below 15 minutes

The driver response process in assisted and automated driving

Background: Safe assisted and automated driving can be achieved through a detailed understanding of the driver response process (the timing and quality of driver actions and visual behavior) triggered by an event such as a take-over request or a safety-relevant event. Importantly, most current evidence on driver response process in vehicle automation, and on automation effects (unsafe response process) is based on driving-simulator studies, whose results may not generalize to the real world. Objectives: To improve our understanding of the driver response process 1) in automated driving, which takes full responsibility for the driving task but assumes the driver is available to resume manual control upon request and 2) assisted driving, which supports the driver with longitudinal and lateral control but assumes the driver is responsible for safe driving at all times. Method: Data was collected in four experiments on a test track and public roads using the Wizard-of-Oz approach to simulate vehicle automation (assisted or automated). Results: The safety of the driver responses was found to depend on the type of vehicle automation. While a notable number of drivers crashed with a conflict object after experiencing highly reliable assisted driving, an automated driving function that issued a take-over request prior to the same event reduced the crash rate to zero. All participants who experienced automated driving were able to respond to the take-over requests and to potential safety-relevant events that occurred after automation deactivation. The responses to the take-over requests consisted of actions such as looking toward the instrument cluster, placing the hands on the steering wheel, deactivating automation, and moving the feet to the pedals. The order and timing of these actions varied among participants. Importantly, it was observed that the driver response process after receiving a take-over request included several off-path glances; in fact, drivers showed reduced visual attention to the forward road (compared to manual driving) for up to 15 s after the take-over request. Discussion: Overall, the work in this thesis could not confirm the presence of severe automation effects in terms of delayed response and a degraded intervention performance in safety-relevant events previously observed in driving simulators after automated driving. These differing findings likely stem from a combination of differences in the test environments and in the assumptions about the capabilities of the automated driving system. Conclusions: Assisted driving and automated driving should be designed separately: what is unsafe for assisted driving is not necessarily unsafe for automated driving and vice versa. While supervising drivers may crash in safety-relevant events without prior notification during highly reliable assisted driving, a clear and timely take-over request in automated driving ensures that drivers understand their responsibilities of acting in events when back in manual driving. In addition, when take-over requests are issued prior to the event onset, drivers generally perform similar manual driving and intervention performance as in a baseline. However, both before and just after the take-over requests, several drivers directed their gaze mainly off-road. Therefore, it is essential to consider the effect of take-over request designs not only on the time needed to deactivate automation, but also on drivers’ visual behavior. Overall, by reporting the results of tests of a future automated driving system (which is in line with future vehicle regulations and insurance company definitions) in realistic environments, this thesis provides novel findings that enhance the picture of automation effects that, before this thesis, seemed more severe

Identifying accident causes of driver-vehicle interactions using system theoretic process analysis (STPA)

Latest generations of automobiles are gradually being equipped with technologies that have increasing automation, a trend which had led to increase in the system complexity as well as increased human-automation interactions. Failures in such complex human-automation interactions increasingly occur due to the mismatch between what operators know about the system and what the designers expect operators to know. Causes of road accidents also change due to role shift of drivers from controlling the vehicle to monitoring the in-vehicle controllers. Failures in such complex systems involving human-automation interactions increasingly occur due to the emergent behaviours from the interactions, and are less likely due to reliability of individual components. Traditional safety analysis methods fall short in identifying such emergent failures. This paper focuses on using a systems thinking inspired safety analysis method called System Theoretic Process Analysis (STPA) to identify potential failures. The analysis focuses on a SAE Level-4 Vehicle that is in the development phase, and is controlled partially by a safety driver and its built-in Autonomous Driving System (ADS). The analysis yields that while increase in complexity does increase system functionality, it also brings a challenge to evaluate the safety of the system and potentially causes incorrect human-automation interactions, leading to an accident. After the possible inadequate driver-vehicle interactions are identified by STPA, corresponding requirements were then proposed in order to avoid the unsafe behaviour and thus preventing the hazards

Towards a safe MLOps Process for the Continuous Development and Safety Assurance of ML-based Systems in the Railway Domain

Traditional automation technologies alone are not sufficient to enable driverless operation of trains (called Grade of Automation (GoA) 4) on non-restricted infrastructure. The required perception tasks are nowadays realized using Machine Learning (ML) and thus need to be developed and deployed reliably and efficiently. One important aspect to achieve this is to use an MLOps process for tackling improved reproducibility, traceability, collaboration, and continuous adaptation of a driverless operation to changing conditions. MLOps mixes ML application development and operation (Ops) and enables high frequency software releases and continuous innovation based on the feedback from operations. In this paper, we outline a safe MLOps process for the continuous development and safety assurance of ML-based systems in the railway domain. It integrates system engineering, safety assurance, and the ML life-cycle in a comprehensive workflow. We present the individual stages of the process and their interactions. Moreover, we describe relevant challenges to automate the different stages of the safe MLOps process

Glows Co Automated Chemical Etching Machine for Fiber Optic Cable

This report summarizes the conceptualization and development of an automated machine designed for a fiber optic cable stripping process used by Lumentum LLC. This process is currently manually operated by Lumentum’s technicians and involves unavoidable handling of corrosive chemicals. To increase technician safety, the process will be automated to reduce chemical - operator interactions. Improving safety conditions for technicians is the primary motivation for automating this process. Automation will also decrease process variation and increase product quality. GLOWS CO was tasked with creating this automated solution, leading to the design of the Automatic Chemical Etching Machine (or A-CHEM) for the fiber etching process for Lumentum LLC. At the conclusion of this project, the A-CHEM successfully fulfilled all of the requirements set out by Lumentum, namely improving technician safety and making the process more ergonomic