Safety engineering, role responsibility and lessons from the Uber ATG Tempe Accident

Safety critical autonomous systems (SCAS) require a safety assurance case (SAC) to justify why they are considered acceptably safe to use, despite the residual risk associated with their operation. Reducing risk is an overarching principle of all safety critical systems development and operation. The SAC should demonstrate that the risk is tolerable and has been reduced as far as possible, through robust design and operational controls. As a SCAS may not have an operator, safety engineers have a more direct responsibility for operational decisions. Following an accident it may be useful to understand which engineering decisions causally contributed to it, and roles responsible for those decisions. This paper contains a review of how different senses of responsibility (role, moral, legal and causal) apply to SCAS engineering and operation. We use this to illustrate how considering role responsibility can help support a defensible SAC, and potentially improve system safety practice. Our findings are illustrated with an analysis the Uber/Tempe Arizona fatal collision accident report. We found that existing safety practice may not identify all role responsibilities in a way that supports causal safety analysis. This paper is intended for the whole TAS community, but with an emphasis on safety professionals

Unravelling Responsibility for AI

To reason about where responsibility does and should lie in complex situations involving AI-enabled systems, we first need a sufficiently clear and detailed cross-disciplinary vocabulary for talking about responsibility. Responsibility is a triadic relation involving an actor, an occurrence, and a way of being responsible. As part of a conscious effort towards 'unravelling' the concept of responsibility to support practical reasoning about responsibility for AI, this paper takes the three-part formulation, 'Actor A is responsible for Occurrence O' and identifies valid combinations of subcategories of A, is responsible for, and O. These valid combinations - which we term "responsibility strings" - are grouped into four senses of responsibility: role-responsibility; causal responsibility; legal liability-responsibility; and moral responsibility. They are illustrated with two running examples, one involving a healthcare AI-based system and another the fatal collision of an AV with a pedestrian in Tempe, Arizona in 2018. The output of the paper is 81 responsibility strings. The aim is that these strings provide the vocabulary for people across disciplines to be clear and specific about the different ways that different actors are responsible for different occurrences within a complex event for which responsibility is sought, allowing for precise and targeted interdisciplinary normative deliberations

What's my role? Modelling responsibility for AI-based safety-critical systems

AI-Based Safety-Critical Systems (AI-SCS) are being increasingly developed and deployed in the real world. These can pose a risk of harm to people and the environment, hence reducing that risk is an overarching priority during development and operation. Many contain Machine Learning (ML) components whose performance is hard to predict. As more AI systems become autonomous, a layer of risk management via human intervention has been removed. Following an accident it will be important to identify causal contributions and the different responsible actors behind those to learn from mistakes and prevent similar future events. Many authors have commented on the "responsibility gap" where it is difficult for developers and manufacturers to be held responsible for behaviour of an AI-SCS which contributes to harm. This is due to the complex development cycle for AI, the uncertainty in black-box AI components performance, and dynamic operating environment. Instead, a human operator of the AI-SCS can become a "liability sink" absorbing blame for the consequences of AI-SCS outputs they weren't responsible for creating, and may not have sufficient understanding of. This cross-disciplinary paper considers different types of responsibility (role, moral, legal and causal), and how they apply in the context of AI-SCS safety. We use a core conceptual formulation \textit{Actor(A) is responsible for Occurrence(O)} to create detailed role responsibility models, including related tasks and resources. Our aim is to present a practical method to precisely capture responsibility relationships, and provide clarity on the previously identified responsibility issues. We propose an analysis method to review the models for safety impacts. Our paper demonstrates the utility of the approach with two practical examples. The first is a retrospective analysis of the Tempe Arizona fatal collision involving an autonomous vehicle. The second is a safety focused predictive role-responsibility analysis for an AI-based diabetes co-morbidity prediction system. We show how our notation and analysis can illuminate responsibility issues during the AI development process and identify the impact of uncertainty surrounding how tasks were performed. For both examples, our focus was primarily on safety, with an aim of reducing unfair or disproportionate blame being placed on operators or developers of AI-SCS. We present a discussion and avenues for future research, including the development of a richer causal contribution model

Identifying Run-time Monitoring Requirements for Autonomous Systems through the Analysis of Safety Arguments

It is crucial that safety assurance continues to be managed for autonomous systems (AS) throughout their operation. This can be particularly challenging where AS operate in complex and dynamic environments. The importance of effective safety monitoring in ensuring the safety of AS through-life is already well documented. These current approaches often rely on utilising monitored information that happens to be available, or are reliant solely on engineering judgement to determine the requirements. Instead, we propose to use a systematic analysis of the safety case as the basis for determining the run-time monitoring requirements. Safety cases are created for AS prior to deployment in order to demonstrate why they are believed to be sufficiently safe to go into operation. The safety case is therefore inevitably based upon predictions and assumptions about the system and its operation which may become untrue due to changes post-deployment. Our approach identifies specific run-time monitoring requirements for AS based upon a dialectic analysis of the safety case developed for the system. The advantage of the approach described is that it is systematic (through explicit consideration of elements of the safety case for the AS) and provides a way to justify the sufficiency of the resulting monitoring requirements (through creating explicit links the safety claims made about the AS)

The Impact of Training Data Shortfalls on Safety of AI-based Clinical Decision Support Systems

Decision support systems with Artificial intelligence (AI) and specifically Machine Learning (ML) components present many challenges when assuring trust in operational performance, particularly in a safety-critical domain such as healthcare. During operation the Human in/on The Loop (HTL) may need assistance in determining when to trust the ML output and when to override it, particularly to prevent hazardous situations. In this paper, we consider how issues with training data shortfalls can cause varying safety performance in ML. We present a case study using an ML-based clinical decision support system for Type-2 diabetes related co-morbidity prediction (DCP). The DCP ML component is trained using real patient data, but the data was taken from a very large live database gathered over many years, and the records vary in distribution and completeness. Research developing similar clinical predictor systems describe different methods to compensate for training data shortfalls, but concentrate only on fixing the data to maximise the ML performance without considering a system safety perspective. This means the impact of the ML's varying performance is not fully understood at the system level. Further, methods such as data imputation can introduce a further risk of bias which is not addressed. This paper combines the use of ML data shortfall compensation measures with exploratory safety analysis to ensure all means of reducing risk are considered. We demonstrate that together these provide a richer picture allowing more effective identification and mitigation of risks from training data shortfalls

WCET analysis methods: Pitfalls and challenges on their trustworthiness

In the last three decades a number of methods have been devised to find upper-bounds for the execution time of critical tasks in time-critical systems. Most of such methods aim to compute Worst-Case Execution Time (WCET) estimates, which can be used as trustworthy upper-bounds for the execution time that the analysed programs will ever take during operation. The range of analysis approaches used include static, measurement-based and probabilistic methods, as well as hybrid combinations of them. Each of those approaches delivers its results on the assumption that certain hypotheses hold on the timing behaviour of the system as well that the user is able to provide the needed input information. Often enough the trustworthiness of those methods is only adjudged on the basis of the soundness of the method itself. However, trustworthiness rests a great deal also on the viability of the assumptions that the method makes on the system and on the user's ability, and on the extent to which those assumptions hold in practice. This paper discusses the hypotheses on which the major state-of-the-art timing analyses methods rely, identifying pitfalls and challenges that cause uncertainty and reduce confidence on the computed WCET estimates. While identifying weaknesses, this paper does not wish to discredit any method but rather to increase awareness on their limitations and enable an informed selection of the technique that best fits the user needs