In this paper, I propose a definition of AI safety. I then explore the fundamental concepts that underlie this definition. My aim is to contribute to a constructive discussion and further the discourse on AI safety

Habli, Ibrahim

There is a growing urgency and global demand to address Artificial Intelligence (AI) safety. But swift and sustained progress is unlikely to emerge without a shared understanding of what it means for the use of AI to be safe. This paper advances a comprehensive definition of AI safety, followed by an exploration of the fundamental concepts that underpin this definition. The aim is to contribute to a meaningful and inclusive discussion and further the public discourse on AI safety

White Rose Research Online

This is a repository copy of On the Meaning of AI Safety.White Rose Research Online URL for this paper:https://eprints.whiterose.ac.uk/204545/Monograph:Habli, Ibrahim orcid.org/0000-0003-2736-8238 (2023) On the Meaning of AI Safety. Working Paper. eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. On the Meaning of AI SafetyIbrahim HabliDepartment of Computer ScienceUniversity of YorkYork, United Kingdomibrahim.habli@york.ac.ukAbstractIn this paper, I propose a definition of AI safety. I then explore the fundamental concepts that underlie thisdefinition. My aim is to contribute to a constructive discussion and further the discourse on AI safety.Keywords: Artificial Intelligence · Safety · Risk · HarmIntroductionAI systems now assist with everyday tasks, spanning from routine activities like driving to specialiseddecisions, such as clinical diagnosis. Yet, a fundamental concern arises: Is AI safe? The somewhat vaguebut commonly provided response is, 'it depends’1. Here, I propose a definition of AI safety. Next, I explorekey concepts on which the meaning of AI safety depends. My aim is to help inform the debate and advancethe safety argument about AI2.Defining AI SafetyMy proposed definition of AI safety is as follows:Absence of unacceptable risk of harm caused by the use of AIIn this definition, safety is characterised as a negative condition where the absence of risk is the focus.Alternatively, a more constructive and affirmative description, emphasising the existence of protectivecapabilities, can be articulated as follows:Protection from unacceptable risk of harm caused by the use of AIThese two definitions are interconnected. The protective capability in the latter definition is designed toresult in the absence of risk as described in the former definition.Explaining AI SafetyI will now address each key concept individually. I will also acknowledge any interrelated aspects of theseconcepts as necessary. For a visual summary of this discussion, please refer to Figure 1.2 Which could reveal that AI may be unsafe to deploy in particular contexts and why.1 Unfortunately, it is a common response by professionals to many complex questions!Artificial Intelligence (AI), according to the National Institute of Standards and Technology, is defined asthe "capability of a device to perform functions that are normally associated with human intelligence such asreasoning, learning, and self-improvement" [1]. The dominant AI technique driving most currentAI-enabled capabilities is Deep Learning (DL). In its simplest form, DL is a neural network with multiplelayers, trained on large datasets. However, two characteristics of AI, particularly DL, present significantchallenges to existing safety practices: the under-specificity of function3 and the opacity of the model.Under-specificity represents the delta between the underlying human intentions for the deployment of AIand the specific, tangible specifications employed in constructing the technology. Under-specificity hindersengineers in their efforts to establish and evaluate concrete safety requirements against which AIfunctions can be developed and tested. This challenge is compounded by the overwhelming focus in theliterature on overall AI performance. Opacity, i.e. the inability to understand how AI arrives at its outputs,complicates accountability, especially in terms of "explaining" and "dealing with the consequences" of AIfunctions and their development [3]. Under-specificity and opacity pose challenges to ensuring the safetyof both narrow4 and general-purpose5 AI models.Figure 1: An Illustrative depiction of the AI definition, with a machine learning failure as a source of a hazardous chain of complexchain of events leading to harm (though here graphically simplified), with protective human and technical lines of defence that areused for risk reduction.Use considers the algorithm or model in its intended physical and social context. Safety iscontext-sensitive. AI systems often exhibit brittleness and susceptibility to being ‘fooled’ or ‘confused’ byirrelevant environmental factors, such as stickers on stop signs [10]. The notion of context is multifacetedand varied, and includes how an AI component interacts with (1) other software components, e.g. cloudcomputing, (2) hardware components, e.g. image scanner devices, (3) the broader physical environment,e.g. communication between self-driving cars in vehicle platooning, (4) humans, e.g. how AI output is5 Also known as ‘ Frontier AI’: “It refers to highly capable general-purpose AI models, most often foundation models, that can perform a wide varietyof tasks and match or exceed the capabilities present in today’s most advanced models. It can then enable narrow use cases” [4].4 “Narrow artificial intelligence (narrow AI) is AI that is designed to perform a specific task. It is a specific type of artificial intelligence in which alearning algorithm is designed to perform a single task or narrow set of tasks, and any knowledge gained from performing the task will notautomatically be applicable or transferable” [4].3 Also see the discussion of ‘semantic gap’ by Burton et al. [2].presented and ‘explained’ and (5) the social context in which AI is developed and deployed, e.g. staffinglevels on clinical wards. The AI context is often described as "open" and "adaptive," as seen in scenarioslike urban driving or patient-facing services. This introduces challenges in terms of the scale, quality andquality of the safety evidence required and the assumptions underlying its ongoing validity in a changingenvironment. Describing AI and its context is a prerequisite for considering the subsequent concepts.Causation should be interpreted in its wider socio-technical sense, with the impact of AI on its physicaland social context represented using complex cause-and-effect chains. The impact can be both direct, asseen in end-to-end machine learning for autonomous driving, and indirect, as in clinical decision supportsystems where clinicians make the final decisions and take actions. Causation should also cover the AIsupply chains, upstream (what and how data is collected6), all the way downstream (how, and under whatconditions, end-users and other people interact with, and are influenced by, the use of AI). The opacity ofAI and the interactive complexity within its context complicate the modelling of causation. This in turnchallenges our ability to proactively mitigate risk and reactively hold people accountable for harms causedby AI.Harm in system safety is typically defined against physical damage. Traditionally, the focus is on damageto human physical health. This is followed by damage to property, with, more recently, the inclusion ofdamage to human psychological health and to the environment. A key perspective here is intent. Was harmintended, and if so, by whom? Was it justified? If harm is unintended, its occurrence is treated as a safetyaccident or incident. If harm is intended and it involves malice, it is flagged as a security event. Healthcareand defence present complex cases in this respect. Harm there may be intended, say for surgery, but maybe justified, given anticipated clinical benefit. In the AI literature, the discussion seems to favour an‘expansive’ scope of harm [5]. The particular focus appears to be on discrimination, bias, misinformation,privacy violation and threats to democratic institutions, amongst other moral, political, social and financialharms. These kinds of harm are significant and concerning. They should be proactively addressed andmitigated in an integrated manner (e.g. avoiding safety measures that unjustifiably constrain personalfreedom or entrench existing inequalities). However, we need to strike the right balance between AI safety,as an umbrella and loose term, encompassing all kinds of harm, and AI safety as a specialist term, with itslinks to the system safety field and its well-established methods and particular focus on physical andpsychological harm.Risk is the ‘idea of a possibility of danger’ [6]. More commonly, risk is the product of likelihood andseverity of harm. The term risk is central here because complete avoidance of harm is rarely feasible. Inrisk analysis, harm is considered in relation to a particular context. Further, harm is often framed inrelation to the use of a system and, more specifically, how system outputs may become sources of harm, i.e.hazards7. For narrow AI, hazard-based risk analysis is feasible though challenging. Underspecification inthe definition of the intended output, combined with model opacity, makes it hard to estimate thelikelihood of AI leading to harm via its potential hazardous output. For general-purpose AI, identifyingharm, and its severity and likelihood, may be infeasible since the technology is often presented ascontext-independent. Even when context is identified for a specific use case, deployers of ageneral-purpose AI often lack access to the AI model and its training and testing datasets to allow them toassess the likelihood of harm (due to potential hazardous outputs of the system).7 In other words, risk determination is typically framed by how harm could be caused “in a stipulated way by the hazard” [7].6 There is an implicit assumption in AI development that large datasets replace the need for detailed requirements. This is a root cause ofunder-specificity.Unacceptable risk to whom and given what else are two factors that need to be assessed as foundationalinputs into the AI risk decision-making process. Risk acceptability, and the lack of it, is a complex socialconstruct not a technical one. As such, risk decision-making needs to be participatory. Affectedstakeholders, or their trusted representatives, e.g. regulators, need to be meaningfully involved in how theuse of AI could present them with potential benefits and risks. The risks communicated should beexpansive, covering physical, psychological, moral and legal ones, amongst others, to allow the affectedstakeholders to understand any necessary tradeoffs. This will enable an open and reflective dialogue aboutthe distribution of benefits and risks from the use of an AI system and whether it is equitable acrossaffected stakeholders [8].Absence of unacceptable risk is rarely, if ever, an absolute goal. Rather, it is communicated with a degreeof confidence. Confidence is determined given the effectiveness of the protection, control or managementmeasures deployed, evidence available, sources of uncertainty declared and assumptions made. For AI,under-specificity and opacity are significant sources of epistemic uncertainty. It represents deficits in ourknowledge about the AI implementation and outputs and the impact AI may have on its environment. Insafety, confidence may be communicated using safety cases. Such arguments provide a means for justifyingand evaluating confidence in the safety of complex systems. An AI safety case should help facilitate thescrutiny of the otherwise implicit reasoning, the interrogation of sufficiency of the evidence, and thevalidity of the assumptions. This, in turn, helps foster increased transparency throughout the entire AIlifecycle [9][11].AcknowledgementsThis work was supported by the UKRI project (EP/W011239/1) “Assuring Responsibility for TrustworthyAutonomous Systems” and the Assuring Autonomy International Programme, a partnership betweenLloyd’s Register Foundation and the University of York.References1. National Institute of Standards and Technology (NIST), Definition of AI, accessed: 21 October 2023https://csrc.nist.gov/Topics/technologies/artificial-intelligence2. Burton, S., Habli, I., Lawton, T., McDermid, J., Morgan, P., & Porter, Z. (2020). Mind the gaps: Assuring the safety ofautonomous systems from an engineering, ethical, and legal perspective. Artificial Intelligence, 279, 103201.3. Porter, Z., Zimmermann, A., Morgan, P., McDermid, J., Lawton, T., & Habli, I. (2022). Distinguishing two features ofaccountability for AI technologies. Nature Machine Intelligence, 4(9), 734-736.4. Department for Science, Innovation & TechnologyAI Safety Summit, accessed: 21 October 2023,https://www.gov.uk/government/publications/ai-safety-summit-introduction/ai-safety-summit-introduction-html5. Ada Lovelace Institute, Regulating AI in the UK (2023), accessed: 21 October 2023,https://www.adalovelaceinstitute.org/report/regulating-ai-in-the-uk/6. R v Board of Trustees of the Science Museum [1993] 1 WLR 11717. Health and Safety Executive, Reducing risks, protecting people - R2P2, accessed 21 October 2023,https://www.hse.gov.uk/enforce/expert/r2p2.htm8. Porter, Z., Habli, I., McDermid, J., & Kaas, M. (2023). A principles-based ethics assurance argument pattern for AIand autonomous systems. AI and Ethics, 1-24.9. The Health Foundation, Using safety cases in industry and healthcare, December 2012.10. Heaven, D. (2019). Why deep-learning AIs are so easy to fool. Nature, 574(7777), 163-166.11. Hawkins, R., Paterson, C., Picardi, C., Jia, Y., Calinescu, R., & Habli, I. (2021). Guidance on the assurance of machinelearning in autonomous systems (AMLAS). arXiv preprint arXiv:2102.01564.

On the Meaning of AI Safety

https://eprints.whiterose.ac.uk/204545/1/On_the_Meaning_of_AI_Safety.pdf

On the Meaning of AI Safety

Abstract

Similar works

Full text

Available Versions

White Rose Research Online