Onnes, Annet

Pérez-Ortiz, Maria

Schlobach, Stefan

Tielman, Myrthe

Utrecht University Repository

Monitoring AI Systems: A ProblemAnalysis, Framework and OutlookAnnet ONNES a,1,2a Information and Computing Sciences, Utrecht University, The NetherlandsIn order to collaborate or interact with AI systems we have expectations about theirbehaviour. In this project we present a novel framework to study how to monitor AIsystems in such a way that the expectations for the performance are formulated intoan interpretable, knowledge-based system for monitoring. Monitoring is comparing thein- and output of the AI system, i.e. observed model behaviour, to intended behaviour.We have to consider how to represent this intended behaviour because modelling theintended behaviour using data would result in another model of a target system similarto the AI system’s model. We suggest a knowledge-based model constructed using theprescriptions is more effective.The problem setting consists of an AI system, which has to be monitored becausethere are expectations about its functioning. The AI system has to be monitored while itis running, since the expectations can be context specific. An impression of this is givenin Figure .Target systemAI systemHumansPrescriptionsMonitoring system FlagFigure 1. Impression of the settingThe target system is the process the AI system attempts to model [1]. The focus hereis on statistical models, trained using parts of the data from the target system that areobservable. Prescriptions are expectations about the output, that is the behaviour of theAI system, as defined by humans interacting with the system. This is information that isnot included in the AI system because it can be context specific. Output that is accurateaccording to the AI system and that also adheres to the prescription is referred to asintended behaviour. If this is not the case, the output is flagged. A monitoring systemwould be designed to use the prescriptions in order to monitor the AI systems behaviour.1Corresponding Author: Annet Onnes; E-mail: a.t.onnes@uu.nl.2This research was funded by the NWO Hybrid Intelligence Center.HHAI2022: Augmenting Human IntellectS. Schlobach et al. (Eds.)© 2022 The authors and IOS Press.This article is published online with Open Access by IOS Press and distributed under the termsof the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).doi:10.3233/FAIA220202238It is one thing to monitor an AI system to see whether expectations are fulfilled, butanother to make decisions about what the expectations are and thus whether the output is‘good’ or ‘bad’. The monitoring system is not defined as deciding what output is wrong;Prescriptions are determined externally, meaning monitoring itself is a technical opera-tion devoid of assigning value. In social situations norms are expectations of behaviour.A norm is characterised as a particular rule or normative principle that is accepted in aparticular group or community [2] or as “an expected behaviour in a social setting” [3].Norm monitoring is then described as “determining whether an individual is adhering tothis expected behaviour”. This fits the characterisation of monitoring. Norms are how-ever only suggested as a potential partial representation of intended behaviour becausethey are indicators of the expectations of humans. They do not dictate what is ’good’,define a strict form of representation or fully model intended behaviour.actor:human generate:engineer model:semanticdata generate:train model:statisticdata infer:deduce instance infer:deduce instance123 4Figure 2. The full monitoring setting with AI system and monitoring system. Marked are four individualsections which are elementary patterns. This framework represents all elements of the setting that are designedsystems with in- and outputs, this does not include the target system.Using the design patterns from Bekkum et al.[4], we can formally depict in Figure 1the systems and processes of the monitoring setting. Box 1 describes the process throughwhich prescriptions are modelled from human knowledge. The resulting semantic modelinforms the monitoring system. Box 2 describes training a statistical model from data.model:statistic is the model used by the AI system represented by pattern 3. Box 4is the monitoring process, a deductive process.Thusfar we only have the stepping stones from which to continue research in thismonitoring setting. This analysis helps to identify the questions: how to do the compar-ison between the intended behaviour and the observed AI system behaviour? and howto represent the intended behaviour using prescriptions? Like with most research thereare a vast amount of possible research directions and factors to take into consideration.Further work will have to look into uncertainty, comparing distributions and learning AIsystems. Using the abstract framework of the setting presented here these challenges canbe formulated in a more structured manner. The next step in this endeavour will also in-clude finding technical representations and notations. For a further exploration of futureresearch possibilities, as well as a more thorough analysis, see [5].A. Onnes / Monitoring AI Systems: A Problem Analysis, Framework and Outlook 239References[1] Frigg R, Hartmann S. Models in Science. In: Zalta EN, editor. The Stanford Encyclopedia of Philosophy.Spring 2020 ed. Metaphysics Research Lab, Stanford University; 2020. .[2] Brennan G, Eriksson L, Goodin RE, Southwood N. Explaining Norms. Oxford University Press; 2013.[3] Dastani M, Torroni P, Yorke-Smith N. Monitoring Norms: a Multi-Disciplinary Perspective. The Knowl-edge Engineering Review. 2018;33:1–22.[4] van Bekkum M, de Boer M, van Harmelen F, Meyer-Vitali A, Teije At. Modular design patternsfor hybrid learning and reasoning systems: a taxonomy, patterns and use cases. Applied Intelligence.2021;51(9):6528–6546.[5] Onnes A. Monitoring AI systems: A Problem Analysis, Framework and Outlook. arXiv; 2022. Availablefrom: https://arxiv.org/abs/2205.02562.A. Onnes / Monitoring AI Systems: A Problem Analysis, Framework and Outlook240

Monitoring AI systems: A Problem Analysis, Framework and Outlook

https://dspace.library.uu.nl/bitstream/1874/424497/1/FAIA_354_FAIA220202.pdf

Monitoring AI systems: A Problem Analysis, Framework and Outlook

Abstract

Similar works

Full text

Available Versions

Utrecht University Repository