17 research outputs found
Accountability Infrastructure: How to implement limits on platform optimization to protect population health
Attention capitalism has generated design processes and product development
decisions that prioritize platform growth over all other considerations. To the
extent limits have been placed on these incentives, interventions have
primarily taken the form of content moderation. While moderation is important
for what we call "acute harms," societal-scale harms -- such as negative
effects on mental health and social trust -- require new forms of institutional
transparency and scientific investigation, which we group under the term
accountability infrastructure.
This is not a new problem. In fact, there are many conceptual lessons and
implementation approaches for accountability infrastructure within the history
of public health. After reviewing these insights, we reinterpret the societal
harms generated by technology platforms through reference to public health. To
that end, we present a novel mechanism design framework and practical
measurement methods for that framework. The proposed approach is iterative and
built into the product design process, and is applicable for both
internally-motivated (i.e. self regulation by companies) and
externally-motivated (i.e. government regulation) interventions for a range of
societal problems, including mental health.
We aim to help shape a research agenda of principles for the design of
mechanisms around problem areas on which there is broad consensus and a firm
base of support. We offer constructive examples and discussion of potential
implementation methods related to these topics, as well as several new data
illustrations for potential effects of exposure to online content.Comment: 63 pages, 5 tables and 6 figure
Optimization's Neglected Normative Commitments
Optimization is offered as an objective approach to resolving complex,
real-world decisions involving uncertainty and conflicting interests. It drives
business strategies as well as public policies and, increasingly, lies at the
heart of sophisticated machine learning systems. A paradigm used to approach
potentially high-stakes decisions, optimization relies on abstracting the real
world to a set of decision(s), objective(s) and constraint(s). Drawing from the
modeling process and a range of actual cases, this paper describes the
normative choices and assumptions that are necessarily part of using
optimization. It then identifies six emergent problems that may be neglected:
1) Misspecified values can yield optimizations that omit certain imperatives
altogether or incorporate them incorrectly as a constraint or as part of the
objective, 2) Problematic decision boundaries can lead to faulty modularity
assumptions and feedback loops, 3) Failing to account for multiple agents'
divergent goals and decisions can lead to policies that serve only certain
narrow interests, 4) Mislabeling and mismeasurement can introduce bias and
imprecision, 5) Faulty use of relaxation and approximation methods,
unaccompanied by formal characterizations and guarantees, can severely impede
applicability, and 6) Treating optimization as a justification for action,
without specifying the necessary contextual information, can lead to ethically
dubious or faulty decisions. Suggestions are given to further understand and
curb the harms that can arise when optimization is used wrongfully.Comment: 14 pages, 1 figure, presentation at FAccT2
The History and Risks of Reinforcement Learning and Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as a powerful
technique to make large language models (LLMs) easier to use and more
effective. A core piece of the RLHF process is the training and utilization of
a model of human preferences that acts as a reward function for optimization.
This approach, which operates at the intersection of many stakeholders and
academic disciplines, remains poorly understood. RLHF reward models are often
cited as being central to achieving performance, yet very few descriptors of
capabilities, evaluations, training methods, or open-source models exist. Given
this lack of information, further study and transparency is needed for learned
RLHF reward models. In this paper, we illustrate the complex history of
optimizing preferences, and articulate lines of inquiry to understand the
sociotechnical context of reward models. In particular, we highlight the
ontological differences between costs, rewards, and preferences at stake in
RLHF's foundations, related methodological tensions, and possible research
directions to improve general understanding of how reward models function.Comment: 14 pages, 3 figure
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) is a technique for training
AI systems to align with human goals. RLHF has emerged as the central method
used to finetune state-of-the-art large language models (LLMs). Despite this
popularity, there has been relatively little public work systematizing its
flaws. In this paper, we (1) survey open problems and fundamental limitations
of RLHF and related methods; (2) overview techniques to understand, improve,
and complement RLHF in practice; and (3) propose auditing and disclosure
standards to improve societal oversight of RLHF systems. Our work emphasizes
the limitations of RLHF and highlights the importance of a multi-faceted
approach to the development of safer AI systems
On Assessing Trustworthy AI in Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls
Artificial Intelligence (AI) has the potential to greatly improve the delivery of healthcare and other services that advance population health and wellbeing. However, the use of AI in healthcare also brings potential risks that may cause unintended harm. To guide future developments in AI, the High-Level Expert Group on AI set up by the European Commission (EC), recently published ethics guidelines for what it terms “trustworthy” AI. These guidelines are aimed at a variety of stakeholders, especially guiding practitioners toward more ethical and more robust applications of AI. In line with efforts of the EC, AI ethics scholarship focuses increasingly on converting abstract principles into actionable recommendations. However, the interpretation, relevance, and implementation of trustworthy AI depend on the domain and the context in which the AI system is used. The main contribution of this paper is to demonstrate how to use the general AI HLEG trustworthy AI guidelines in practice in the healthcare domain. To this end, we present a best practice of assessing the use of machine learning as a supportive tool to recognize cardiac arrest in emergency calls. The AI system under assessment is currently in use in the city of Copenhagen in Denmark. The assessment is accomplished by an independent team composed of philosophers, policy makers, social scientists, technical, legal, and medical experts. By leveraging an interdisciplinary team, we aim to expose the complex trade-offs and the necessity for such thorough human review when tackling socio-technical applications of AI in healthcare. For the assessment, we use a process to assess trustworthy AI, called 1Z-Inspection® to identify specific challenges and potential ethical trade-offs when we consider AI in practice.</jats:p
Co-Design of a Trustworthy AI System in Healthcare: Deep Learning Based Skin Lesion Classifier
This paper documents how an ethically aligned co-design methodology ensures trustworthiness in the early design phase of an artificial intelligence (AI) system component for healthcare. The system explains decisions made by deep learning networks analyzing images of skin lesions. The co-design of trustworthy AI developed here used a holistic approach rather than a static ethical checklist and required a multidisciplinary team of experts working with the AI designers and their managers. Ethical, legal, and technical issues potentially arising from the future use of the AI system were investigated. This paper is a first report on co-designing in the early design phase. Our results can also serve as guidance for other early-phase AI-similar tool developments.</jats:p
Fleets on the streets: How number, affiliation and purpose of shared-lane automated vehicle convoys influence public perception and blame
Automated vehicles (AVs) may have broad uses in society, but some applications may be more acceptable than others. Determining contexts in which AVs can acceptably operate is a substantial challenge for policy makers. In an online YouGov survey (N = 1175) with text-and-image vignettes of a one- or three-lane road section, AV convoys that shared lanes with a normal vehicle were perceived less positively and led to greater blame toward their owning institutions than lone AVs. AVs affiliated with a private commercial company were perceived less positively than those affiliated with a public transit agency. AVs used to regulate traffic of the vehicles behind them were blamed more than AVs used to navigate through traffic to reach a destination. These results suggest that numerical balance, vehicle affiliation and intended purpose are aspects of future AV policies for mixed traffic in shared lanes that will influence people's impressions of automated vehicles on public roads
Hard choices in artificial intelligence
As AI systems are integrated into high stakes social domains, researchers now examine how to design and operate them in a safe and ethical manner. However, the criteria for identifying and diagnosing safety risks in complex social contexts remain unclear and contested. In this paper, we examine the vagueness in debates about the safety and ethical behavior of AI systems. We show how this vagueness cannot be resolved through mathematical formalism alone, instead requiring deliberation about the politics of development as well as the context of deployment. Drawing from a new sociotechnical lexicon, we redefine vagueness in terms of distinct design challenges at key stages in AI system development. The resulting framework of Hard Choices in Artificial Intelligence (HCAI) empowers developers by 1) identifying points of overlap between design decisions and major sociotechnical challenges; 2) motivating the creation of stakeholder feedback channels so that safety issues can be exhaustively addressed. As such, HCAI contributes to a timely debate about the status of AI development in democratic societies, arguing that deliberation should be the goal of AI Safety, not just the procedure by which it is ensured.Information and Communication Technolog
On Assessing Trustworthy AI in Healthcare: Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls
Artificial Intelligence (AI) has the potential to greatly improve the delivery of healthcare and other services that advance population health and wellbeing. However, the use of AI in healthcare also brings potential risks that may cause unintended harm. To guide future developments in AI, the High-Level Expert Group on AI set up by the European Commission (EC), recently published ethics guidelines for what it terms “trustworthy” AI. These guidelines are aimed at a variety of stakeholders, especially guiding practitioners toward more ethical and more robust applications of AI. In line with efforts of the EC, AI ethics scholarship focuses increasingly on converting abstract principles into actionable recommendations. However, the interpretation, relevance, and implementation of trustworthy AI depend on the domain and the context in which the AI system is used. The main contribution of this paper is to demonstrate how to use the general AI HLEG trustworthy AI guidelines in practice in the healthcare domain. To this end, we present a best practice of assessing the use of machine learning as a supportive tool to recognize cardiac arrest in emergency calls. The AI system under assessment is currently in use in the city of Copenhagen in Denmark. The assessment is accomplished by an independent team composed of philosophers, policy makers, social scientists, technical, legal, and medical experts. By leveraging an interdisciplinary team, we aim to expose the complex trade-offs and the necessity for such thorough human review when tackling socio-technical applications of AI in healthcare. For the assessment, we use a process to assess trustworthy AI, called 1 Z-Inspection ® to identify specific challenges and potential ethical trade-offs when we consider AI in practice