Working together to face humanity’s greatest threats: Introduction to The Future of Research on Catastrophic and Existential Risk.

Ours is a resilient species. Around 70,000 years ago our total population may have fallen to between three and ten thousand individuals, possibly due to a supervolcanic eruption (Ambrose 1998) . Yet our ancestors survived, squeezed through the bottleneck, and flourished. But this resilience cannot be taken for granted. We are interconnected and interdependent as never before; the power and scale of our technological capacities are unprecedented. We are in uncharted waters and thus our previous survival is no longer a reason to expect our continued survival (Bostrom 2013). As a result, it is urgent that we develop a systematic understanding of the nature and causes of catastrophic and existential risks

Safeguarding the safeguards: How best to promote AI alignment in the public interest

AI alignment work is important from both a commercial and a safety lens. With this paper, we aim to help actors who support alignment efforts to make these efforts as effective as possible, and to avoid potential adverse effects. We begin by suggesting that institutions that are trying to act in the public interest (such as governments) should aim to support specifically alignment work that reduces accident or misuse risks. We then describe four problems which might cause alignment efforts to be counterproductive, increasing large-scale AI risks. We suggest mitigations for each problem. Finally, we make a broader recommendation that institutions trying to act in the public interest should think systematically about how to make their alignment efforts as effective, and as likely to be beneficial, as possible.Comment: Update Dec-15: Added a missing acknowledgement and fixed minor formatting error

AI Systems of Concern

Concerns around future dangers from advanced AI often centre on systems hypothesised to have intrinsic characteristics such as agent-like behaviour, strategic awareness, and long-range planning. We label this cluster of characteristics as "Property X". Most present AI systems are low in "Property X"; however, in the absence of deliberate steering, current research directions may rapidly lead to the emergence of highly capable AI systems that are also high in "Property X". We argue that "Property X" characteristics are intrinsically dangerous, and when combined with greater capabilities will result in AI systems for which safety and control is difficult to guarantee. Drawing on several scholars' alternative frameworks for possible AI research trajectories, we argue that most of the proposed benefits of advanced AI can be obtained by systems designed to minimise this property. We then propose indicators and governance interventions to identify and limit the development of systems with risky "Property X" characteristics.Comment: 9 pages, 1 figure, 2 table

Research community dynamics behind popular AI benchmarks

[EN] The widespread use of experimental benchmarks in AI research has created competition and collaboration dynamics that are still poorly understood. Here we provide an innovative methodology to explore these dynamics and analyse the way different entrants in these challenges, from academia to tech giants, behave and react depending on their own or others' achievements. We perform an analysis of 25 popular benchmarks in AI from Papers With Code, with around 2,000 result entries overall, connected with their underlying research papers. We identify links between researchers and institutions (that is, communities) beyond the standard co-authorship relations, and we explore a series of hypotheses about their behaviour as well as some aggregated results in terms of activity, performance jumps and efficiency. We characterize the dynamics of research communities at different levels of abstraction, including organization, affiliation, trajectories, results and activity. We find that hybrid, multi-institution and persevering communities are more likely to improve state-of-the-art performance, which becomes a watershed for many community members. Although the results cannot be extrapolated beyond our selection of popular machine learning benchmarks, the methodology can be extended to other areas of artificial intelligence or robotics, and combined with bibliometric studies.F.M.-P. acknowledges funding from the AI-Watch project by DG CONNECT and DG JRC of the European Commission. J.H.-O. and S.O.h. were funded by the Future of Life Institute, FLI, under grant RFP2-152. J.H.-O. was supported by the EU (FEDER) and Spanish MINECO under RTI2018-094403-B-C32, Generalitat Valenciana under PROMETEO/2019/098 and European Union's Horizon 2020 grant no. 952215 (TAILOR).Martínez-Plumed, F.; Barredo, P.; Ó Héigeartaigh, S.; Hernández-Orallo, J. (2021). Research community dynamics behind popular AI benchmarks. Nature Machine Intelligence. 3(7):581-589. https://doi.org/10.1038/s42256-021-00339-6S5815893

The Facets of Artificial Intelligence: A Framework to Track the Evolution of AI.

We present nine facets for the analysis of the past and future evolution of AI. Each facet has also a set of edges that can summarise different trends and contours in AI. With them, we first conduct a quantitative analysis using the information from two decades of AAAI/IJCAI conferences and around 50 years of documents from AI topics, an official database from the AAAI, illustrated by several plots. We then perform a qualitative analysis using the facets and edges, locating AI systems in the intelligence landscape and the discipline as a whole. This analytical framework provides a more structured and systematic way of looking at the shape and boundaries of AI.Leverhulme Centre for the Future of Intel- ligence, Leverhulme Trust, under Grant RC-2015-067

Reconfiguring Resilience for Existential Risk: Submission of Evidence to the Cabinet Office on the new UK National Resilience Strategy

This submission provides input on the UK Government's National Resilience Strategy Call for Evidence, which sought “public engagement to inform the development of a new Strategy that will outline an ambitious new vision for UK National Resilience and set objectives for achieving it.” In response, an interdisciplinary team of experts at the Centre for the Study of Existential Risk worked to prepare a concrete response to this call. In this document, we aim to share the contents of our submission for public deliberation. While we laud the UK government's inititiative to develop a new National Resilience Strategy, we argue that more work can and should be done to categorize and identify catastrophic, and existential risks; we emphasize the importance of taking a long-term perspective on mitigating and responding to the challenges these pose; and we encourage the development of a more comprehensive strategy, as these risks are all intertwined in an interconnected and complex environment. In our responses, we focus on the six broad thematic areas of the National Resilience Strategy (Risk and Resilience, Responsibilities and Accountability, Partnerships, Community, Investment, and Resilience in an Interconnected World), and provide key recommendations for improving UK national resilience, both from a general perspective on existential and global catastrophic risks, as well as with regards to policies in key risk domains such as in biorisk, climate risk, or emerging technologies within critical national infrastructure & - defence systems. While we laud the UK government's initial to develop a new National Resilience Strategy, we argue that more work can and should be done to categorize and identify catastrophic, complex, and existential risks; we emphasize a long-term perspective on mitigating and responding to the threats these pose; and we encourage the development of a more comprehensive strategy, as these risks are all intertwined in an interconnected and complex environment. In our responses, we focus on the six broad thematic areas of the National Resilience Strategy (Risk and Resilience, Responsibilities and Accountability, Partnerships, Community, Investment, and Resilience in an Interconnected World), and provide key recommendations for improving UK national resilience, both from a general perspective on existential and global catastrophic risks, as well as with regards to policies in key risk domains such as in biorisk, climate risk, or emerging technologies within critical national infrastructure & - defence systems

General intelligence disentangled via a generality metric for natural and artificial intelligence.

Success in all sorts of situations is the most classical interpretation of general intelligence. Under limited resources, however, the capability of an agent must necessarily be limited too, and generality needs to be understood as comprehensive performance up to a level of difficulty. The degree of generality then refers to the way an agent's capability is distributed as a function of task difficulty. This dissects the notion of general intelligence into two non-populational measures, generality and capability, which we apply to individuals and groups of humans, other animals and AI systems, on several cognitive and perceptual tests. Our results indicate that generality and capability can decouple at the individual level: very specialised agents can show high capability and vice versa. The metrics also decouple at the population level, and we rarely see diminishing returns in generality for those groups of high capability. We relate the individual measure of generality to traditional notions of general intelligence and cognitive efficiency in humans, collectives, non-human animals and machines. The choice of the difficulty function now plays a prominent role in this new conception of generality, which brings a quantitative tool for shedding light on long-standing questions about the evolution of general intelligence and the evaluation of progress in Artificial General Intelligence

International Governance of Civilian AI: A Jurisdictional Certification Approach

This report describes trade-offs in the design of international governance arrangements for civilian artificial intelligence (AI) and presents one approach in detail. This approach represents the extension of a standards, licensing, and liability regime to the global level. We propose that states establish an International AI Organization (IAIO) to certify state jurisdictions (not firms or AI projects) for compliance with international oversight standards. States can give force to these international standards by adopting regulations prohibiting the import of goods whose supply chains embody AI from non-IAIO-certified jurisdictions. This borrows attributes from models of existing international organizations, such as the International Civilian Aviation Organization (ICAO), the International Maritime Organization (IMO), and the Financial Action Task Force (FATF). States can also adopt multilateral controls on the export of AI product inputs, such as specialized hardware, to non-certified jurisdictions. Indeed, both the import and export standards could be required for certification. As international actors reach consensus on risks of and minimum standards for advanced AI, a jurisdictional certification regime could mitigate a broad range of potential harms, including threats to public safety

Predictable Artificial Intelligence

We introduce the fundamental ideas and challenges of Predictable AI, a nascent research area that explores the ways in which we can anticipate key indicators of present and future AI ecosystems. We argue that achieving predictability is crucial for fostering trust, liability, control, alignment and safety of AI ecosystems, and thus should be prioritised over performance. While distinctive from other areas of technical and non-technical AI research, the questions, hypotheses and challenges relevant to Predictable AI were yet to be clearly described. This paper aims to elucidate them, calls for identifying paths towards AI predictability and outlines the potential impact of this emergent field.Comment: 11 pages excluding references, 4 figures, and 2 tables. Paper Under Revie