Investigating Algorithm Review Boards for Organizational Responsible Artificial Intelligence Governance

Organizations including companies, nonprofits, governments, and academic institutions are increasingly developing, deploying, and utilizing artificial intelligence (AI) tools. Responsible AI (RAI) governance approaches at organizations have emerged as important mechanisms to address potential AI risks and harms. In this work, we interviewed 17 technical contributors across organization types (Academic, Government, Industry, Nonprofit) and sectors (Finance, Health, Tech, Other) about their experiences with internal RAI governance. Our findings illuminated the variety of organizational definitions of RAI and accompanying internal governance approaches. We summarized the first detailed findings on algorithm review boards (ARBs) and similar review committees in practice, including their membership, scope, and measures of success. We confirmed known robust model governance in finance sectors and revealed extensive algorithm and AI governance with ARB-like review boards in health sectors. Our findings contradict the idea that Institutional Review Boards alone are sufficient for algorithm governance and posit that ARBs are among the more impactful internal RAI governance approaches. Our results suggest that integration with existing internal regulatory approaches and leadership buy-in are among the most important attributes for success and that financial tensions are the greatest challenge to effective organizational RAI. We make a variety of suggestions for how organizational partners can learn from these findings when building their own internal RAI frameworks. We outline future directions for developing and measuring effectiveness of ARBs and other internal RAI governance approaches

Reproducibility

Science is allegedly in the midst of a reproducibility crisis, but questions of reproducibility and related principles date back nearly 80 years. Numerous controversies have arisen, especially since 2010, in a wide array of disciplines that stem from the failure to reproduce studies or their findings:biology, biomedical and preclinical research, business and organizational studies, computational sciences, drug discovery, economics, education, epidemiology and statistics, genetics, immunology, policy research, political science, psychology, and sociology. This monograph defines terms and constructs related to reproducible research, weighs key considerations and challenges in reproducing or replicating studies, and discusses transparency in publications that can support reproducible research goals. It attempts to clarify reproducible research, with its attendant (andconfusing or even conflicting) lexicon and aims to provide useful background, definitions, and practical guidance for all readers. Among its conclusions: First, researchers must become better educated about these issues, particularly the differences between the concepts and terms. The main benefit is being able to communicate clearly within their own fields and, more importantly, across multiple disciplines. In addition, scientists need to embrace these concepts as part of their responsibilities as good stewards of research funding and as providers of credible information for policy decision making across many areas of public concern. Finally, although focusing on transparency and documentation is essential, ultimately the goal is achieving the most rigorous, high-quality science possible given limitations on time, funding, or other resources.Publishe

Reproducibility: A primer on semantics and implications for research

Science is allegedly in the midst of a reproducibility crisis, but questions of reproducibility and related principles date back nearly 80 years. Numerous controversies have arisen, especially since 2010, in a wide array of disciplines that stem from the failure to reproduce studies or their findings:biology, biomedical and preclinical research, business and organizational studies, computational sciences, drug discovery, economics, education, epidemiology and statistics, genetics, immunology, policy research, political science, psychology, and sociology. This monograph defines terms and constructs related to reproducible research, weighs key considerations and challenges in reproducing or replicating studies, and discusses transparency in publications that can support reproducible research goals. It attempts to clarify reproducible research, with its attendant (and confusing or even conflicting) lexicon and aims to provide useful background, definitions, and practical guidance for all readers. Among its conclusions: First, researchers must become better educated about these issues, particularly the differences between the concepts and terms. The main benefit is being able to communicate clearly within their own fields and, more importantly, across multiple disciplines. In addition, scientists need to embrace these concepts as part of their responsibilities as good stewards of research funding and as providers of credible information for policy decision making across many areas of public concern. Finally, although focusing on transparency and documentation is essential, ultimately the goal is achieving the most rigorous, high-quality science possible given limitations on time, funding, or other resources

New science on the Open Science Grid

The Open Science Grid (OSG) includes work to enable new science, new scientists, and new modalities in support of computationally based research. There are frequently significant sociological and organizational changes required in transformation from the existing to the new. OSG leverages its deliverables to the large-scale physics experiment member communities to benefit new communities at all scales through activities in education, engagement, and the distributed facility. This paper gives both a brief general description and specific examples of new science enabled on the OSG. More information is available at the OSG web site: www.opensciencegrid.org

The Open Science Grid Status and Architecture The Open Science Grid Executive Board on behalf of the OSG Consortium: Ruth Pordes, Don Petravick: Fermi National Accelerator Laboratory

Abstract. The Open Science Grid (OSG) provides a distributed facility where the Consortium members provide guaranteed and opportunistic access to shared computing and storage resources. The OSG project[1] is funded by the National Science Foundation and the Department of Energy Scientific Discovery through Advanced Computing program. The OSG project provides specific activities for the operation and evolution of the common infrastructure. The US ATLAS and US CMS collaborations contribute to and depend on OSG as the US infrastructure contributing to the World Wide LHC Computing Grid on which the LHC experiments distribute and analyze their data. Other stakeholders include the STAR RHIC experiment, the Laser Interferometer Gravitational-Wave Observatory (LIGO), the Dark Energy Survey (DES) and several Fermilab Tevatron experiments-CDF, D0, MiniBoone etc. The OSG implementation architecture brings a pragmatic approach to enabling vertically integrated community specific distributed systems over a common horizontal set of shared resources and services. More information can be found at the OSG web site: www.opensciencegrid.org

Dear Colleague Letter for the Science, Engineering and Education for Sustainability (SEES) NSF-Wide Investment Area

The purpose of this DCL is to explain the scope of the SEES investment area, alert the community to activities that are being planned for the near term, and point to sources of additional information about future SEES plans

The Open Science Grid

The Open Science Grid (OSG) provides a distributed facility where the Consortium members provide guaranteed and opportunistic access to shared computing and storage resources. OSG provides support for and evolution of the infrastructure through activities that cover operations, security, software, troubleshooting, addition of new capabilities, and support for existing and engagement with new communities. The OSG SciDAC-2 project provides specific activities to manage and evolve the distributed infrastructure and support it's use. The innovative aspects of the project are the maintenance and performance of a collaborative (shared & common) petascale national facility over tens of autonomous computing sites, for many hundreds of users, transferring terabytes of data a day, executing tens of thousands of jobs a day, and providing robust and usable resources for scientific groups of all types and sizes. More information can be found at the OSG web site: www.opensciencegrid.org

HEP Science Network Requirements--Final Report

The Energy Sciences Network (ESnet) is the primary provider of network connectivity for the US Department of Energy Office of Science, the single largest supporter of basic research in the physical sciences in the United States. In support of the Office of Science programs, ESnet regularly updates and refreshes its understanding of the networking requirements of the instruments, facilities, scientists, and science programs that it serves. This focus has helped ESnet to be a highly successful enabler of scientific discovery for over 20 years. In August 2009 ESnet and the Office of High Energy Physics (HEP), of the DOE Office of Science, organized a workshop to characterize the networking requirements of the programs funded by HEP. The International HEP community has been a leader in data intensive science from the beginning. HEP data sets have historically been the largest of all scientific data sets, and the communty of interest the most distributed. The HEP community was also the first to embrace Grid technologies. The requirements identified at the workshop are summarized below, and described in more detail in the case studies and the Findings section: (1) There will be more LHC Tier-3 sites than orginally thought, and likely more Tier-2 to Tier-2 traffic than was envisioned. It it not yet known what the impact of this will be on ESnet, but we will need to keep an eye on this traffic. (2) The LHC Tier-1 sites (BNL and FNAL) predict the need for 40-50 Gbps of data movement capacity in 2-5 years, and 100-200 Gbps in 5-10 years for HEP program related traffic. Other key HEP sites include LHC Tier-2 and Tier-3 sites, many of which are located at universities. To support the LHC, ESnet must continue its collaborations with university and international networks. (3) While in all cases the deployed 'raw' network bandwidth must exceed the user requirements in order to meet the data transfer and reliability requirements, network engineering for trans-Atlantic connectivity is more complex than network engineering for intra-US connectivity. This is because transoceanic circuits have lower reliability and longer repair times when compared with land-based circuits. Therefore, trans-Atlantic connectivity requires greater deployed bandwidth and diversity to ensure reliability and service continuity of the user-level required data transfer rates. (4) Trans-Atlantic traffic load and patterns must be monitored, and projections adjusted if necessary. There is currently a shutdown planned for the LHC in 2012 that may affect projections of trans-Atlantic bandwidth requirements. (5) There is a significant need for network tuning and troubleshooting during the establishment of new LHC Tier-2 and Tier-3 facilities. ESnet will work with the HEP community to help new sites effectively use the network. (6) SLAC is building the CCD camera for the LSST. This project will require significant bandwidth (up to 30Gbps) to NCSA over the next few years. (7) The accelerator modeling program at SLAC could require the movement of 1PB simulation data sets from the Leadership Computing Facilities at Argonne and Oak Ridge to SLAC. The data sets would need to be moved overnight, and moving 1PB in eight hours requires more than 300Gbps of throughput. This requirement is dependent on the deployment of analysis capabilities at SLAC, and is about five years away. (8) It is difficult to achieve high data transfer throughput to sites in China. Projects that need to transfer data in or out of China are encouraged to deploy test and measurement infrastructure (e.g. perfSONAR) and allow time for performance tuning