Privacy and Accountability in Black-Box Medicine

Black-box medicine—the use of big data and sophisticated machine learning techniques for health-care applications—could be the future of personalized medicine. Black-box medicine promises to make it easier to diagnose rare diseases and conditions, identify the most promising treatments, and allocate scarce resources among different patients. But to succeed, it must overcome two separate, but related, problems: patient privacy and algorithmic accountability. Privacy is a problem because researchers need access to huge amounts of patient health information to generate useful medical predictions. And accountability is a problem because black-box algorithms must be verified by outsiders to ensure they are accurate and unbiased, but this means giving outsiders access to this health information. This article examines the tension between the twin goals of privacy and accountability and develops a framework for balancing that tension. It proposes three pillars for an effective system of privacy-preserving accountability: substantive limitations on the collection, use, and disclosure of patient information; independent gatekeepers regulating information sharing between those developing and verifying black-box algorithms; and information-security requirements to prevent unintentional disclosures of patient information. The article examines and draws on a similar debate in the field of clinical trials, where disclosing information from past trials can lead to new treatments but also threatens patient privacy

Manufacturing Barriers to Biologics Competition and Innovation

As finding breakthrough small-molecule drugs gets harder, drug companies are increasingly turning to “large molecule” biologics. Although biologics represent many of the most promising new therapies for previously intractable diseases, they are extremely expensive. Moreover, the pathway for generic-type competition set up by Congress in 2010 is unlikely to yield significant cost savings. In this Article, we provide a fresh diagnosis of, and prescription for, this major public policy problem. We argue that the key cause is pervasive trade secrecy in the complex area of biologics manufacturing. Under the current regime, this trade secrecy, combined with certain features of FDA regulation, not only creates high barriers to entry of indefinite duration but also undermines efforts to advance fundamental knowledge. In sharp contrast, offering incentives for information disclosure to originator manufacturers would leverage the existing interaction of trade secrecy and the regulatory state in a positive direction. Although trade secrecy, particularly in complex areas like biologics manufacturing, often involves tacit knowledge that is difficult to codify and thus transfer, in this case regulatory requirements that originator manufacturers submit manufacturing details have already codified the relevant tacit knowledge. Incentivizing disclosure of these regulatory submissions would not only spur competition but it would provide a rich source of information upon which additional research, including fundamental research into the science of manufacturing, could build. In addition to provide fresh diagnosis and prescription in the specific area of biologics, the Article contributes to more general scholarship on trade secrecy and tacit knowledge. Prior scholarship has neglected the extent to which regulation can turn tacit knowledge not only into codified knowledge but into precisely the type of codified knowledge that is most likely to be useful and accurate. The Article also draws a link to the literature on adaptive regulation, arguing that greater regulatory flexibility is necessary and that more fundamental knowledge should spur flexibility. A vastly shortened version of the central argument that manufacturing trade secrecy hampers biosimilar development was published at 348 Science 188 (2015), available online

Nudging the FDA

[Excerpt] The FDA’s regulation of drugs is frequently the subject of policy debate, with arguments falling into two camps. On the one hand, a libertarian view of patients and the health care system holds high the value of consumer choice. Patients should get all the information and the drugs they want; the FDA should do what it can to enforce some basic standards but should otherwise get out of the way. On the other hand, a paternalist view values the FDA’s role as an expert agency standing between patients and a set of potentially dangerous drugs and potentially unscrupulous or at least insufficiently careful drug companies. We lay out here some of the ways the FDA regulates drugs, including some normally left out of the debate, and suggest a middle ground between libertarian and paternalistic approaches focused on correcting information asymmetry and aligning incentives.

Am I My Son? Human Clones and the Modern Family

As increasingly complex assisted reproductive technologies (ART) become available, legal and social conceptions of family become ambiguous and sometimes misaligned. The as-yet unrealized technology of cloning provides the clearest example of this confusion: is the legal parent of a clone the individual cloned, or are that individual‘s parents also the parents of the clone? These issues have been generally obscured by the debates around the deployment of ART, especially cloning; far less consideration has been given to the way these new technologies impact the way we think about and develop law on the relationships between genetic, social, gestational, and legal parenthood. This article considers these issues in depth, looking at competing common-law frameworks for determining parentage, the statutory framework of parentage, and deeper theoretical concerns underlying the area. The article concludes that an intent-based framework, with at least some external limitations, most accurately matches law to social views of parents using new forms of ART

Describing Black-Box Medicine

Personalized medicine is a touchstone of modern medical science, and is increasingly addressed in the legal literature. In personalized medicine, treatments are chosen and tailored based on the characteristics of the individual patient. However, personalized medicine today is largely limited to those relatively simple relationships that can be explicitly characterized and validated through the scientific process and through clinical trial

Risks and Remedies for Artificial Intelligence in Healthcare

Artificial intelligence (AI) is rapidly entering health care and serving major roles, from automating drudgery and routine tasks in medical practice to managing patients and medical resources. As developers create AI systems to take on these tasks, several risks and challenges emerge, including the risk of injuries to patients from AI system errors, the risk to patient privacy of data acquisition and AI inference, and more. Potential solutions are complex but involve investment in infrastructure for high-quality, representative data; collaborative oversight by both the Food and Drug Administration and other health-care actors; and changes to medical education that will prepare providers for shifting roles in an evolving system

Biobanks as Innovation Infrastructure for Translational Medicine

Biobanks represent an opportunity for the use of big data to drive translational medicine. Precision medicine demands data to shape treatments to individual patient characteristics; large datasets can also suggest new uses for old drugs or relationships between previously unlinked conditions. But these tasks can be stymied when data are siloed in different datasets, smaller biobanks, or completely proprietary private resources. This hampers not only analysis of the data themselves, but also efforts to translate data-based insights into actionable recommendations and to transfer the discovered technology into a commercialization pipeline. Cross-project technological innovation, development, and validation are all more difficult when data are divided between different biobanks and other data repositories. One way to conceive of biobanks and the big medical datasets they create and embody uses the lens of infrastructure: how can biobanks and their data serve as infrastructure to support later innovation? Some efforts already fit into this model; for example, the United States’ Precision Medicine Cohort—now renamed All of Us—aims to create a large, uniform dataset to be used for widespread future research. Other biobank-related data efforts, like Myriad’s dataset on BRCA1/2 genetic variations, still function as entirely private resources. Treating medical big data as infrastructure has implications for how they should be governed, and suggests advantages to centralized control and relatively broad access. More broadly, viewing biobank-related data as infrastructure would place them at a distinctly earlier point in the commercialization pipeline, serving more to facilitate later steps in translational medicine rather than being viewed as potentially commercializable products themselves. This chapter is divided into two parts. In the first, I briefly describe big data in medicine: the sources of medical data, the promises of medical big data, and a key challenge: data fragmentation. In the second, I discuss the role of biobanks in medical big data, focusing on their role in infrastructure for innovation and their potential for facilitating translational research

Artificial Intelligence in Health Care: Applications and Legal Implications

Artificial intelligence (AI) is rapidly moving to change the healthcare system. Driven by the juxtaposition of big data and powerful machine learning techniques—terms I will explain momentarily—innovators have begun to develop tools to improve the process of clinical care, to advance medical research, and to improve efficiency. These tools rely on algorithms, programs created from healthcare data that can make predictions or recommendations. However, the algorithms themselves are often too complex for their reasoning to be understood or even stated explicitly. Such algorithms may be best described as “black-box.” This article briefly describes the concept of AI in medicine, including several possible applications, then considers its legal implications in four areas of law: regulation, tort, intellectual property, and privacy

Part I - AI and Data as Medical Devices

It may seem counterintuitive to open a book on medical devices with chapters on software and data, but these are the frontiers of new medical device regulation and law. Physical devices are still crucial to medicine, but they – and medical practice as a whole – are embedded in and permeated by networks of software and caches of data. Those software systems are often mindbogglingly complex and largely inscrutable, involving artificial intelligence and machine learning. Ensuring that such software works effectively and safely remains a substantial challenge for regulators and policymakers. Each of the three chapters in this part examines different aspects of how best to meet this challenge, focusing on review by drug regulators and, crucially, what aspects of oversight fall outside that purview. Collectively, these chapters demonstrate the challenge of regulating and overseeing the AI- and data-powered software which increasingly shapes medical practice, both behind the scenes and within the examining room. These technologies bring immense potential along with real risk, but present new regulatory challenges due to their opacity, their plasticity, and the speed with which they are being incorporated into the health system. Ensuring the right sort of oversight so that medical devices centered on AI and big data are safe, effective, and deployed in such a way as to actually help the health system demands concerted action from stakeholders across the board

Medical AI and Contextual Bias

Artificial intelligence will transform medicine. One particularly attractive possibility is the democratization of medical expertise. If black-box medical algorithms can be trained to match the performance of high-level human experts — to identify malignancies as well as trained radiologists, to diagnose diabetic retinopathy as well as board-certified ophthalmologists, or to recommend tumor-specific courses of treatment as well as top-ranked oncologists — then those algorithms could be deployed in medical settings where human experts are not available, and patients could benefit. But there is a problem with this vision. Privacy law, malpractice, insurance reimbursement, and FDA approval standards all encourage developers to train medical AI in high-resource contexts, such as academic medical centers. And put simply, care is different in high-resource settings than it is in low-resource settings such as community health centers or rural providers in less-developed countries. Patient populations differ, as do the resources available to administer treatment and the resources available to pay for that treatment. This development pattern will lead to decreases in the quality of the algorithm’s recommendations, reflected in problematic care and increased costs. Perniciously, such quality problems in low-resource contexts are likely to go unrecognized for exactly the same reasons that promote algorithmic training in high-resource contexts. Solutions are not trivial. Labeling products the same way that drugs are labeled is unlikely to work, and truly addressing the problem may require a combination of public investment in data to train medical AI and regulatory requirements for cross-context validation. Nevertheless, if black-box medicine is to achieve its goal of bringing excellent medicine to broad sets of patients, the problem of contextual bias should be recognized and addressed sooner rather than later