Automating FDA Regulation

In the twentieth century, the Food and Drug Administration (“FDA”) rose to prominence as a respected scientific agency. By the middle of the century, it transformed the U.S. medical marketplace from an unregulated haven for dangerous products and false claims to a respected exemplar of public health. More recently, the FDA’s objectivity has increasingly been questioned. Critics argue the agency has become overly political and too accommodating to industry while lowering its standards for safety and efficacy. The FDA’s accelerated pathways for product testing and approval are partly to blame. They require lower-quality evidence, such as surrogate endpoints, and shift the FDA’s focus from premarket clinical trials toward postmarket surveillance, requiring less evidence up front while promising enhanced scrutiny on the back end. To further streamline product testing and approval, the FDA is adopting outputs from computer models, enhanced by artificial intelligence (“AI”), as surrogates for direct evidence of safety and efficacy. This Article analyzes how the FDA uses computer models and simulations to save resources, reduce costs, infer product safety and efficacy, and make regulatory decisions. To test medical products, the FDA assembles cohorts of virtual humans and conducts digital clinical trials. Using molecular modeling, it simulates how substances interact with cellular targets to predict adverse effects and determine how drugs should be regulated. Though legal scholars have commented on the role of AI as a medical product that is regulated by the FDA, they have largely overlooked the role of AI as a medical product regulator. Modeling and simulation could eventually reduce the exposure of volunteers to risks and help protect the public. However, these technologies lower safety and efficacy standards and may erode public trust in the FDA while undermining its transparency, accountability, objectivity, and legitimacy. Bias in computer models and simulations may prioritize efficiency and speed over other values such as maximizing safety, equity, and public health. By analyzing FDA guidance documents and industry and agency simulation standards, this Article offers recommendations for safer and more equitable automation of FDA regulation

Computer-Assisted versus Oral-and-Written History Taking for the Prevention and Management of Cardiovascular Disease: a Systematic Review of the Literature

Background and objectives: CVD is an important global healthcare issue; it is the leading cause of global mortality, with an increasing incidence identified in both developed and developing countries. It is also an extremely costly disease for healthcare systems unless managed effectively. In this review we aimed to: – Assess the effect of computer-assisted versus oral-and-written history taking on the quality of collected information for the prevention and management of CVD. – Assess the effect of computer-assisted versus oral-and-written history taking on the prevention and management of CVD. Methods: Randomised controlled trials that included participants of 16 years or older at the beginning of the study, who were at risk of CVD (prevention) or were either previously diagnosed with CVD (management). We searched all major databases. We assessed risk of bias using the Cochrane Collaboration tool. Results: We identified two studies. One comparing the two methods of history-taking for the prevention of cardiovascular disease n = 75. The study shows that generally the patients in the experimental group underwent more laboratory procedures, had more biomarker readings recorded and/or were given (or had reviewed), more dietary changes than the control group. The other study compares the two methods of history-taking for the management of cardiovascular disease (n = 479). The study showed that the computerized decision aid appears to increase the proportion of patients who responded to invitations to discuss CVD prevention with their doctor. The Computer-Assisted History Taking Systems (CAHTS) increased the proportion of patients who discussed CHD risk reduction with their doctor from 24% to 40% and increased the proportion who had a specific plan to reduce their risk from 24% to 37%. Discussion: With only one study meeting the inclusion criteria, for prevention of CVD and one study for management of CVD we did not gather sufficient evidence to address all of the objectives of the review. We were unable to report on most of the secondary patient outcomes in our protocol. Conclusions: We tentatively conclude that CAHTS can provide individually-tailored information about CVD prevention. However, further primary studies are needed to confirm these findings. We cannot draw any conclusions in relation to any other clinical outcomes at this stage. There is a need to develop an evidence base to support the effective development and use of CAHTS in this area of practice. In the absence of evidence on effectiveness, the implementation of computer-assisted history taking may only rely on the clinicians’ tacit knowledge, published monographs and viewpoint articles

Towards automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review.

Systematic reviews (SRs) constitute a critical foundation for evidence-based decision-making and policy formulation across various disciplines, particularly in healthcare and beyond. However, the inherently rigorous and structured nature of the SR process renders it laborious for human reviewers. Moreover, the exponential growth in daily published literature exacerbates the challenge, as SRs risk missing out on incorporating recent studies that could potentially influence research outcomes. This pressing need to streamline and enhance the efficiency of SRs has prompted significant interest in leveraging Artificial Intelligence (AI) techniques to automate various stages of the SR process. This review paper provides a comprehensive overview of the current AI methods employed for SR automation, a subject area that has not been exhaustively covered in previous literature. Through an extensive analysis of 52 related works and an original online survey, the primary AI techniques and their applications in automating key SR stages, such as search, screening, data extraction, and risk of bias assessment, are identified. The survey results offer practical insights into the current practices, experiences, opinions, and expectations of SR practitioners and researchers regarding future SR automation. Synthesis of the literature review and survey findings highlights gaps and challenges in the current landscape of SR automation using AI techniques. Based on these insights, potential future directions are discussed. This review aims to equip researchers and practitioners with a foundational understanding of the basic concepts, primary methodologies and recent advancements in AI-driven SR automation, while guiding computer scientists in exploring novel techniques to further invigorate and advance this field

Machine learning to assist risk of bias assessments in systematic reviews

Background: Risk-of-bias assessments are now a standard component of systematic reviews. At present, reviewers need to manually identify relevant parts of research articles for a set of methodological elements that affect the risk of bias, in order to make a risk-of-bias judgement for each of these elements. We investigate the use of text mining methods to automate risk-of-bias assessments in systematic reviews. We aim to identify relevant sentences within the text of included articles, to rank articles by risk of bias and to reduce the number of risk-of-bias assessments that the reviewers need to perform by hand. Methods: We use supervised machine learning to train two types of models, for each of the three risk-of-bias properties of sequence generation, allocation concealment and blinding. The first model predicts whether a sentence in a research article contains relevant information. The second model predicts a risk-of-bias value for each research article. We use logistic regression, where each independent variable is the frequency of a word in a sentence or article, respectively. Results: We found that sentences can be successfully ranked by relevance with area under the receiver operating characteristic (ROC) curve (AUC) > 0.98. Articles can be ranked by risk of bias with AUC > 0.72. We estimate that more than 33% of articles can be assessed by just one reviewer, where two reviewers are normally required. Conclusions: We show that text mining can be used to assist risk-of-bias assessments

Automated Paper Screening for Clinical Reviews Using Large Language Models

Objective: To assess the performance of the OpenAI GPT API in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review datasets and compare its performance against ground truth labelling by two independent human reviewers. Methods: We introduce a novel workflow using the OpenAI GPT API for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the GPT API with the screening criteria in natural language and a corpus of title and abstract datasets that have been filtered by a minimum of two human reviewers. We compared the performance of our model against human-reviewed papers across six review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. On a randomly selected subset of papers, the GPT API demonstrated the ability to provide reasoning for its decisions and corrected its initial decision upon being asked to explain its reasoning for a subset of incorrect classifications. Conclusion: The GPT API has the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, the GPT API can enhance efficiency and lead to more accurate and reliable conclusions in medical research.Comment: 15 pages, 2 figures, 4 table

Data extraction methods for systematic review (semi)automation: Update of a living systematic review [version 2; peer review: 3 approved]

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually

When and how to update systematic reviews: consensus and checklist.

Updating of systematic reviews is generally more efficient than starting all over again when new evidence emerges, but to date there has been no clear guidance on how to do this. This guidance helps authors of systematic reviews, commissioners, and editors decide when to update a systematic review, and then how to go about updating the review.This is the final version of the article. It first appeared from the BMJ Publishing Group via http://dx.doi.org/10.1136/bmj.i350