How to use FDA drug approval documents for evidence syntheses

There is compelling evidence that published trial information is selectively reported and that results not showing favourable effects of the tested treatments often remain unpublished. Clinical trial information published by regulatory authorities such as the US Food and Drug Administration (FDA) may help to reduce such reporting biases. FDA approval documents are long and do not follow the typical structure of medical journal articles, which may discourage reviewers from using them for evidence syntheses. Our practical guidance on how to efficiently identify and use approval documents to find the relevant information may help promoting the use of this valuable data source for evidence syntheses

Epidemiology and characteristics of clinical trials supporting US FDA approval of novel cancer drugs

The US Food and Drug Administration (FDA) approves novel drugs that appear to be effective for their intended uses and whose benefits outweigh their risks. The legal standards for drug approval are widely understood to require evidence of efficacy from two or more clinical trials that independently demonstrate statistically significant effects in favor of the experimental drug and on endpoints that reflect a clinical benefit to patients. However, the FDA has the authority to be flexible in applying the approval standards, particularly in the case of drugs that are intended to treat serious medical conditions. This regulatory flexibility, however, is frequently and repeatedly criticized for putting patients at risk. The two overall aims of this thesis were a) to develop a guide on how to access, retrieve, and use of clinical trial data that supported FDA approval of novel treatments published by the agency itself; and b) to describe characteristics and extent of clinical trial evidence that supported FDA approval of novel drugs for cancer indications between 2000 and 2016. These aims were addressed in three manuscripts, and we describe the methods used to retrieve and manage the clinical trial information in a fourth manuscript. Manuscript 1 “How to use FDA drug approval documents for evidence syntheses”: The FDA publishes information about the clinical trial evidence that supported approval of novel drugs and therapeutic biologics in the drugs@FDA database in form of “drug approval packages”. Information in the main document extends over hundreds of pages and is structured in a way that may be unintuitive for researchers. Although the value of this source of potentially unpublished clinical trial information is undisputed, its use in evidence syntheses of drug interventions remains limited. Based on our experience in using the drugs@FDA database and drug approval packages, we provide step-by-step instructions on how to navigate through the database as well as how to access, efficiently find and retrieve, and use the clinical trial information. Our guide may promote better use of this information, which may improve the completeness and validity of future evidence syntheses of drug interventions. Manuscript 2 “The Comparative Effectiveness of Innovative Treatments for Cancer (CEIT-Cancer) project: rationale and design of the database and the collection of evidence available at approval of novel drugs”: We describe the rationale and efforts made to identify all clinical trials that supported FDA drug approval between 2000 and 2016 for the treatment of cancer and to retrieve pertinent information about trial design and treatment effects on overall survival, progression-free survival, and objective tumor response. Most data retrieval steps were conducted by two data reviewers (who worked independently and who were guided by an instruction manual) to reduce random errors that would affect the quality of the collected data. The study design can be applied in the future for projects with similar scopes, and the collected data will be used in the future for numerous research projects. Manuscript 3 “Clinical trial evidence supporting US FDA approval of novel cancer therapies between 2000 and 2016”: Using the data collected in the CEIT-Cancer project, we analyzed the 127 clinical trials that supported FDA approval of 92 novel drugs for the treatment of 100 cancer indications. The median number of enrolled patients was 193 (interquartile range [IQR]: 106, 448). Fifty-one percent (51%) were randomized controlled, and 75% were open-label. The hazard ratio (HR) for the pooled average treatment effect on overall survival was 0.77 (95% confidence interval [CI]: 0.73, 0.81; I-squared [I2] = 47%), and HR = 0.52 (95% CI: 0.47, 0.57; I2 = 88%) for progression-free survival. The odds ratio for objective tumor response was 3.58 (95% CI: 2.77, 4.62; I2 = 87%). The median absolute survival gain was 2.40 months (IQR: 1.25, 3.89). These findings indicate that novel cancer treatments are supported by trials with design features that have the potential to threaten the validity of the findings and that the overall absolute survival difference is small. Manuscript 4 “Corroborating characteristics of single pivotal trial evidence supporting FDA approval of novel cancer therapies”: For experimental new drugs intended to treat serious conditions, the FDA may grant marketing approvals based on evidence from a single trial alone (instead of two or more) if certain trial characteristics are met. The presence of one or more of these trial characteristics defined by the FDA may increase the FDA’s confidence in the validity of a single clinical trial and may therefore provide corroborating evidence of efficacy. Our results show that 36 out of 100 approvals of novel cancer treatments were based on evidence from a single trial alone. Sixty-four percent (64%) were large and multicentric trials; 64% had consistent effects across different subgroups; 42% had consistent effects across endpoints; and 33% had very low p-values for the primary endpoint. Overall, 92% of clinical trials that supported FDA approval alone fulfilled at least one of the corroborating characteristics. Whether the presence of one or more of these corroborating characteristics indeed provides a safeguard against threats to the validity of trials remains to be answered. The background information provided in the manuscripts and this thesis improve the understanding of the regulatory considerations that are made to bring novel cancer treatments to market. The results of the analyses provide insight into the characteristics of the clinical trial evidence and the number of clinical trials that supported drug approval in the fields of oncology and malignant hematology

Nonrandomized studies using causal-modeling may give different answers than RCTs: a meta-epidemiological study

To evaluate how estimated treatment effects agree between nonrandomized studies using causal modeling with marginal structural models (MSM-studies) and randomized trials (RCTs).; Meta-epidemiological study.; MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals.; The main analysis included 19 MSM-studies (1,039,570 patients) and 141 RCTs (120,669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for eight clinical questions (42%), and their 95% CI (confidence interval) did not include the RCT estimate in nine clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR [odds ratio] 1.58; IQR [interquartile range] 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios [sROR] 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09).; Nonrandomized studies using causal modeling with MSM may give different answers than RCTs. Caution is still required when nonrandomized "real world" evidence is used for healthcare decisions

Marginal structural models and other analyses allow multiple estimates of treatment effects in randomized clinical trials: Meta-epidemiological analysis

To determine how marginal structural models (MSMs), which are increasingly used to estimate causal effects, are used in randomized clinical trials (RCTs) and compare their results with those from intention-to-treat (ITT) or other analyses.; We searched PubMed, Scopus, citations of key references, and Clinicaltrials.gov. Eligible RCTs reported clinical effects based on MSMs and at least one other analysis.; We included 12 RCTs reporting 138 analyses for 24 clinical questions. In 19/24 (79%), MSM-based and other effect estimates were all in the same direction, 22/22 had overlapping 95% confidence intervals (CIs), and in 19/22 (86%), the MSM effect estimate lay within all 95% CIs of all other effects (in two cases no CIs were reported). For the same clinical question, the largest effect estimate from any analysis was 1.19-fold (median; interquartile range 1.13-1.34) larger than the smallest. All MSM and ITT effect estimates were in the same direction and had overlapping 95% CIs. In 71% (12/17), they also agreed on the presence of statistical significance. MSM-based effect estimates deviated more from the null than those based on ITT (P = 0.18). The effect estimates of both approaches differed 1.12-fold (median; interquartile range 1.02-1.22).; MSMs provided largely similar effect estimates as other available analyses. Nevertheless, some of the differences in effect estimates or statistical significance may become important in clinical decision-making, and the multiple estimates require utmost attention of possible selective reporting bias

Off-label treatments were not consistently better or worse than approved drug treatments in randomized trials

Off-label drug use is highly prevalent but controversial and often discouraged assuming generally inferior medical effects associated with off-label use.; We searched PubMed, MEDLINE, PubMed Health, and the Cochrane Library up to May 2015 for systematic reviews including meta-analyses of randomized clinical trials (RCTs) comparing off-label and approved drugs head-to-head in any population and on any medical outcome. We combined the comparative effects in meta-analyses providing summary odds ratios (sOR) for each treatment comparison and outcome, and then calculated an overall summary of the sOR across all comparisons (ssOR).; We included 25 treatment comparisons with 153 RCTs and 24,592 patients. In six of 25 comparisons (24%), off-label drugs were significantly superior (five of 25) or inferior (one of 25) to approved treatments. There was substantial statistical heterogeneity across comparisons (I2 = 43%). Overall, off-label drugs were more favorable than approved treatments (ssOR 0.72; 95% CI = 0.54-0.95). Analyses of patient-relevant outcomes were similar (statistical significant differences in 24% (six of 25); ssOR 0.74; 95% CI = 0.56-0.98; I2 = 60%). Analyses of primary outcomes of the systematic reviews (n = 22 comparisons) indicated less heterogeneity and no statistically significant difference overall (ssOR 0.85; 95% CI = 0.67-1.06; I2 = 0%).; Approval status does not reliably indicate which drugs are more favorable in situations with clinical trial evidence comparing off-label with approved use. Drug effectiveness assessments without considering off-label use may provide incomplete information. To ensure that patients receive the best available care, funding, policy, reimbursement, and treatment decisions should be evidence based considering the entire spectrum of available therapeutic choices

Interpretation of epidemiologic studies very often lacked adequate consideration of confounding

Confounding bias is a most pervasive threat to validity of observational epidemiologic research. We assessed whether authors of observational epidemiologic studies consider confounding bias when interpreting the findings.; We randomly selected 120 cohort or case-control studies published in 2011 and 2012 by the general medical, epidemiologic, and specialty journals with the highest impact factors. We used Web of Science to assess citation metrics through January 2017.; Sixty-eight studies (56.7%, 95% confidence interval: 47.8-65.5%) mentioned "confounding" in the Abstract or Discussion sections, another 20 (16.7%; 10.0-23.3%) alluded to it, and there was no mention or allusion at all in 32 studies (26.7%; 18.8-34.6%). Authors often acknowledged that for specific confounders, there was no adjustment (34 studies; 28.3%) or deem it possible or likely that confounding affected their main findings (29 studies; 24.2%). However, only two studies (1.7%; 0-4.0%) specifically used the words "caution" or "cautious" for the interpretation because of confounding-related reasons and eventually only four studies (3.3%; 0.1-6.5%) had limitations related to confounding or any other bias in their Conclusions. Studies mentioning that the findings were possibly or likely affected by confounding were more frequently cited than studies with a statement that findings were unlikely affected (median 6.3 vs. 4.0 citations per year, P = 0.04).; Many observational studies lack satisfactory discussion of confounding bias. Even when confounding bias is mentioned, authors are typically confident that it is rather irrelevant to their findings and they rarely call for cautious interpretation. More careful acknowledgment of possible impact of confounding is not associated with lower citation impact

Single pivotal trials with few corroborating characteristics were used for FDA approval of cancer therapies

Novel cancer therapies are often approved with evidence from a single pivotal trial alone. There are concerns about the credibility of this evidence. Higher validity may be indicated by five methodological and statistical characteristics of pivotal trial evidence that were described by the U.S. Food and Drug Administration (FDA), which may corroborate the reliance on a single trial alone for approval decisions.; We did a metaepidemiologic evaluation of all single pivotal trials supporting FDA approval of novel drugs and therapeutic biologicals for cancers between 2000 and 2016. For each trial, we determined the presence of these five characteristics, which we operationalized as (1) large and multicenter trial (≥200 patients; more than one center); consistent treatment benefits across (2) multiple patient subgroups (in view of FDA reviewers), (3) multiple endpoints (including overall survival, progression-free survival, response rate, health related quality of life), and (4) multiple treatment comparisons (e.g., multi-arm studies); and (5) "statistically very persuasive" results (P-values <0.00125).; Thirty-five of 100 approvals were based on evidence from a single pivotal trial without any further supporting evidence on beneficial effects (20 randomized controlled trials and 15 single-arm trials). The number increased substantially from one approval before 2006 to 23 after 2011. Sixty-six percent (23/35) of the trials were large multicenter trials (median 301 patients and 63 centers). Consistent effects were demonstrated across subgroups in 66% (23/35), across endpoints in 43% (15/35), and across multiple comparisons in 3% (1/35). Very low P-values for the primary endpoint were seen in 34% (12/35). At least one of the corroborating characteristics was present in 94% (33/35) of all approvals, two or more were present in 54% (19/35), and none had all characteristics.; Single pivotal trials typically have some of the corroborating characteristics, but often only one or two. These characteristics need to be better operationalized, defined, and reported and whether single trials with such characteristics provide similar evidence about benefits and harms of novel treatments as multiple trials would do needs to be shown

Current use and costs of electronic health records for clinical trial research : a descriptive study

Electronic health records (EHRs) may support randomized controlled trials (RCTs). We aimed to describe the current use and costs of EHRs in RCTs, with a focus on recruitment and outcome assessment.; This descriptive study was based on a PubMed search of RCTs published since 2000 that evaluated any medical intervention with the use of EHRs. Cost information was obtained from RCT investigators who used EHR infrastructures for recruitment or outcome measurement but did not explore EHR technology itself.; We identified 189 RCTs, most of which (153 [81.0%]) were carried out in North America and were published recently (median year 2012 [interquartile range 2009-2014]). Seventeen RCTs (9.0%) involving a median of 732 (interquartile range 73-2513) patients explored interventions not related to EHRs, including quality improvement, screening programs, and collaborative care and disease management interventions. In these trials, EHRs were used for recruitment (14 [82%]) and outcome measurement (15 [88%]). Overall, in most of the trials (158 [83.6%]), the outcome (including many of the most patient-relevant clinical outcomes, from unscheduled hospital admission to death) was measured with the use of EHRs. The per-patient cost in the 17 EHR-supported trials varied from US$44 to US$2000, and total RCT costs from US$67 750 to US$5 026 000. In the remaining 172 RCTs (91.0%), EHRs were used as a modality of intervention.; Randomized controlled trials are frequently and increasingly conducted with the use of EHRs, but mainly as part of the intervention. In some trials, EHRs were used successfully to support recruitment and outcome assessment. Costs may be reduced once the data infrastructure is established

The Comparative Effectiveness of Innovative Treatments for Cancer (CEIT-Cancer) project: Rationale and design of the database and the collection of evidence available at approval of novel drugs

Abstract Background The available evidence on the benefits and harms of novel drugs and therapeutic biologics at the time of approval is reported in publicly available documents provided by the US Food and Drug Administration (FDA). We aimed to create a comprehensive database providing the relevant information required to systematically analyze and assess this early evidence in meta-epidemiological research. Methods We designed a modular and flexible database of systematically collected data. We identified all novel cancer drugs and therapeutic biologics approved by the FDA between 2000 and 2016, recorded regulatory characteristics, acquired the corresponding FDA approval documents, identified all clinical trials reported therein, and extracted trial design characteristics and treatment effects. Herein, we describe the rationale and design of the data collection process, particularly the organization of the data capture, the identification and eligibility assessment of clinical trials, and the data extraction activities. Discussion We established a comprehensive database on the comparative effects of drugs and therapeutic biologics approved by the FDA over a time period of 17 years for the treatment of cancer (solid tumors and hematological malignancies). The database provides information on the clinical trial evidence available at the time of approval of novel cancer treatments. The modular nature and structure of the database and the data collection processes allow updates, expansions, and adaption for a continuous meta-epidemiological analysis of novel drugs. The database allows us to systematically evaluate benefits and harms of novel drugs and therapeutic biologics. It provides a useful basis for meta-epidemiological research on the comparative effects of innovative cancer treatments and continuous evaluations of regulatory developments