Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

AbstractObjective To assess the risk of bias associated with missing outcome data in systematic reviews. Design Imputation study. Setting Systematic reviews. Population 100 systematic reviews that included a group level meta-analysis with a statistically significant effect on a patient important dichotomous efficacy outcome. Main outcome measures Median percentage change in the relative effect estimate when applying each of the following assumption (four commonly discussed but implausible assumptions (best case scenario, none had the event, all had the event, and worst case scenario) and four plausible assumptions for missing data based on the informative missingness odds ratio (IMOR) approach (IMOR 1.5 (least stringent), IMOR 2, IMOR 3, IMOR 5 (most stringent)); percentage of meta-analyses that crossed the threshold of the null effect for each method; and percentage of meta-analyses that qualitatively changed direction of effect for each method. Sensitivity analyses based on the eight different methods of handling missing data were conducted. Results 100 systematic reviews with 653 randomised controlled trials were included. When applying the implausible but commonly discussed assumptions, the median change in the relative effect estimate varied from 0% to 30.4%. The percentage of meta-analyses crossing the threshold of the null effect varied from 1% (best case scenario) to 60% (worst case scenario), and 26% changed direction with the worst case scenario. When applying the plausible assumptions, the median percentage change in relative effect estimate varied from 1.4% to 7.0%. The percentage of meta-analyses crossing the threshold of the null effect varied from 6% (IMOR 1.5) to 22% (IMOR 5) of meta-analyses, and 2% changed direction with the most stringent (IMOR 5). Conclusion Even when applying plausible assumptions to the outcomes of participants with definite missing data, the average change in pooled relative effect estimate is substantive, and almost a quarter (22%) of meta-analyses crossed the threshold of the null effect. Systematic review authors should present the potential impact of missing outcome data on their effect estimates and use this to inform their overall GRADE (grading of recommendations assessment, development, and evaluation) ratings of risk of bias and their interpretation of the results

Meta-Analyses Proved Inconsistent in How Missing Data Were Handled Across Their Included Primary Trials: A Methodological Survey

Background: How systematic review authors address missing data among eligible primary studies remains uncertain. Objective: To assess whether systematic review authors are consistent in the way they handle missing data, both across trials included in the same meta-analysis, and with their reported methods. Methods: We first identified 100 eligible systematic reviews that included a statistically significant meta-analysis of a patient-important dichotomous efficacy outcome. Then, we successfully retrieved 638 of the 653 trials included in these systematic reviews' meta-analyses. From each trial report, we extracted statistical data used in the analysis of the outcome of interest to compare with the data used in the meta-analysis. First, we used these comparisons to classify the "analytical method actually used" for handling missing data by the systematic review authors for each included trial. Second, we assessed whether systematic reviews explicitly reported their analytical method of handling missing data. Third, we calculated the proportion of systematic reviews that were consistent in their "analytical method actually used" across trials included in the same meta-analysis. Fourth, among systematic reviews that were consistent in the "analytical method actually used" across trials and explicitly reported on a method for handling missing data, we assessed whether the "analytical method actually used" and the reported methods were consistent. Results: We were unable to determine the "analytical method reviews actually used" for handling missing outcome data among 397 trials. Among the remaining 241, systematic review authors most commonly conducted "complete case analysis" (n=128, 53%) or assumed "none of the participants with missing data had the event of interest" (n=58, 24%). Only eight of 100 systematic reviews were consistent in their approach to handling missing data across included trials, but none of these reported methods for handling missing data. Among seven reviews that did explicitly report their analytical method of handling missing data, only one was consistent in their approach across included trials (using complete case analysis), and their approach was inconsistent with their reported methods (assumed all participants with missing data had the event). Conclusion: The majority of systematic review authors were inconsistent in their approach towards reporting and handling missing outcome data across eligible primary trials, and most did not explicitly report their methods to handle missing data. Systematic review authors should clearly identify missing outcome data among their eligible trials, specify an approach for handling missing data in their analyses, and apply their approach consistently across all primary trials

Requirements of Clinical Journals for Authors’ Disclosure of Financial and Non-Financial Conflicts of Interest: A Cross Sectional Study

<div><p>Importance</p><p>It is unclear how medical journals address authors’ financial and non-financial conflict of interest (COI).</p><p>Objective</p><p>To assess the policies of clinical journals for disclosure of financial and non-financial COI.</p><p>Methods</p><p>Cross sectional study that included both review of public documents as well as a simulation of a manuscript submission for the National Library of Medicine’s “core clinical journals”. The study did not involve human subjects. Investigators who abstracted the data, reviewed “instructions for authors” on the journal website and, in order to reflect the actual implementation of the COI disclosure policy, simulated the submission of a manuscript. Two individuals working in duplicate and independently to abstract information using a standardized data abstraction form, resolved disagreements by discussion or with the help of a third person.</p><p>Results</p><p>All but one of 117 core clinical journals had a COI policy. All journals required disclosure of financial COI pertaining to the authors and a minority (35%) asked for financial COI disclosure pertaining to the family members or authors' institution (29%). Over half required the disclosure of at least one form of non-financial COI (57%), out of which only two (3%) specifically referred to intellectual COI. Small minorities of journals (17% and 24% respectively) described a potential impact of disclosed COI and of non-disclosure of COI on the editorial process.</p><p>Conclusion</p><p>While financial COI disclosure was well defined by the majority of the journals, many did not have clear policies on disclosure of non-financial COI, disclosure of financial COI of family members and institutions of the authors, and effect of disclosed COI or non-disclosure of COI on editorial policies.</p></div

Requirements for financial conflicts of interest (COI) disclosure; N = 116.

<p>Requirements for financial conflicts of interest (COI) disclosure; N = 116.</p

Characteristics of conflicts of interest (COI) policies (N = 116).

<p>Characteristics of conflicts of interest (COI) policies (N = 116).</p

General characteristics of the journals (N = 117).

<p>General characteristics of the journals (N = 117).</p

Types for non-financial COI for which disclosure is required; N = 66.

<p>Types for non-financial COI for which disclosure is required; N = 66.</p