Dwan, KM

Kirkham, JJ

Williamson, PR

University of Liverpool Repository

Dr Kerry Dwan, PhD, Cochrane Editorial Unit, Cochrane, UK Professor Paula R Williamson, PhD, Department of Biostatistics, University of Liverpool, UK Dr Jamie Kirkham, PhD, Department of Biostatistics, University of Liverpool, UK  597 words  Dr Kerry Dwan Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK kdwan@cochrane.org   To the Editor: The prevalence and impact of outcome reporting bias in randomized controlled trials (RCTs) within Cochrane systematic reviews has previously been investigated [1]. A recommendation from this research was that studies should not be excluded from reviews on the basis that there was ‘no relevant outcome data’ (NROD), as failure to report on review outcomes does not imply that the outcomes were not measured. Moreover, this recommendation is an expected methodological standard for Cochrane intervention reviews [2]. Quality assurance screening of reviews carried out by the Cochrane Editorial Unit (CEU) has identified that reviews still exclude studies on the basis of NROD. We investigated the proportion of Cochrane reviews excluding studies on the basis of NROD and whether the proportion had changed following the implementation of review screening.   Methods.  New Cochrane reviews were included from all Cochrane review groups published from June to August in 2013 (pre-screening), 2014 (screening of all new reviews), 2015 (screening of all new reviews) and 2016 (screening based on a referral basis by the Cochrane review groups). For each included review, investigators extracted the number of included studies, the number of excluded studies and the number of excluded studies due to NROD. To determine whether studies were excluded due to NROD, the relevant methods, results and characteristics of studies section of the review were scrutinised. Any uncertainties regarding the reasons for excluded studies were resolved through discussion between the investigators. If a review excluded a study due to NROD, the review protocol was checked to ascertain whether exclusion based on NROD was a pre-specified criterion for study exclusion. The proportion of reviews excluding studies due to NROD for each year was calculated. Relative risks (RR) and 95% confidence intervals (CI) were calculated to determine whether full screening or referred screening reduced the number of reviews excluding studies due to NROD.        Results.  434 new reviews were identified in the reference period. Over a quarter of reviews excluded studies based on NROD in the pre-screening period, while this figure reduced to under a quarter in the new review screen and referred screening phases (TABLE). The result was almost significant for a reduced risk of reviews excluding studies due to NROD if all new reviews were screened (RR 0.91 CI (0.81, 1.03)) or were referred for screening (RR 0.93 CI (0.80, 1.08)) compared to pre-screening. Results were similar when removing reviews that pre-specified that studies would be excluded due to NROD.     Comment.  Since the CEU introduced the screening programme the percentage of reviews excluding studies on the basis of NROD has reduced. However, around a fifth of reviews are still excluding studies based on the lack of reporting of outcomes of interest in trial reports. Restricting synthesis to only studies that report on relevant outcome constitutes research waste, if other, otherwise eligible studies are discarded based on failure to report outcome data. Excluding outcome data from meta-analysis in this way has previously been shown to overestimate the treatment effect, which may potentially lead to incorrect recommendations regarding treatment [1]. Potential missing outcome data from excluded studies could be obtained from contact with trial authors or results posted on trial registries. Methods are available to help authors identify whether outcomes are likely to have been measured [1, 3] and sensitivity analyses have been developed to assess whether the exclusion of data from studies is likely to impact on the results [4]. Future strategies are needed (e.g. specific checks at an earlier point in the process) to prevent authors publishing reviews with NROD as a reason for exclusion and reasons for exclusions need to be improved.    Cohort data Data on reviews that exclude due to NROD Year No. of reviews Summary of No. included studies  Median (IQR) Summary of No. excluded studies  Median (IQR) Summary of No. reviews that excluded studies due to NROD (%)1  Pre-specified in protocol that studies would be excluded due to NROD (%)2 Summary of No. studies excluded due to NROD  Median (IQR) [%]3 2013 (pre-screening) 144 6 (3, 14.25) 14 (5, 33.5)  39  (27%) 6 (15%) 4 (2, 10) [31%] 2014 (new review screening) 91 6 (2, 12) 14 (5.5, 27) 17 (19%) 3 (18%) 4 (2, 7) [27%] 2015 (new review screening) 112 6 (2, 14) 14 (6, 33) 24 (21%) 5 (21%) 2 (1.75, 7) [24%] 2016 (referred screening) 87 6 (2, 13.5) 14 (5, 41.75) 19 (22%) 2 (11%) 3 (1.5, 5) [45%] 1. Denominator taken to be the number of reviews in each three month period 2. Denominator taken to be the number of reviews that excluded studies due to NROD in each three month period 3. Denominator taken from the number of reviews that excluded studies due to NROD and is the total number of included studies plus studies excluded due to NROD in each three month period  References 1. Kirkham, J.J., et al., The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 2010. 340: p. c365. 2. Higgins, J.P.T., et al., Methodological Expectations of Cochrane Intervention Reviews. 2016, Cochrane: London. 3. Saini, P., et al., Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ, 2014. 349: p. g6501. 4. Copas, J., et al., A model-based correction for outcome reporting bias in meta-analysis. Biostatistics, 2014. 15(2): p. 370-83.  

The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews [Excluded studies response]

Abstract

Similar works

Full text

Available Versions

University of Liverpool Repository