Bias Due to Changes in Specified Outcomes during the Systematic Review Process

Background Adding, omitting or changing outcomes after a systematic review protocol is published can result in bias because it increases the potential for unacknowledged or post hoc revisions of the planned analyses. The main objective of this study was to look for discrepancies between primary outcomes listed in protocols and in the subsequent completed reviews published on the Cochrane Library. A secondary objective was to quantify the risk of bias in a set of meta-analyses where discrepancies between outcome specifications in protocols and reviews were found. Methods and Findings New reviews from three consecutive issues of the Cochrane Library were assessed. For each review, the primary outcome(s) listed in the review protocol and the review itself were identified and review authors were contacted to provide reasons for any discrepancies. Over a fifth (64/288, 22%) of protocol/review pairings were found to contain a discrepancy in at least one outcome measure, of which 48 (75%) were attributable to changes in the primary outcome measure. Where lead authors could recall a reason for the discrepancy in the primary outcome, there was found to be potential bias in nearly a third (8/28, 29%) of these reviews, with changes being made after knowledge of the results from individual trials. Only 4(6%) of the 64 reviews with an outcome discrepancy described the reason for the change in the review, with no acknowledgment of the change in any of the eight reviews containing potentially biased discrepancies. Outcomes that were promoted in the review were more likely to be significant than if there was no discrepancy (relative risk 1.66 95% CI (1.10, 2.49), p = 0.02). Conclusion In a review, making changes after seeing the results for included studies can lead to biased and misleading interpretation if the importance of the outcome (primary or secondary) is changed on the basis of those results. Our assessment showed that reasons for discrepancies with the protocol are not reported in the review, demonstrating an under-recognition of the problem. Complete transparency in the reporting of changes in outcome specification is vital; systematic reviewers should ensure that any legitimate changes to outcome specification are reported with reason in the review

Use of methods for specifying the target difference in randomised controlled trial sample size calculations : Two surveys of trialists' practice

© The Author(s), 2014.Peer reviewedPublisher PD

Consensus-based recommendations for investigating clinical heterogeneity in systematic reviews

Abstract Background Critics of systematic reviews have argued that these studies often fail to inform clinical decision making because their results are far too general, that the data are sparse, such that findings cannot be applied to individual patients or for other decision making. While there is some consensus on methods for investigating statistical and methodological heterogeneity, little attention has been paid to clinical aspects of heterogeneity. Clinical heterogeneity, true effect heterogeneity, can be defined as variability among studies in the participants, the types or timing of outcome measurements, and the intervention characteristics. The objective of this project was to develop recommendations for investigating clinical heterogeneity in systematic reviews. Methods We used a modified Delphi technique with three phases: (1) pre-meeting item generation; (2) face-to-face consensus meeting in the form of a modified Delphi process; and (3) post-meeting feedback. We identified and invited potential participants with expertise in systematic review methodology, systematic review reporting, or statistical aspects of meta-analyses, or those who published papers on clinical heterogeneity. Results Between April and June of 2011, we conducted phone calls with participants. In June 2011 we held the face-to-face focus group meeting in Ann Arbor, Michigan. First, we agreed upon a definition of clinical heterogeneity: Variations in the treatment effect that are due to differences in clinically related characteristics. Next, we discussed and generated recommendations in the following 12 categories related to investigating clinical heterogeneity: the systematic review team, planning investigations, rationale for choice of variables, types of clinical variables, the role of statistical heterogeneity, the use of plotting and visual aids, dealing with outlier studies, the number of investigations or variables, the role of the best evidence synthesis, types of statistical methods, the interpretation of findings, and reporting. Conclusions Clinical heterogeneity is common in systematic reviews. Our recommendations can help guide systematic reviewers in conducting valid and reliable investigations of clinical heterogeneity. Findings of these investigations may allow for increased applicability of findings of systematic reviews to the management of individual patients.http://deepblue.lib.umich.edu/bitstream/2027.42/112777/1/12874_2012_Article_987.pd

A bayesian meta-analysis of multiple treatment comparisons of systemic regimens for advanced pancreatic cancer

© 2014 Chan et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.Background: For advanced pancreatic cancer, many regimens have been compared with gemcitabine (G) as the standard arm in randomized controlled trials. Few regimens have been directly compared with each other in randomized controlled trials and the relative efficacy and safety among them remains unclear

A systematic review of the use of an expertise-based randomised controlled trial design

Acknowledgements JAC held a Medical Research Council UK methodology (G1002292) fellowship, which supported this research. The Health Services Research Unit, Institute of Applied Health Sciences (University of Aberdeen), is core-funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorates. Views express are those of the authors and do not necessarily reflect the views of the funders.Peer reviewedPublisher PD

Stem cell transplantation in traumatic spinal cord injury:a systematic review and meta-analysis of animal studies

Spinal cord injury (SCI) is a devastating condition that causes substantial morbidity and mortality and for which no treatments are available. Stem cells offer some promise in the restoration of neurological function. We used systematic review, meta-analysis, and meta-regression to study the impact of stem cell biology and experimental design on motor and sensory outcomes following stem cell treatments in animal models of SCI. One hundred and fifty-six publications using 45 different stem cell preparations met our prespecified inclusion criteria. Only one publication used autologous stem cells. Overall, allogeneic stem cell treatment appears to improve both motor (effect size, 27.2%; 95% Confidence Interval [CI], 25.0%-29.4%; 312 comparisons in 5,628 animals) and sensory (effect size, 26.3%; 95% CI, 7.9%-44.7%; 23 comparisons in 473 animals) outcome. For sensory outcome, most heterogeneity between experiments was accounted for by facets of stem cell biology. Differentiation before implantation and intravenous route of delivery favoured better outcome. Stem cell implantation did not appear to improve sensory outcome in female animals and appeared to be enhanced by isoflurane anaesthesia. Biological plausibility was supported by the presence of a dose-response relationship. For motor outcome, facets of stem cell biology had little detectable effect. Instead most heterogeneity could be explained by the experimental modelling and the outcome measure used. The location of injury, method of injury induction, and presence of immunosuppression all had an impact. Reporting of measures to reduce bias was higher than has been seen in other neuroscience domains but were still suboptimal. Motor outcomes studies that did not report the blinded assessment of outcome gave inflated estimates of efficacy. Extensive recent preclinical literature suggests that stem-cell-based therapies may offer promise, however the impact of compromised internal validity and publication bias mean that efficacy is likely to be somewhat lower than reported here

Erratum to: Methods for evaluating medical tests and biomarkers

[This corrects the article DOI: 10.1186/s41512-016-0001-y.]

Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal

Mounting evidence suggests that there is frequently considerable variation in the risk of the outcome of interest in clinical trial populations. These differences in risk will often cause clinically important heterogeneity in treatment effects (HTE) across the trial population, such that the balance between treatment risks and benefits may differ substantially between large identifiable patient subgroups; the "average" benefit observed in the summary result may even be non-representative of the treatment effect for a typical patient in the trial. Conventional subgroup analyses, which examine whether specific patient characteristics modify the effects of treatment, are usually unable to detect even large variations in treatment benefit (and harm) across risk groups because they do not account for the fact that patients have multiple characteristics simultaneously that affect the likelihood of treatment benefit. Based upon recent evidence on optimal statistical approaches to assessing HTE, we propose a framework that prioritizes the analysis and reporting of multivariate risk-based HTE and suggests that other subgroup analyses should be explicitly labeled either as primary subgroup analyses (well-motivated by prior evidence and intended to produce clinically actionable results) or secondary (exploratory) subgroup analyses (performed to inform future research). A standardized and transparent approach to HTE assessment and reporting could substantially improve clinical trial utility and interpretability

Ten-year mortality, disease progression, and treatment-related side effects in men with localised prostate cancer from the ProtecT randomised controlled trial according to treatment received

Background The ProtecT trial reported intention-to-treat analysis of men with localised prostate cancer randomly allocated to active monitoring (AM), radical prostatectomy, and external beam radiotherapy. Objective To report outcomes according to treatment received in men in randomised and treatment choice cohorts. Design, setting, and participants This study focuses on secondary care. Men with clinically localised prostate cancer at one of nine UK centres were invited to participate in the treatment trial comparing AM, radical prostatectomy, and radiotherapy. Intervention Two cohorts included 1643 men who agreed to be randomised and 997 who declined randomisation and chose treatment. Outcome measurements and statistical analysis Analysis was carried out to assess mortality, metastasis and progression and health-related quality of life impacts on urinary, bowel, and sexual function using patient-reported outcome measures. Analysis was based on comparisons between groups defined by treatment received for both randomised and treatment choice cohorts in turn, with pooled estimates of intervention effect obtained using meta-analysis. Differences were estimated with adjustment for known prognostic factors using propensity scores. Results and limitations According to treatment received, more men receiving AM died of PCa (AM 1.85%, surgery 0.67%, radiotherapy 0.73%), whilst this difference remained consistent with chance in the randomised cohort (p = 0.08); stronger evidence was found in the exploratory analyses (randomised plus choice cohort) when AM was compared with the combined radical treatment group (p = 0.003). There was also strong evidence that metastasis (AM 5.6%, surgery 2.4%, radiotherapy 2.7%) and disease progression (AM 20.35%, surgery 5.87%, radiotherapy 6.62%) were more common in the AM group. Compared with AM, there were higher risks of sexual dysfunction (95% at 6 mo) and urinary incontinence (55% at 6 mo) after surgery, and of sexual dysfunction (88% at 6 mo) and bowel dysfunction (5% at 6 mo) after radiotherapy. The key limitations are the potential for bias when comparing groups defined by treatment received and changes in the protocol for AM during the lengthy follow-up required in trials of screen-detected PCa. Conclusions Analyses according to treatment received showed increased rates of disease-related events and lower rates of patient-reported harms in men managed by AM compared with men managed by radical treatment, and stronger evidence of greater PCa mortality in the AM group. Patient summary More than 95 out of every 100 men with low or intermediate risk localised prostate cancer do not die of prostate cancer within 10 yr, irrespective of whether treatment is by means of monitoring, surgery, or radiotherapy. Side effects on sexual and bladder function are better after active monitoring, but the risks of spreading of prostate cancer are more common