A randomised trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement

Peer reviewedPostprintPostprintPostprintPostprintPostprintPostprintPostprintPostprintPostprintPostprintPostprin

Bias Due to Changes in Specified Outcomes during the Systematic Review Process

Background Adding, omitting or changing outcomes after a systematic review protocol is published can result in bias because it increases the potential for unacknowledged or post hoc revisions of the planned analyses. The main objective of this study was to look for discrepancies between primary outcomes listed in protocols and in the subsequent completed reviews published on the Cochrane Library. A secondary objective was to quantify the risk of bias in a set of meta-analyses where discrepancies between outcome specifications in protocols and reviews were found. Methods and Findings New reviews from three consecutive issues of the Cochrane Library were assessed. For each review, the primary outcome(s) listed in the review protocol and the review itself were identified and review authors were contacted to provide reasons for any discrepancies. Over a fifth (64/288, 22%) of protocol/review pairings were found to contain a discrepancy in at least one outcome measure, of which 48 (75%) were attributable to changes in the primary outcome measure. Where lead authors could recall a reason for the discrepancy in the primary outcome, there was found to be potential bias in nearly a third (8/28, 29%) of these reviews, with changes being made after knowledge of the results from individual trials. Only 4(6%) of the 64 reviews with an outcome discrepancy described the reason for the change in the review, with no acknowledgment of the change in any of the eight reviews containing potentially biased discrepancies. Outcomes that were promoted in the review were more likely to be significant than if there was no discrepancy (relative risk 1.66 95% CI (1.10, 2.49), p = 0.02). Conclusion In a review, making changes after seeing the results for included studies can lead to biased and misleading interpretation if the importance of the outcome (primary or secondary) is changed on the basis of those results. Our assessment showed that reasons for discrepancies with the protocol are not reported in the review, demonstrating an under-recognition of the problem. Complete transparency in the reporting of changes in outcome specification is vital; systematic reviewers should ensure that any legitimate changes to outcome specification are reported with reason in the review

Outcome measures in rheumatoid arthritis randomised trials over the last 50 years

BACKGROUND: The development and application of standardised sets of outcomes to be measured and reported in clinical trials have the potential to increase the efficiency and value of research. One of the most notable of the current outcome sets began nearly 20 years ago: the World Health Organization and International League of Associations for Rheumatology core set of outcomes for rheumatoid arthritis clinical trials, originating from the OMERACT (Outcome Measures in Rheumatology) Initiative. This study assesses the use of this core outcome set by randomised trials in rheumatology. METHODS: An observational review was carried out of 350 randomised trials for the treatment of rheumatoid arthritis identified through The Cochrane Library (up to and including September 2012 issue). Reports of these trials were evaluated to determine whether or not there were trends in the proportion of trials reporting on the full set of core outcomes over time. Researchers who conducted trials after the publication of the core set were contacted to assess their awareness of it and to collect reasons for non-inclusion of the full core set of outcomes in the study. RESULTS: Since the introduction of the core set of outcomes for rheumatoid arthritis, the consistency of measurement of the core set of outcomes has improved, although variation in the choice of measurement instrument remains. The majority of trialists who responded said that they would consider using the core outcome set in the design of a new trial. CONCLUSIONS: This observational review suggests that a higher percentage of trialists conducting trials in rheumatoid arthritis are now measuring the rheumatoid arthritis core outcome set. Core outcome sets have the potential to improve the evidence base for health care, but consideration must be given to the methods for disseminating their availability amongst the relevant communities

Core Outcome Set-STAndards for Development: The COS-STAD recommendations

Background The use of core outcome sets (COS) ensures that researchers measure and report those outcomes that are most likely to be relevant to users of their research. Several hundred COS projects have been systematically identified to date, but there has been no formal quality assessment of these studies. The Core Outcome Set-STAndards for Development (COS-STAD) project aimed to identify minimum standards for the design of a COS study agreed upon by an international group, while other specific guidance exists for the final reporting of COS development studies (Core Outcome Set-STAndards for Reporting [COS-STAR]). Methods and findings An international group of experienced COS developers, methodologists, journal editors, potential users of COS (clinical trialists, systematic reviewers, and clinical guideline developers), and patient representatives produced the COS-STAD recommendations to help improve the quality of COS development and support the assessment of whether a COS had been developed using a reasonable approach. An open survey of experts generated an initial list of items, which was refined by a 2-round Delphi survey involving nearly 250 participants representing key stakeholder groups. Participants assigned importance ratings for each item using a 1–9 scale. Consensus that an item should be included in the set of minimum standards was defined as at least 70% of the voting participants from each stakeholder group providing a score between 7 and 9. The Delphi survey was followed by a consensus discussion with the study management group representing multiple stakeholder groups. COS-STAD contains 11 minimum standards that are the minimum design recommendations for all COS development projects. The recommendations focus on 3 key domains: the scope, the stakeholders, and the consensus process. Conclusions The COS-STAD project has established 11 minimum standards to be followed by COS developers when planning their projects and by users when deciding whether a COS has been developed using reasonable methods

The importance of integration of stakeholder views in core outcome set development: Otitis Media with Effusion in children with cleft palate

© 2015 Harman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Background: Approximately 75% of children with cleft palate (CP) have Otitis Media with Effusion (OME) histories. Evidence for the effective management of OME in these children is lacking. The inconsistency in outcome measurement in previous studies has led to a call for the development of a Core Outcome Set (COS). Despite the increase in the number of published COS, involvement of patients in the COS development process, and methods to integrate the views of patients and health professionals, to date have been limited. Methods and Findings: A list of outcomes measured in previous research was identified through reviewing the literature. Opinion on the importance of each of these outcomes was then sought from key stakeholders: Ear, Nose and Throat (ENT) surgeons, audiologists, cleft surgeons, speech and language therapists, specialist cleft nurses, psychologists, parents and children. The opinion of health professionals was sought in a three round Delphi survey where participants were asked to score each outcome using a bespoke online system. Parents and children were also asked to score outcomes in a survey and provided an in-depth insight into having OME through semi-structured interviews. The results of the Delphi survey, interviews and parent/patient survey were brought together in a final consensus meeting with representation from all stakeholders. A final set of eleven outcomes reached the definition of "consensus in" to form the recommended COS: hearing; chronic otitis media (COM); OME; receptive language skills; speech development; psycho social development; acute otitis media (AOM); cholesteatoma; side effects of treatment; listening skills; otalgia. Conclusions: We have produced a recommendation about the outcomes that should be measured, as a minimum, in studies of the management of OME in children with CP. The development process included input from key stakeholders and used novel methodology to integrate the opinion of healthcare professionals, parents and children