Impact of question order on prioritisation of outcomes in the development of a core outcome set: a randomised controlled trial

BACKGROUND: Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery. METHODS: A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey. RESULTS: In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]). CONCLUSIONS: In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias. TRIAL REGISTRATION: The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable

Guillain-Barré syndrome and adjuvanted pandemic influenza A (H1N1) 2009 vaccine: multinational case-control study in Europe

Objective To assess the association between pandemic influenza A (H1N1) 2009 vaccine and Guillain-Barré syndrome

The QuinteT Recruitment Intervention supported five randomized trials to recruit to target: a mixed-methods evaluation

Objective To evaluate the impact of the QuinteT Recruitment Intervention (QRI) on recruitment in challenging randomized controlled trials (RCTs) that have applied the intervention. The QRI aims to understand recruitment difficulties and then implements “QRI actions” to address these as recruitment proceeds. Study Design and Setting A mixed-methods study, comprising (1) before-and-after comparisons of recruitment rates and the numbers of patients approached and (2) qualitative case studies, including documentary analysis and interviews with RCT investigators. Results Five UK-based publicly funded RCTs were included in the evaluation. All recruited to target. Randomized controlled trial 2 and RCT 5 both received up-front prerecruitment training before the intervention was applied. Randomized controlled trial 2 did not encounter recruitment issues and recruited above target from its outset. Recruitment difficulties, particularly communication issues, were identified and addressed through QRI actions in RCTs 1, 3, 4, and 5. Randomization rates significantly improved after QRI action in RCTs 1, 3, and 4. Quintet Recruitment Intervention actions addressed issues with approaching eligible patients in RCTs 3 and 5, which both saw significant increases in the number of patients approached. Trial investigators reported that the QRI had unearthed issues they had been unaware of and reportedly changed their practices after QRI action. Conclusion There is promising evidence to suggest that the QRI can support recruitment to difficult RCTs. This needs to be substantiated with future controlled evaluations

Impact of question order on prioritisation of outcomes in the development of a core outcome set: A randomised controlled trial

Background: Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery. Methods: A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey. Results: In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]). Conclusions: In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias. Trial registration: The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.</p