Reporting of observational studies explicitly aiming to emulate randomized trials: a systematic review

Importance: Observational (nonexperimental) studies that aim to emulate a randomized trial (ie, the target trial) are increasingly informing medical and policy decision-making, but it is unclear how these studies are reported in the literature. Consistent reporting is essential for quality appraisal, evidence synthesis, and translation of evidence to policy and practice. Objective: To assess the reporting of observational studies that explicitly aimed to emulate a target trial. Evidence review: We searched Medline, Embase, PsycINFO, and Web of Science for observational studies published between March 2012 and October 2022 that explicitly aimed to emulate a target trial of a health or medical intervention. Two reviewers double-screened and -extracted data on study characteristics, key predefined components of the target trial protocol and its emulation (eligibility criteria, treatment strategies, treatment assignment, outcome[s], follow-up, causal contrast[s], and analysis plan), and other items related to the target trial emulation. Findings: A total of 200 studies that explicitly aimed to emulate a target trial were included. These studies included 26 subfields of medicine, and 168 (84%) were published from January 2020 to October 2022. The aim to emulate a target trial was explicit in 70 study titles (35%). Forty-three studies (22%) reported use of a published reporting guideline (eg, Strengthening the Reporting of Observational Studies in Epidemiology). Eighty-five studies (43%) did not describe all key items of how the target trial was emulated and 113 (57%) did not describe the protocol of the target trial and its emulation. Conclusion and relevance: In this systematic review of 200 studies that explicitly aimed to emulate a target trial, reporting of how the target trial was emulated was inconsistent. A reporting guideline for studies explicitly aiming to emulate a target trial may improve the reporting of the target trial protocols and other aspects of these emulation attempts.</p

Reporting of observational studies explicitly aiming to emulate randomized trials: a systematic review

Importance: Observational (nonexperimental) studies that aim to emulate a randomized trial (ie, the target trial) are increasingly informing medical and policy decision-making, but it is unclear how these studies are reported in the literature. Consistent reporting is essential for quality appraisal, evidence synthesis, and translation of evidence to policy and practice. Objective: To assess the reporting of observational studies that explicitly aimed to emulate a target trial. Evidence review: We searched Medline, Embase, PsycINFO, and Web of Science for observational studies published between March 2012 and October 2022 that explicitly aimed to emulate a target trial of a health or medical intervention. Two reviewers double-screened and -extracted data on study characteristics, key predefined components of the target trial protocol and its emulation (eligibility criteria, treatment strategies, treatment assignment, outcome[s], follow-up, causal contrast[s], and analysis plan), and other items related to the target trial emulation. Findings: A total of 200 studies that explicitly aimed to emulate a target trial were included. These studies included 26 subfields of medicine, and 168 (84%) were published from January 2020 to October 2022. The aim to emulate a target trial was explicit in 70 study titles (35%). Forty-three studies (22%) reported use of a published reporting guideline (eg, Strengthening the Reporting of Observational Studies in Epidemiology). Eighty-five studies (43%) did not describe all key items of how the target trial was emulated and 113 (57%) did not describe the protocol of the target trial and its emulation. Conclusion and relevance: In this systematic review of 200 studies that explicitly aimed to emulate a target trial, reporting of how the target trial was emulated was inconsistent. A reporting guideline for studies explicitly aiming to emulate a target trial may improve the reporting of the target trial protocols and other aspects of these emulation attempts.</p

Irish media coverage of COVID-19 evidence-based research reports from one national agency

Background: How research findings are presented through domestic news can influence behaviour and risk perceptions, particularly during emergencies such as the coronavirus disease 2019 (COVID-19) pandemic. Monitoring media communications to track misinformation and find information gaps is an important component of emergency risk communication. Therefore, this study investigated the traditional media coverage of nine selected COVID-19 evidence-based research reports and associated press releases (PRs) published during the initial phases of the pandemic (April to July 2020) by one national agency.Methods: NVivo was used for summative content analysis. 'Key messages' from each research report were proposed and 488 broadcast, print, and online media sources were coded at the phrase level. Manifest content was coded and counted to locate patterns in the data (what and how many) while latent content was analysed to further investigate these patterns (why and how). This included the coding of the presence of political and public health actors in coverage.Results: Coverage largely did not misrepresent the results of the reports, however, selective reporting and the variability in the use of quotes from governmental and public health stakeholders changed and contextualised results in different manners than perhaps originally intended in the PR. Reports received varying levels of media attention. Coverage focused on more 'human-interest' stories (eg, spread of COVID-19 by children and excess mortality) as opposed to more technical reports (eg, focusing on viral load, antibodies, testing, etc).Conclusion: Our findings provide a case-study of European media coverage of evidence reports produced by a national agency. Results highlighted several strengths and weaknesses of current communication efforts.</div

Strong recommendations from low certainty evidence: a cross-sectional analysis of a suite of national guidelines

Background: Clinical guidelines should be based on a thorough evaluation of the evidence and generally include a rating of the quality of evidence and assign a strength to recommendations. Grading of Recommendations Assessment, Development and Evaluation (GRADE) guidance warns against making strong recommendations when the certainty of the evidence is low or very low, but has identified five paradigmatic situations (e.g. life-threatening situations) where this may be justified. Aims and objectives: We aimed to characterize the strength of recommendations and certainty of the evidence in Irish National Clinical Guidelines using the GRADE approach. Methods: All National Clinical Guidelines from the National Clinical Effectiveness Committee (NCEC) website using the GRADE approach (fully or partially) were included. All recommendations and their corresponding certainty of the evidence, strength of recommendations and justifications were extracted. Authors classified instances of strong recommendations with low certainty evidence (referred to as discordant recommendations) into one of the five paradigmatic situations. Descriptive statistics were calculated. Results: From the 29 NCEC Clinical Guidelines available at the time of analysis, we identified 8 guidelines using GRADE with a total of 240 recommendations; 38 recommendations did not use the GRADE approach and were excluded. Half of the included guidelines focused on emergency situations. In the final dataset of 202 recommendations, 151 (74.7%) were classified as strong and 51 (25.3%) as conditional. Of the 151 strong recommendations, 55 (36.4%) were supported by high or moderate certainty evidence and 96 (63.6%) by low or very low certainty evidence and were considered discordant. Of these 96 discordant recommendations, 55 (73.7%) were consistent with one of the five paradigmatic situations. However, none were specifically described as such within the guidelines. Conclusions: The proportion of discordant recommendations identified in this analysis was higher than some previous international studies (range of all strong recommendations being discordant 30-50%), but similar to other guidelines focused on emergency situations. The majority of discordant recommendations could be mapped to one of the five situations, but no National Clinical Guideline explicitly referenced this. Guideline developers require further guidance to enable greater transparency in the reporting of the reasons for discordant recommendations.</p

Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol [version 1; peer review: 1 approved, 1 approved with reservations]

Introduction: Evidence syntheses, often in the form of systematic reviews, are essential for clinical guideline development and informing changes to health policies. However, clinical guideline development groups (CGDG) are multidisciplinary, and participants such as policymakers, healthcare professionals and patient representatives can face obstacles when trying to understand and use evidence synthesis findings. Summary formats to communicate the results of evidence syntheses have become increasingly common, but it is currently unclear which format is most effective for different stakeholders. This mixed-methods systematic review (MMSR) evaluates the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. Methods: This protocol follows guidance from the Joanna Briggs Institute on MMSRs and is reported according to the Preferred Reporting Items for Systematic Reviews (PRISMA)-P guideline. A comprehensive search of six databases will be performed with no language restrictions. Primary outcomes are those relating to the effectiveness and preferences for and attitudes towards the different summary formats. We will include qualitative research and randomised controlled trials. Two reviewers will perform title, abstract, and full-text screening. Independent double-extraction of study characteristics and critical appraisal items will be undertaken using a standardised form. We will use a convergent segregated approach to analyse quantitative and qualitative data separately; results will then be integrated. Discussion: The results of this systematic review will provide an overview of the effectiveness and acceptability of different summary formats for evidence synthesis findings. These findings can be helpful for those in or communicating to guideline development groups. The results can also inform the development and pilot-testing of summary formats for evidence summaries.</p

The efectiveness and acceptability of evidence synthesis summary formats for clinical guideline development groups: a mixed-methods systematic review

Introduction: Clinical guideline development often involves a rigorous synthesis of evidence involving multidisciplinary stakeholders with diferent priorities and knowledge of evidence synthesis; this makes communicating fndings complex. Summary formats are typically used to communicate the results of evidence syntheses; however, there is little consensus on which formats are most efective and acceptable for diferent stakeholders.  Methods: This mixed-methods systematic review (MMSR) aimed to evaluate the efectiveness and acceptability (e.g. preferences and attitudes and preferences towards) of evidence synthesis summary formats for GDG members. We followed the PRISMA 2020 guideline and Joanna Briggs Institute Manual for Evidence Synthesis for MMSRs. We searched six databases (inception to April 20, 2021) for randomised controlled trials (RCTs), RCTs with a qualitative component, and qualitative studies. Screening, data extraction, and quality appraisal were performed in duplicate. Qualitative fndings were synthesised using meta-aggregation, and quantitative fndings are described narratively.  Results: We identifed 17,240 citations and screened 54 full-text articles, resulting in 22 eligible articles (20 unique studies): 4 articles reported the results of 5 RCTs, one of which also had a qualitative component. The other 18 articles discussed the results of 16 qualitative studies. Therefore, we had 5 trials and 17 qualitative studies to extract data from. Studies were geographically heterogeneous and included a variety of stakeholders and summary formats. All 5 RCTs assessed knowledge or understanding with 3 reporting improvement with newer formats. The qualitative analysis identifed 6 categories of recommendations: ‘presenting information’, ‘tailoring information’ for end users, ‘trust in producers and summary’, ‘knowledge required’ to understand fndings, ‘quality of evidence’, and properly ‘contextualising information’. Across these categories, the synthesis resulted in 126 recommendations for practice. Nine recommendations were supported by both quantitative and qualitative evidence and 116 by only qualitative. A majority focused on how to present information (n = 64) and tailor content for diferent end users (n = 24).  Conclusions: This MMSR provides guidance on how to improve evidence summary structure and layout. This can be used by synthesis producers to better communicate to GDGs. Study fndings will inform the co-creation of evidence summary format prototypes based on GDG member’s needs.    Trial registration The protocol for this project was previously published, and the project was preregistered on Open Science Framework (Clyne and Sharp, Evidence synthesis and translation of fndings for national clinical guideline development: addressing the needs and preferences of guideline development groups, 2021; Sharp and Clyne, Evidence synthesis summary formats for decision-makers and Clinical Guideline Development Groups: A mixed-methods systematic review protocol, 2021).</p

Evidence synthesis summary formats for clinical guideline development group members: a mixed-methods systematic review protocol [version 2; peer review: 2 approved]

Introduction: Evidence syntheses, often in the form of systematic reviews, are essential for clinical guideline development and informing changes to health policies. However, clinical guideline development groups (CGDG) are multidisciplinary, and participants such as policymakers, healthcare professionals and patient representatives can face obstacles when trying to understand and use evidence synthesis findings. Summary formats to communicate the results of evidence syntheses have become increasingly common, but it is currently unclear which format is most effective for different stakeholders. This mixed-methods systematic review (MMSR) evaluates the effectiveness and acceptability of different evidence synthesis summary formats for CGDG members. Methods: This protocol follows guidance from the Joanna Briggs Institute on MMSRs and is reported according to the Preferred Reporting Items for Systematic Reviews (PRISMA)-P guideline. A comprehensive search of six databases will be performed with no language restrictions. Primary outcomes are those relating to the effectiveness and preferences for and attitudes towards the different summary formats. We will include qualitative research and randomised controlled trials. Two reviewers will perform title, abstract, and full-text screening. Independent double-extraction of study characteristics and critical appraisal items will be undertaken using a standardised form. We will use a convergent segregated approach to analyse quantitative and qualitative data separately; results will then be integrated. Discussion: The results of this systematic review will provide an overview of the effectiveness and acceptability of different summary formats for evidence synthesis findings. These findings can be helpful for those in or communicating to guideline development groups. The results can also inform the development and pilot-testing of summary formats for evidence summaries.</p

Using preprints in evidence synthesis: commentary on experience during the COVID-19 pandemic

Key findings: Evidence syntheses are increasingly drawing on preprint servers as a source for emergent literature on COVID-19. Our research group, has conducted a large number of rapid reviews of a broad range of public health topics related to COVID-19. We outline several considerations when including preprints in rapid reviews and lessons learned from this process.What this adds to what is known?: Including preprints in rapid reviews has implications for the rapid review process and review teams should have clear protocol regarding the selection and coverage of bibliographic databases, indication within reviews where an included study is a preprint and prespecifying any sensitivity analysis (quantitative or narrative) to assess the impact of inclusion of preprints on the overall results and conclusions.Specific challenges encountered in including preprints in rapid reviews such as those related to matching preprints to subsequent peer review publications and dealing with changes between preprints and peer review publications are presented using three exemplar review, and suggestions for study authors and review teams are provided.What is the implication and what should change now?: We suggest that preprint study authors include a statement in the final peer-reviewed version of the manuscript with the citation of the preprint version.Rapid review teams should have a clear policy around whether they will or will not check peer review status of preprints included in a rapid review, and at what point in the review process this would occur.</div

Reporting guidelines used varying methodology to develop recommendations

Background and objectives: We investigated the developing methods of reporting guidelines in the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network's database. Methods: In October 2018, we screened all records and excluded those not describing reporting guidelines from further investigation. Twelve researchers performed duplicate data extraction on bibliometrics, scope, development methods, presentation, and dissemination of all publications. Descriptive statistics were used to summarize the findings. Results: Of the 405 screened records, 262 described a reporting guidelines development. The number of reporting guidelines increased over the past 3 decades, from 5 in the 1990s and 63 in the 2000s to 157 in the 2010s. Development groups included 2-151 people. Literature appraisal was performed during the development of 56% of the reporting guidelines; 33% used surveys to gather external opinion on items to report; and 42% piloted or sought external feedback on their recommendations. Examples of good reporting for all reporting items were presented in 30% of the reporting guidelines. Eighteen percent of the reviewed publications included some level of spin. Conclusion: Reporting guidelines have been developed with varying methodology. Reporting guideline developers should use existing guidance and take an evidence-based approach, rather than base their recommendations on expert opinion of limited groups of individuals.</p

Biomedical doctoral students' research practices when facing dilemmas: two vignette-based randomized control trials

Our aim was to describe the research practices of doctoral students facing a dilemma to research integrity and to assess the impact of inappropriate research environments, i.e. exposure to (a) a post-doctoral researcher who committed a Detrimental Research Practice (DRP) in a similar situation and (b) a supervisor who did not oppose the DRP. We conducted two 2-arm, parallel-group randomized controlled trials. We created 10 vignettes describing a realistic dilemma with two alternative courses of action (good practice versus DRP). 630 PhD students were randomized through an online system to a vignette (a) with (n = 151) or without (n = 164) exposure to a post-doctoral researcher; (b) with (n = 155) or without (n = 160) exposure to a supervisor. The primary outcome was a score from - 5 to + 5, where positive scores indicated the choice of DRP and negative scores indicated good practice. Overall, 37% of unexposed participants chose to commit DRP with important variation across vignettes (minimum 10%; maximum 66%). The mean difference [95%CI] was 0.17 [- 0.65 to 0.99;], p = 0.65 when exposed to the post-doctoral researcher, and 0.79 [- 0.38; 1.94], p = 0.16, when exposed to the supervisor. In conclusion, we did not find evidence of an impact of postdoctoral researchers and supervisors on student research practices.Trial registration: NCT04263805, NCT04263506 (registration date 11 February 2020). </p