Searching two or more databases decreased the risk of missing relevant studies: a metaresearch study

BACKGROUND AND OBJECTIVES: Assessing changes in coverage, recall, review, conclusions and references not found when searching fewer databases. METHODS: In randomly selected 60 Cochrane reviews, we checked included study publications' coverage (indexation) and recall (findability) using different search approaches with MEDLINE, Embase, and CENTRAL and related them to authors' conclusions and certainty. We assessed characteristics of unfound references. RESULTS: Overall 1989/2080 included references, were indexed in ≥1 database (coverage = 96%). In reviews where using one of our search approaches would not change conclusions and certainty (n = 44-54), median coverage and recall were highest (range 87.9%-100.0% and 78.2%-93.3%, respectively). Here, searching ≥2 databases reached >95% coverage and ≥87.9% recall. In reviews with unchanged conclusions but less certainty (n = 2-8): 63.3%-79.3% coverage and 45.0%-75.0% recall. In reviews with opposite conclusions (n = 1-3): 63.3%-96.6% and 52.1%-78.7%. In reviews where a conclusion was no longer possible (n = 3-7): 60.6%-86.0% and 20.0%-53.8%. The 265 references that were indexed but unfound were more often abstractless (30% vs. 11%) and older (28% vs. 17% published before 1991) than found references. CONCLUSION: Searching ≥2 databases improves coverage and recall and decreases the risk of missing eligible studies. If researchers suspect that relevant articles are difficult to find, supplementary search methods should be used

How to develop rapid reviews of diagnostic tests according to experts: A qualitative exploration of researcher views

Background: Rapid reviews (RRs) have been used to provide timely evidence for policymakers, health providers, and the public in several healthcare scenarios, most recently during the coronavirus disease 2019 pandemic. Despite the essential role of diagnosis in clinical management, data about how to perform RRs of diagnostic tests are scarce. We aimed to explore the views and perceptions of experts in evidence synthesis and diagnostic evidence about the value of methods used to accelerate the review process. Methods: We performed semistructured interviews with a purposive sample of experts in evidence synthesis and diagnostic evidence. We carried out the interviews in English between July and December 2021. Initial reading and coding of the transcripts were performed using NVIVO qualitative data analysis software. Results: Of a total of 23 invited experts, 16 (70%) responded. We interviewed all 16 participants representing key roles in evidence synthesis. We identified 14 recurring themes including the review question, characteristics of the review team, and use of automation, as the topics with the highest number of quotes. Some participants considered several methodological “shortcuts” to be ineffective or risky, such as automating quality appraisal, using only one reviewer for diagnostic data extraction and only performing descriptive analysis. The introduction of limits might depend on whether the test being assessed is a new test, the availability of alternative tests, the needs of providers and patients, and the availability of high‐quality systematic reviews. Conclusions: Our findings suggest that organizational strategies (e.g., defining the review question, availability of a highly experienced team) may have a role in conducting RRs of diagnostic tests. Several methodological shortcuts were considered inadequate for accelerating the review process, though they need to be assessed in well‐designed studies. Improved reporting of RRs would support evidence‐based decision‐making and help users of RRs understand their limitations

Abbreviated and comprehensive literature searches led to identical or very similar effect estimates: a meta-epidemiological study

OBJECTIVES The objective of this study was to assess the agreement of treatment effect estimates from meta-analyses based on abbreviated or comprehensive literature searches. STUDY DESIGN AND SETTING This was a meta-epidemiological study. We abbreviated 47 comprehensive Cochrane review searches and searched MEDLINE/Embase/CENTRAL alone, in combination, with/without checking references (658 new searches). We compared one meta-analysis from each review with recalculated ones based on abbreviated searches. RESULTS The 47 original meta-analyses included 444 trials (median 6 per review interquartile range (IQR) 3-11) with 360045 participants (median 1,371 per review IQR 685-8,041). Depending on the search approach, abbreviated searches led to identical effect estimates in 34-79{\%} of meta-analyses, to different effect estimates with the same direction and level of statistical significance in 15-51{\%}, and to opposite effects (or effects could not be estimated anymore) in 6-13{\%}. The deviation of effect sizes was zero in 50{\%} of the meta-analyses and in 75{\%} not larger than 1.07-fold. Effect estimates of abbreviated searches were not consistently smaller or larger (median ratio of odds ratio 1 IQR 1-1.01) but more imprecise (1.02-1.06-fold larger standard errors). CONCLUSION Abbreviated literature searches often led to identical or very similar effect estimates as comprehensive searches with slightly increased confidence intervals. Relevant deviations may occur

Delphi survey on the most promising areas and methods to improve systematic reviews' production and updating

Background: Systematic reviews (SRs) are invaluable evidence syntheses, widely used in biomedicine and other scientific areas. Tremendous resources are being spent on the production and updating of SRs. There is a continuous need to automatize the process and use the workforce and resources to make it faster and more efficient.Methods: Information gathered by previous EVBRES research was used to construct a questionnaire for round 1 which was partly quantitative, partly qualitative. Fifty five experienced SR authors were invited to participate in a Del‑ phi study (DS) designed to identify the most promising areas and methods to improve the efficient production and updating of SRs. Topic questions focused on which areas of SRs are most time/effort/resource intensive and should be prioritized in further research. Data were analysed using NVivo 12 plus, Microsoft Excel 2013 and SPSS. Thematic analysis findings were used on the topics on which agreement was not reached in round 1 in order to prepare the questionnaire for round 2.Results: Sixty percent (33/55) of the invited participants completed round 1; 44% (24/55) completed round 2. Participants reported average of 13.3 years of experience in conducting SRs (SD 6.8). More than two thirds of the respondents agreed/strongly agreed the following topics should be prioritized: extracting data, literature searching, screen‑ ing abstracts, obtaining and screening full texts, updating SRs, finding previous SRs, translating non-English studies, synthesizing data, project management, writing the protocol, constructing the search strategy and critically appraising. Participants have not considered following areas as priority: snowballing, GRADE-ing, writing SR, deduplication, formulating SR question, performing meta-analysis.Conclusions: Data extraction was prioritized by the majority of participants as an area that needs more research/ methods development. Quality of available language translating tools has dramatically increased over the years (Google translate, DeepL). The promising new tool for snowballing emerged (Citation Chaser). Automation cannot substitute human judgement where complex decisions are needed (GRADE-ing). Trial registration Study protocol was registered at https://osf.io/bp2hu/peer-reviewe

Workplace interventions to reduce the risk of SARS-CoV-2 infection outside of healthcare settings

This is a protocol for a Cochrane Review (intervention). The objectives are as follows: To assess the benefits and harms of interventions in non‐healthcare‐related workplaces to reduce the risk of SARS‐CoV‐2 infection relative to other interventions or no intervention

Pathways leading to success and non-success: a process evaluation of a cluster randomized physical activity health promotion program applying fuzzy-set qualitative comparative analysis

Abstract Background Health promotion programs can only lead to improvements in health outcomes if they are effectively implemented. However, most studies assessing implementation success focus on only one condition, although more conditions influence this process. Therefore, evidence is scarce on what conditions play a role in successful implementation and how they interact. Hence, we aimed to identify which combinations of teacher and implementation process characteristics affected the emotional and social school experience (SCE) of pupils participating in a school-based health promotion program. Methods This study was part of an effectiveness and process evaluation including 24 intervention and 27 control classes. We used fuzzy-set qualitative comparative analysis (fsQCA) to identify combinations of conditions that were associated with either an increase or no increase in the outcome SCE in comparison to the control group at 20 months post intervention. We deductively selected five conditions based on the Consolidated Framework for Implementation Research: teachers’ perceived self-efficacy, teachers’ expectations of the benefits of the intervention, teachers’ previous knowledge about the intervention, dosage of physical activity breaks, and quality of the implementation. Results We identified five different pathways that led to no increase in the pupils’ outcome (parameters of fit: consistency 94%, coverage 66%). The combination of an unsatisfying quality of implementing the intervention and a low previous knowledge about the intervention showed the highest empirical relevance. Similarly, fewer physical activity breaks in combination with other conditions impeded the program’s success. Furthermore, we identified two different pathways characterizing ways to success (consistency: 81%, coverage: 52%). The most relevant combination was good quality implementation of physical activity breaks, implemented by teachers with a high self-efficacy, and a good previous knowledge about the intervention. Conclusions QCA has potential for an in-depth analysis of complex interventions as it can rely on small to medium sample sizes and analyze pathways to success and non-success separately. The investigated program can be improved by considering the following suggestions: The quality of the implementation process should be monitored during the implementation phase, and regular feedback loops and learning opportunities for teachers should accompany a program. Clear recommendations regarding the dosage should be established. Trial registration German register of clinical studies: DRKS00000622. Retrospectively registered: December 3, 2010, (http://www.drks.de/drks_web/setLocale_EN.do). Approved by the Ethics Committee of Lower Austria (GS4-EK-4/107–2010)

Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study

Abstract Background Systematic reviews offer the most reliable and valid support for health policy decision-making, patient information, and guideline development. However, they are labor intensive and frequently take longer than 1 year to complete. Consequently, they often do not meet the needs of those who need to make decisions quickly. Rapid reviews have therefore become a pragmatic alternative to systematic reviews. They are knowledge syntheses that abbreviate certain methodological aspects of systematic reviews to produce information more quickly. Methodological shortcuts often take place in literature identification. A potential drawback is less reliable results. To date, the impact of abbreviated searches on estimates of treatment effects and subsequent conclusions has not been analyzed systematically across multiple bodies of evidence. We aim to answer the research question: Do bodies of evidence that are based on abbreviated literature searches lead to different conclusions about benefits and harms of interventions compared with bodies of evidence that are based on comprehensive, systematic literature searches? Methods We will use a non-inferiority and meta-epidemiologic design. The primary outcome is the proportion of discordant conclusions based on different search approaches. Drawing of a pool of Cochrane reports published between 2012 and 2016, we will randomly select 60 reports. Eligible reports are those that present a summary-of-findings table, draw a clear conclusion, present data for meta-analyses, and document the search strategy clearly. We will conduct several abbreviated searches to detect whether included studies in these Cochrane reviews could be detected. If searches could not detect all studies, we will revise the original summary-of-findings table and ask review authors whether the missed evidence would change conclusions of their report. We will determine the proportion of discordant conclusions for each abbreviated search approach. We will consider an abbreviated search as non-inferior if the lower limit of the 95% confidence interval of the proportion of discordant conclusions is below the non-inferiority margin, which is determined based on results of a survey for clinical and public health scenarios. Discussion This will be the first study to assess whether the reduced sensitivity of abbreviated searches has an impact on conclusions across multiple bodies of evidence, not only on effect estimates

Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey

Abstract Background Decisionmakers and guideline developers demand rapid syntheses of the evidence when time sensitive evidence-informed decisions are required. A potential trade-off of such rapid reviews is that their results can have less reliability than results of systematic reviews that can lead to an increased risk of making incorrect decisions or recommendations. We sought to determine how much incremental uncertainty about the correctness of an answer guideline developers and health policy decisionmakers are willing to accept in exchange for a rapid evidence-synthesis. Methods Employing a purposive sample, we conducted an international web-based, anonymous survey of decisionmakers and guideline developers. Based on a clinical treatment, a public health, and a clinical prevention scenario, participants indicated the maximum risk of getting an incorrect answer from a rapid review that they would be willing to accept. We carefully reviewed data and performed descriptive statistical analyses. Results In total, 325 (58.5%) of 556 participants completed our survey and were eligible for analysis. The median acceptable incremental risk for getting an incorrect answer from a rapid review across all three scenarios was 10.0% (interquartile range [IQR] 5.0–15.0). Acceptable risks were similar for the clinical treatment (n = 313, median 10.0% [IQR 5.0–15.0]) and the public health scenarios (n = 320, median 10.0% [IQR 5.0–15.0]) and lower for the clinical prevention scenario (n = 312, median 6.5% [IQR 5.0–10.5]). Conclusions Findings suggest that decisionmakers are willing to accept some trade-off in validity in exchange for a rapid review. Nevertheless, they expect the validity of rapid reviews to come close to that of systematic reviews