Interpretive analysis of 85 systematic reviews suggests narrative syntheses and meta-analyses are incommensurate in argumentation

Introduction. Using Toulmin’s argumentation theory, we analysed the texts of systematic reviews in the area of workplace health promotion to explore differences in the modes of reasoning embedded in reports of narrative synthesis as compared to reports of meta-analysis. Methods. We used framework synthesis, grounded theory and cross-case analysis methods to analyse 85 systematic reviews addressing intervention effectiveness in workplace health promotion. Results. Two core categories, or ‘modes of reasoning’, emerged to frame the contrast between narrative synthesis and meta-analysis: practical-configurational reasoning in narrative synthesis (‘what is going on here? what picture emerges?’) and inferential-predictive reasoning in meta-analysis (‘does it work, and how well? will it work again?’). Modes of reasoning examined quality and consistency of the included evidence differently. Meta-analyses clearly distinguished between warrant and claim, whereas narrative syntheses often presented joint warrant-claims. Conclusion. Narrative syntheses and meta-analyses represent different modes of reasoning. Systematic reviewers are likely to be addressing research questions in different ways with each method. It is important to consider narrative synthesis in its own right as a method and to develop specific quality criteria and understandings of how it is done, not merely as a complement to, or second-best option for, meta-analysis

Conceptual and statistical analysis of complex interventions in the presence of confounding variables: An example from public health

Background: Meta-analyses of complex interventions are challenging because causality operates through multiple paths and confounding variables can be difficult to distinguish. Objectives: To meta-analyse public health interventions that engage members of the community in their conception, design, or delivery. To disentangle intervention complexity by analysing according to their theories of change. Study selection criteria: Published after 1990; outcome or process evaluation; community engagement intervention; written in English; reported health or community outcomes; study populations or differential impacts reported according to social determinants of health. Analysis: Intervention complexity was examined by conceptualising, operationalising, and mapping their theories of change; and through random effects subgroup analyses. Main results: 131 studies were included in the synthesis. Three main theories of change were identified, which were useful in describing trends in intervention effectiveness. Statistically significant between-group differences were not detected, since there were likely to have been too many confounding variables. Conclusions: Intervention complexity in systematic reviews can be addressed through examining theories of change and trends in effect size estimates. Such complexity appears to defy current meta-analytical methods when confounding variables undermine analysis of variance

Prioritising references for systematic reviews with RobotAnalyst: A user study

Screening references is a time-consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web-based software system that combines text-mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract-level decisions. The number of references that needed to be screened to identify 95% of the abstract-level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large-scale study using technology-assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings

Are medical procedures that induce coughing or involve respiratory suctioning associated with increased generation of aerosols and risk of SARS-CoV-2 infection? A rapid systematic review.

BACKGROUND: The risk of transmission of SARS-CoV-2 from aerosols generated by medical procedures is a cause for concern. AIM: To evaluate the evidence for aerosol production and transmission of respiratory infection associated with procedures that involve airway suctioning or induce coughing/sneezing. METHODS: The review was informed by PRISMA guidelines. Searches were conducted in PubMed for studies published between January 1st, 2003 and October 6th, 2020. Included studies examined whether nasogastric tube insertion, lung function tests, nasendoscopy, dysphagia assessment, or suctioning for airway clearance result in aerosol generation or transmission of SARS-CoV-2, SARS-CoV, MERS, or influenza. Risk of bias assessment focused on robustness of measurement, control for confounding, and applicability to clinical practice. FINDINGS: Eighteen primary studies and two systematic reviews were included. Three epidemiological studies found no association between nasogastric tube insertion and acquisition of respiratory infections. One simulation study found low/very low production of aerosols associated with pulmonary lung function tests. Seven simulation studies of endoscopic sinus surgery suggested significant increases in aerosols but findings were inconsistent; two clinical studies found airborne particles associated with the use of microdebriders/drills. Some simulation studies did not use robust measures to detect particles and are difficult to equate to clinical conditions. CONCLUSION: There was an absence of evidence to suggest that the procedures included in the review were associated with an increased risk of transmission of respiratory infection. In order to better target precautions to mitigate risk, more research is required to determine the characteristics of medical procedures and patients that increase the risk of transmission of SARS-CoV-2

Variations of mixed methods reviews approaches: A case study

Conducting mixed methods reviews is challenging. The aim of this paper is to describe a range of rationales for and approaches to mixed methods reviews, with a particular focus on one research group. A case study was conducted to describe the mixed methods review process used at the Department of Health and Social Care Reviews Facility in England. The case study used document analysis. A total of 30 mixed methods reviews were identified and analyzed. The analysis revealed five key dimensions on which the reviews varied: review questions and purposes of the mixed methods questions, types of evidence and sources, reasons for using a mixed methods approach, synthesis methods and designs, and integration strategies. The questions in the included reviews addressed stakeholders' views, and intervention processes and/or intervention effectiveness. The mixed methods questions addressed four different purposes: comparing findings, identifying critical intervention features, quantifying effects, and making recommendations. Five main sources of evidence were used: formal evidence from primary studies, informal evidence, policy documents, systematic reviews, and work with stakeholders. Twelve reasons for conducting mixed methods reviews were identified: completeness, contextual understanding, credibility, different research questions, diversity of views, enhancement, explanation, process, triangulation, utility, development of a framework, and identification of promising interventions. Each review employed one or several integration strategies for comparing findings, connecting phases and/or assimilating data. It is hoped that the information garnered from this study will provide useful insights into mixed method review diversity and trigger new ideas for conducting this type of review