Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis

AbstractObjectiveWe aim to illustrate the potential impact of a new study on a meta-analysis, which gives an indication of the robustness of the meta-analysis.Study Design and SettingA number of augmentations are proposed to one of the most widely used of graphical displays, the funnel plot. Namely, 1) statistical significance contours, which define regions of the funnel plot in which a new study would have to be located to change the statistical significance of the meta-analysis; and 2) heterogeneity contours, which show how a new study would affect the extent of heterogeneity in a given meta-analysis. Several other features are also described, and the use of multiple features simultaneously is considered.ResultsThe statistical significance contours suggest that one additional study, no matter how large, may have a very limited impact on the statistical significance of a meta-analysis. The heterogeneity contours illustrate that one outlying study can increase the level of heterogeneity dramatically.ConclusionThe additional features of the funnel plot have applications including 1) informing sample size calculations for the design of future studies eligible for inclusion in the meta-analysis; and 2) informing the updating prioritization of a portfolio of meta-analyses such as those prepared by the Cochrane Collaboration

A re-evaluation of fixed effect(s) meta-analysis

Summary Meta-analysis is a common tool for synthesizing results of multiple studies. Among methods for performing meta-analysis, the approach known as ‘fixed effects’ or ‘inverse variance weighting’ is popular and widely used. A common interpretation of this method is that it assumes that the underlying effects in contributing studies are identical, and for this reason it is sometimes dismissed by practitioners. However, other interpretations of fixed effects analyses do not make this assumption, yet appear to be little known in the literature. We review these alternative interpretations, describing both their strengths and their limitations. We also describe how heterogeneity of the underlying effects can be addressed, with the same minimal assumptions, through either testing or meta-regression. Recommendations for the practice of meta-analysis are given; it is hoped that these will foster more direct connection of the questions that meta-analysts wish to answer with the statistical methods they choose.</jats:p

Tools for assessing risk of reporting biases in studies and syntheses of studies:A systematic review

BackgroundSeveral scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.MethodsWe searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.ResultsWe identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.ConclusionThere are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.</jats:sec

The median and the mode as robust meta-analysis estimators in the presence of small-study effects and outliers

Meta-analyses based on systematic literature reviews are commonly used to obtain a quantitative summary of the available evidence on a given topic. However, the reliability of any meta-analysis is constrained by that of its constituent studies. One major limitation is the possibility of small-study effects, when estimates from smaller and larger studies differ systematically. Small-study effects may result from reporting biases (ie, publication bias), from inadequacies of the included studies that are related to study size, or from reasons unrelated to bias. We propose two estimators based on the median and mode to increase the reliability of findings in a meta-analysis by mitigating the influence of small-study effects. By re-examining data from published meta-analyses and by conducting a simulation study, we show that these estimators offer robustness to a range of plausible bias mechanisms, without making explicit modelling assumptions. They are also robust to outlying studies without explicitly removing such studies from the analysis. When meta-analyses are suspected to be at risk of bias because of small-study effects, we recommend reporting the mean, median and modal pooled estimates.</p

Impact of placebo arms on outcomes in antidepressant trials:systematic review and meta-regression analysis

Background: There is debate in the literature as to whether inclusion of a placebo arm may alter characteristics of antidepressant trials. However, previous research has focused on response rates of various antidepressants on average only, ignoring potential differences among drugs or other aspects of trial findings. Little is known about the impact of a placebo arm on all-cause dropout and dropout due to adverse events.Methods: We carried out a systematic review of published and unpublished double-blind randomized controlled trials (RCTs) for the acute treatment of unipolar major depression (update: January 2016). The probability of being allocated to placebo (π) was the exposure of interest, and we examined its influence on responders (efficacy), all-cause dropouts (acceptability) and dropouts due to adverse events (tolerability), while accounting for differences in drugs, trials and patient characteristics in multivariate random effects meta-regression.Results: We included 421 studies (68 305 participants) comparing 16 antidepressants or placebo; π ranged from 20% to 50%. Response rate was lower [risk ratio (RR) 0.87; 95% confidence interval (CI) 0.83, 0.92] and all-cause dropout rate higher (RR 1.19; 95% CI 1.08, 1.31) for the same antidepressants in placebo-controlled trials compared with head-to-head trials. The probability of responding decreased by 3% (95% CI 2-5%) for every 10% increase in π, whereas the risk of all-cause dropout increased by 4% (95% CI 1-7%). Tolerability was unaffected by π. Response rate was inversely correlated with dropouts due to any cause (correlation coefficient -0.48; 95% CI -0.58, -0.36) and due to adverse events (-0.34; 95% CI -0.44, -0.23).Conclusions: For the same antidepressant, response rate was on average smaller and dropouts higher when placebo was included; however, no association was found with dropouts due to adverse events. Decreased patient expectations, larger dropout rates and use of inappropriate statistical methods to impute missing data may explain this phenomenon. The findings call for caution in the integration of randomized evidence involving placebo arms.</p

ROBIS: A new tool to assess risk of bias in systematic reviews was developed

AbstractObjectiveTo develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies).Study Design and SettingWe used four-stage approach to develop ROBIS: define the scope, review the evidence base, hold a face-to-face meeting, and refine the tool through piloting.ResultsROBIS is currently aimed at four broad categories of reviews mainly within health care settings: interventions, diagnosis, prognosis, and etiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews (“reviews of reviews”), and review authors who might want to assess or avoid risk of bias in their reviews. The tool is completed in three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias. Phase 2 covers four domains through which bias may be introduced into a systematic review: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings. Phase 3 assesses the overall risk of bias in the interpretation of review findings and whether this considered limitations identified in any of the phase 2 domains. Signaling questions are included to help judge concerns with the review process (phase 2) and the overall risk of bias in the review (phase 3); these questions flag aspects of review design related to the potential for bias and aim to help assessors judge risk of bias in the review process, results, and conclusions.ConclusionsROBIS is the first rigorously developed tool designed specifically to assess the risk of bias in systematic reviews

SARS-CoV-2 Infection and the Risk of Suicidal and Self-Harm Thoughts and Behaviour:A Systematic Review

OBJECTIVE: The COVID-19 pandemic has had a complex impact on risks of suicide and non-fatal self-harm worldwide with some evidence of increased risk in specific populations including women, young people, and people from ethnic minority backgrounds. This review aims to systematically address whether SARS-CoV-2 infection and/or COVID-19 disease confer elevated risk directly. METHOD: As part of a larger Living Systematic Review examining self-harm and suicide during the pandemic, automated daily searches using a broad list of keywords were performed on a comprehensive set of databases with data from relevant articles published between January 1, 2020 and July 18, 2021. Eligibility criteria for our present review included studies investigating suicide and/or self-harm in people infected with SARS-CoV-2 with or without manifestations of COVID-19 disease with a comparator group who did not have infection or disease. Suicidal and self-harm thoughts and behaviour (STBs) were outcomes of interest. Studies were excluded if they reported data for people who only had potential infection/disease without a confirmed exposure, clinical/molecular diagnosis or self-report of a positive SARS-CoV-2 test result. Studies of news reports, treatment studies, and ecological studies examining rates of both SARS-CoV-2 infections and suicide/self-harm rates across a region were also excluded. RESULTS: We identified 12 studies examining STBs in nine distinct samples of people with SARS-CoV-2. These studies, which investigated STBs in the general population and in subpopulations, including healthcare workers, generally found positive associations between SARS-CoV-2 infection and/or COVID-19 disease and subsequent suicidal/self-harm thoughts and suicidal/self-harm behaviour. CONCLUSIONS: This review identified some evidence that infection with SARS-CoV-2 and/or COVID-19 disease may be associated with increased risks for suicidal and self-harm thoughts and behaviours but a causal link cannot be inferred. Further research with longer follow-up periods is required to confirm these findings and to establish whether these associations are causal