Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates:empirical study

OBJECTIVE: To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects. DESIGN: Cross-sectional study. DATA SOURCES: We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports. ANALYSIS: We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion). RESULTS: 31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects. CONCLUSION: There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses

Recruitment difficulties in a primary care cluster randomised trial:investigating factors contributing to general practitioners' recruitment of patients

BACKGROUND: Recruitment of patients by health professionals is reported as one of the most challenging steps when undertaking studies in primary care settings. Numerous investigations of the barriers to patient recruitment in trials which recruit patients to receive an intervention have been published. However, we are not aware of any studies that have reported on the recruitment barriers as perceived by health professionals to recruiting patients into cluster randomised trials where patients do not directly receive an intervention. This particular subtype of cluster trial is commonly termed a professional-cluster trial. The aim of this study was to investigate factors that contributed to general practitioners recruitment of patients in a professional-cluster trial which evaluated the effectiveness of an intervention to increase general practitioners adherence to a clinical practice guideline for acute low-back pain. METHOD: General practitioners enrolled in the study were posted a questionnaire, consisting of quantitative items and an open-ended question, to assess possible reasons for poor patient recruitment. Descriptive statistics were used to summarise quantitative items and responses to the open-ended question were coded into categories. RESULTS: Seventy-nine general practitioners completed at least one item (79/94 = 84%), representing 68 practices (85% practice response rate), and 44 provided a response to the open-ended question. General practitioners recalled inviting a median of two patients with acute low-back pain to participate in the trial over a seven-month period; they reported that they intended to recruit patients, but forgot to approach patients to participate; and they did not perceive that patients had a strong interest or disinterest in participating. Additional open-ended comments were generally consistent with the quantitative data. CONCLUSION: A number of barriers to the recruitment of patients with acute low-back pain by general practitioners in a professional-cluster trial were identified. These barriers were similar to those that have been identified in the literature surrounding the recruitment of patients in individual patient randomised trials. To advance the evidence base for patient recruitment strategies in primary care settings, trialists undertaking professional-cluster trials need to develop and evaluate patient recruitment strategies that minimise the efforts required by practice staff to recruit patients, while also meeting privacy and ethical responsibilities and minimising the risk of selection bias. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN012606000098538 (date registered 14/03/2006)

Tools for assessing risk of reporting biases in studies and syntheses of studies:A systematic review

BackgroundSeveral scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.MethodsWe searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.ResultsWe identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.ConclusionThere are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.</jats:sec

The use of segmented regression in analysing interrupted time series studies : an example in pre-hospital ambulance care

Peer reviewedPublisher PD

Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines

Objectives: The aim of the study was to generate a comprehensive bank of systematic review (SR) reporting items to inform an update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2009 statement. Methods: We searched the Enhancing the QUAlity and Transparency Of health Research Network library in May 2019 to identify all reporting guidelines for SRs that were published after 2009, regardless of the scope of the guideline. We also conducted a selective review of four guidance manuals for SRs, three tools for assessing the risk of bias in SRs, six meta-research studies evaluating the reporting quality of SRs using a tailored checklist, and five reporting guidelines for other study designs. One author screened and selected sources for inclusion, extracted reporting guidance from sources, and mapped guidance against the PRISMA 2009 checklist items. Results: We included 60 sources providing guidance on reporting of SRs and meta-analyses. From these, we collated a list of 221 unique reporting items. Items were categorized into title (four items), abstract (10 items), introduction (12 items), methods (111 items), results (61 items), discussion (12 items), funding and conflicts of interest (four items), administrative information (three items), and data availability (four items). This exercise generated 175 reporting items that could be added to the guidance in the PRISMA 2009 statement. Conclusion: Generation of a comprehensive item bank through review and mapping of the literature facilitates identification of missing items and those needing modification, which may not otherwise be identified by the guideline development team or from other activities commonly used to develop reporting guidelines

An investigation of the impact of using different methods for network meta-analysis: a protocol for an empirical evaluation.

BACKGROUND: Network meta-analysis, a method to synthesise evidence from multiple treatments, has increased in popularity in the past decade. Two broad approaches are available to synthesise data across networks, namely, arm- and contrast-synthesis models, with a range of models that can be fitted within each. There has been recent debate about the validity of the arm-synthesis models, but to date, there has been limited empirical evaluation comparing results using the methods applied to a large number of networks. We aim to address this gap through the re-analysis of a large cohort of published networks of interventions using a range of network meta-analysis methods. METHODS: We will include a subset of networks from a database of network meta-analyses of randomised trials that have been identified and curated from the published literature. The subset of networks will include those where the primary outcome is binary, the number of events and participants are reported for each direct comparison, and there is no evidence of inconsistency in the network. We will re-analyse the networks using three contrast-synthesis methods and two arm-synthesis methods. We will compare the estimated treatment effects, their standard errors, treatment hierarchy based on the surface under the cumulative ranking (SUCRA) curve, the SUCRA value, and the between-trial heterogeneity variance across the network meta-analysis methods. We will investigate whether differences in the results are affected by network characteristics and baseline risk. DISCUSSION: The results of this study will inform whether, in practice, the choice of network meta-analysis method matters, and if it does, in what situations differences in the results between methods might arise. The results from this research might also inform future simulation studies

Synthesis without meta-analysis (SWiM) in systematic reviews : reporting guideline

In systematic reviews that lack data amenable to meta-analysis, alternative synthesis methods are commonly used, but these methods are rarely reported. This lack of transparency in the methods can cast doubt on the validity of the review findings. The Synthesis Without Meta-analysis (SWiM) guideline has been developed to guide clear reporting in reviews of interventions in which alternative synthesis methods to meta-analysis of effect estimates are used. This article describes the development of the SWiM guideline for the synthesis of quantitative data of intervention effects and presents the nine SWiM reporting items with accompanying explanations and examples

Investing in updating: how do conclusions change when Cochrane systematic reviews are updated?

BACKGROUND: Cochrane systematic reviews aim to provide readers with the most up-to-date evidence on the effects of healthcare interventions. The policy of updating Cochrane reviews every two years consumes valuable time and resources and may not be appropriate for all reviews. The objective of this study was to examine the effect of updating Cochrane systematic reviews over a four year period. METHODS: This descriptive study examined all completed systematic reviews in the Cochrane Database of Systematic Reviews (CDSR) Issue 2, 1998. The latest version of each of these reviews was then identified in CDSR Issue 2, 2002 and changes in the review were described. For reviews that were updated within this time period and had additional studies, we determined whether their conclusion had changed and if there were factors that were predictive of this change. RESULTS: A total of 377 complete reviews were published in CDSR Issue 2, 1998. In Issue 2, 2002, 14 of these reviews were withdrawn and one was split, leaving 362 reviews to examine for the purpose of this study. Of these reviews, 254 (70%) were updated. Of these updated reviews, 23 (9%) had a change in conclusion. Both an increase in precision and a change in statistical significance of the primary outcome were predictive of a change in conclusion of the review. CONCLUSION: The concerns around a lack of updating for some reviews may not be justified considering the small proportion of updated reviews that resulted in a changed conclusion. A priority-setting approach to the updating of Cochrane systematic reviews may be more appropriate than a time-based approach. Updating all reviews as frequently as every two years may not be necessary, however some reviews may need to be updated more often than every two years

Stakeholder views regarding ethical issues in the design and conduct of pragmatic trials : study protocol

This work is supported by the Canadian Institutes of Health Research through the Project Grant competition (competitive, peer reviewed), award number PJT-153045. Jeremy Grimshaw holds a Canada Research Chair in Health Knowledge Transfer and Uptake and a CIHR Foundation Grant (FDN-143269). Charles Weijer holds a Canada Research Chair in Bioethics. Joanne McKenzie is supported by an Australian National Health and Medical Research Council Career Development Fellowship (1143429). Vipul Jairath hold a personal Endowed Chair at Western University (John and Susan McDonald Endowed Chair). Marion Campbell is based with the Health Services Research Unit which is core-funded by the Chief Scientist Office of the Scottish Government Health and Social Care Directorates. Ian Graham is a CIHR Foundation Grant recipient (FDN# 143237).Peer reviewedPublisher PD

Economic evaluation of the NET intervention versus guideline dissemination for management of mild head injury in hospital emergency departments

Abstract Background Evidence-based guidelines for the management of mild traumatic brain injury (mTBI) in the emergency department (ED) are now widely available, and yet, clinical practice remains inconsistent with the guidelines. The Neurotrauma Evidence Translation (NET) intervention was developed to increase the uptake of guideline recommendations and improve the management of minor head injury in Australian emergency departments (EDs). However, the adoption of this type of intervention typically entails an upfront investment that may or may not be fully offset by improvements in clinical practice, health outcomes and/or reductions in health service utilisation. The present study estimates the cost and cost-effectiveness of the NET intervention, as compared to the passive dissemination of the guideline, to evaluate whether any improvements in clinical practice or health outcomes due to the NET intervention can be obtained at an acceptable cost. Methods and findings Study setting: The NET cluster randomised controlled trial [ACTRN12612001286831]. Study sample: Seventeen EDs were randomised to the control condition and 14 to the intervention. One thousand nine hundred forty-three patients were included in the analysis of clinical practice outcomes (NET sample). A total of 343 patients from 14 control and 10 intervention EDs participated in follow-up interviews and were included in the analysis of patient-reported health outcomes (NET-Plus sample). Outcome measures: Appropriate post-traumatic amnesia (PTA) screening in the ED (primary outcome). Secondary clinical practice outcomes: provision of written information on discharge (INFO) and safe discharge (defined as CT scan appropriately provided plus PTA plus INFO). Secondary patient-reported, post-discharge health outcomes: anxiety (Hospital Anxiety and Depression Scale), post-concussive symptoms (Rivermead), and preference-based health-related quality of life (SF6D). Methods: Trial-based economic evaluations from a health sector perspective, with time horizons set to coincide with the final follow-up for the NET sample (2 months post-intervention) and to 1-month post-discharge for the NET-Plus sample. Results: Intervention and control groups were not significantly different in health service utilisation received in the ED/inpatient ward following the initial mTBI presentation (adjusted mean difference $23.86 per patient; 95$106, $153; p = 0.719) or over the longer follow-up in the NET-plus sample (adjusted mean difference$341.78 per patient; 95%CI − $58,$742; p = 0.094). Savings from lower health service utilisation are therefore unlikely to offset the significantly higher upfront cost of the intervention (mean difference $138.20 per patient; 95$135, $141; p < 0.000). Estimates of the net effect of the intervention on total cost (intervention cost net of health service utilisation) suggest that the intervention entails significantly higher costs than the control condition (adjusted mean difference$169.89 per patient; 95%CI $43,$297, p = 0.009). This effect is larger in absolute magnitude over the longer follow-up in the NET-plus sample (adjusted mean difference $505.06; 95$96, $915; p = 0.016), mostly due to additional health service utilisation. For the primary outcome, the NET intervention is more costly and more effective than passive dissemination; entailing an additional cost of$1246 per additional patient appropriately screened for PTA ($169.89/0.1363; Fieller’s 95$525, $2055). For NET to be considered cost-effective with 95% confidence, decision-makers would need to be willing to trade one quality-adjusted life year (QALY) for 25 additional patients appropriately screened for PTA. While these results reflect our best estimate of cost-effectiveness given the data, it is possible that a NET intervention that has been scaled and streamlined ready for wider roll-out may be more or less cost-effective than the NET intervention as delivered in the trial. Conclusions While the NET intervention does improve the management of mTBI in the ED, it also entails a significant increase in cost and—as delivered in the trial—is unlikely to be cost-effective at currently accepted funding thresholds. There may be a scope for a scaled-up and streamlined NET intervention to achieve a better balance between costs and outcomes. Trial registration Australian New Zealand Clinical Trials Registry ACTRN12612001286831, date registered 12 December 2012