Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates:empirical study

OBJECTIVE: To explore whether systematic reviewers selectively include trial effect estimates in meta-analyses when multiple are available, and what impact this may have on meta-analytic effects. DESIGN: Cross-sectional study. DATA SOURCES: We randomly selected systematic reviews of interventions from 2 clinical specialties published between January 2010 and 2012. The first presented meta-analysis of a continuous outcome in each review was selected (index meta-analysis), and all trial effect estimates that were eligible for inclusion in the meta-analysis (eg, from multiple scales or time points) were extracted from trial reports. ANALYSIS: We calculated a statistic (the Potential Bias Index (PBI)) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 are suggestive of selective inclusion of effect estimates more or less favourable to the intervention, respectively. The impact of any potential selective inclusion was investigated by comparing the index meta-analytic standardised mean difference (SMD) to the median of a randomly constructed distribution of meta-analytic SMDs (representing the meta-analytic SMD expected when there is no selective inclusion). RESULTS: 31 reviews (250 trials) were included. The estimated PBI was 0.57 (95% CI 0.50 to 0.63), suggesting that trial effect estimates that were more favourable to the intervention were included in meta-analyses slightly more often than expected under a process consistent with random selection; however, the 95% CI included the null hypothesis of no selective inclusion. Any potential selective inclusion did not have an important impact on the meta-analytic effects. CONCLUSION: There was no clear evidence that selective inclusion of trial effect estimates occurred in this sample of meta-analyses. Further research on selective inclusion in other clinical specialties is needed. To enable readers to assess the risk of selective inclusion bias, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses

Recruitment difficulties in a primary care cluster randomised trial:investigating factors contributing to general practitioners' recruitment of patients

BACKGROUND: Recruitment of patients by health professionals is reported as one of the most challenging steps when undertaking studies in primary care settings. Numerous investigations of the barriers to patient recruitment in trials which recruit patients to receive an intervention have been published. However, we are not aware of any studies that have reported on the recruitment barriers as perceived by health professionals to recruiting patients into cluster randomised trials where patients do not directly receive an intervention. This particular subtype of cluster trial is commonly termed a professional-cluster trial. The aim of this study was to investigate factors that contributed to general practitioners recruitment of patients in a professional-cluster trial which evaluated the effectiveness of an intervention to increase general practitioners adherence to a clinical practice guideline for acute low-back pain. METHOD: General practitioners enrolled in the study were posted a questionnaire, consisting of quantitative items and an open-ended question, to assess possible reasons for poor patient recruitment. Descriptive statistics were used to summarise quantitative items and responses to the open-ended question were coded into categories. RESULTS: Seventy-nine general practitioners completed at least one item (79/94 = 84%), representing 68 practices (85% practice response rate), and 44 provided a response to the open-ended question. General practitioners recalled inviting a median of two patients with acute low-back pain to participate in the trial over a seven-month period; they reported that they intended to recruit patients, but forgot to approach patients to participate; and they did not perceive that patients had a strong interest or disinterest in participating. Additional open-ended comments were generally consistent with the quantitative data. CONCLUSION: A number of barriers to the recruitment of patients with acute low-back pain by general practitioners in a professional-cluster trial were identified. These barriers were similar to those that have been identified in the literature surrounding the recruitment of patients in individual patient randomised trials. To advance the evidence base for patient recruitment strategies in primary care settings, trialists undertaking professional-cluster trials need to develop and evaluate patient recruitment strategies that minimise the efforts required by practice staff to recruit patients, while also meeting privacy and ethical responsibilities and minimising the risk of selection bias. TRIAL REGISTRATION: Australian New Zealand Clinical Trials Registry ACTRN012606000098538 (date registered 14/03/2006)

The use of segmented regression in analysing interrupted time series studies : an example in pre-hospital ambulance care

Peer reviewedPublisher PD

IMPLEmenting a clinical practice guideline for acute low back pain evidence-based manageMENT in general practice (IMPLEMENT) : cluster randomised controlled trial study protocol

Background: Evidence generated from reliable research is not frequently implemented into clinical practice. Evidence-based clinical practice guidelines are a potential vehicle to achieve this. A recent systematic review of implementation strategies of guideline dissemination concluded that there was a lack of evidence regarding effective strategies to promote the uptake of guidelines. Recommendations from this review, and other studies, have suggested the use of interventions that are theoretically based because these may be more effective than those that are not. An evidencebased clinical practice guideline for the management of acute low back pain was recently developed in Australia. This provides an opportunity to develop and test a theory-based implementation intervention for a condition which is common, has a high burden, and for which there is an evidence-practice gap in the primary care setting. Aim: This study aims to test the effectiveness of a theory-based intervention for implementing a clinical practice guideline for acute low back pain in general practice in Victoria, Australia. Specifically, our primary objectives are to establish if the intervention is effective in reducing the percentage of patients who are referred for a plain x-ray, and improving mean level of disability for patients three months post-consultation. Methods/Design: This study protocol describes the details of a cluster randomised controlled trial. Ninety-two general practices (clusters), which include at least one consenting general practitioner, will be randomised to an intervention or control arm using restricted randomisation. Patients aged 18 years or older who visit a participating practitioner for acute non-specific low back pain of less than three months duration will be eligible for inclusion. An average of twenty-five patients per general practice will be recruited, providing a total of 2,300 patient participants. General practitioners in the control arm will receive access to the guideline using the existing dissemination strategy. Practitioners in the intervention arm will be invited to participate in facilitated face-to-face workshops that have been underpinned by behavioural theory. Investigators (not involved in the delivery of the intervention), patients, outcome assessors and the study statistician will be blinded to group allocation. Trial registration: Australian New Zealand Clinical Trials Registry ACTRN012606000098538 (date registered 14/03/2006).The trial is funded by the NHMRC by way of a Primary Health Care Project Grant (334060). JF has 50% of her time funded by the Chief Scientist Office3/2006). of the Scottish Government Health Directorate and 50% by the University of Aberdeen. PK is supported by a NHMRC Health Professional Fellowship (384366) and RB by a NHMRC Practitioner Fellowship (334010). JG holds a Canada Research Chair in Health Knowledge Transfer and Uptake. All other authors are funded by their own institutions

Tools for assessing risk of reporting biases in studies and syntheses of studies:A systematic review

BackgroundSeveral scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.MethodsWe searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.ResultsWe identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.ConclusionThere are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.</jats:sec

Developing theory-informed behaviour change interventions to implement evidence into practice : a systematic approach using the Theoretical Domains Framework

PMID: 22531013 [PubMed - indexed for MEDLINE] PMCID: PMC3443064 Free PMC ArticlePeer reviewedPublisher PD

Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines

Objectives: The aim of the study was to generate a comprehensive bank of systematic review (SR) reporting items to inform an update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2009 statement. Methods: We searched the Enhancing the QUAlity and Transparency Of health Research Network library in May 2019 to identify all reporting guidelines for SRs that were published after 2009, regardless of the scope of the guideline. We also conducted a selective review of four guidance manuals for SRs, three tools for assessing the risk of bias in SRs, six meta-research studies evaluating the reporting quality of SRs using a tailored checklist, and five reporting guidelines for other study designs. One author screened and selected sources for inclusion, extracted reporting guidance from sources, and mapped guidance against the PRISMA 2009 checklist items. Results: We included 60 sources providing guidance on reporting of SRs and meta-analyses. From these, we collated a list of 221 unique reporting items. Items were categorized into title (four items), abstract (10 items), introduction (12 items), methods (111 items), results (61 items), discussion (12 items), funding and conflicts of interest (four items), administrative information (three items), and data availability (four items). This exercise generated 175 reporting items that could be added to the guidance in the PRISMA 2009 statement. Conclusion: Generation of a comprehensive item bank through review and mapping of the literature facilitates identification of missing items and those needing modification, which may not otherwise be identified by the guideline development team or from other activities commonly used to develop reporting guidelines

Evaluation of a Theory-Informed Implementation Intervention for the Management of Acute Low Back Pain in General Medical Practice: The IMPLEMENT Cluster Randomised Trial

Introduction: This cluster randomised trial evaluated an intervention to decrease x-ray referrals and increase giving advice to stay active for people with acute low back pain (LBP) in general practice. Methods: General practices were randomised to either access to a guideline for acute LBP (control) or facilitated interactive workshops (intervention). We measured behavioural predictors (e.g. knowledge, attitudes and intentions) and fear avoidance beliefs. We were unable to recruit sufficient patients to measure our original primary outcomes so we introduced other outcomes measured at the general practitioner (GP) level: behavioural simulation (clinical decision about vignettes) and rates of x-ray and CT-scan (medical administrative data). All those not involved in the delivery of the intervention were blinded to allocation. Results: 47 practices (53 GPs) were randomised to the control and 45 practices (59 GPs) to the intervention. The number of GPs available for analysis at 12 months varied by outcome due to missing confounder information; a minimum of 38 GPs were available from the intervention group, and a minimum of 40 GPs from the control group. For the behavioural constructs, although effect estimates were small, the intervention group GPs had greater intention of practising consistent with the guideline for the clinical behaviour of x-ray referral. For behavioural simulation, intervention group GPs were more likely to adhere to guideline recommendations about x-ray (OR 1.76, 95%CI 1.01, 3.05) and more likely to give advice to stay active (OR 4.49, 95%CI 1.90 to 10.60). Imaging referral was not statistically significantly different between groups and the potential importance of effects was unclear; rate ratio 0.87 (95%CI 0.68, 1.10) for x-ray or CT-scan. Conclusions: The intervention led to small changes in GP intention to practice in a manner that is consistent with an evidence-based guideline, but it did not result in statistically significant changes in actual behaviour. Trial Registration: Australian New Zealand Clinical Trials Registry ACTRN01260600009853

An investigation of the impact of using different methods for network meta-analysis: a protocol for an empirical evaluation.

BACKGROUND: Network meta-analysis, a method to synthesise evidence from multiple treatments, has increased in popularity in the past decade. Two broad approaches are available to synthesise data across networks, namely, arm- and contrast-synthesis models, with a range of models that can be fitted within each. There has been recent debate about the validity of the arm-synthesis models, but to date, there has been limited empirical evaluation comparing results using the methods applied to a large number of networks. We aim to address this gap through the re-analysis of a large cohort of published networks of interventions using a range of network meta-analysis methods. METHODS: We will include a subset of networks from a database of network meta-analyses of randomised trials that have been identified and curated from the published literature. The subset of networks will include those where the primary outcome is binary, the number of events and participants are reported for each direct comparison, and there is no evidence of inconsistency in the network. We will re-analyse the networks using three contrast-synthesis methods and two arm-synthesis methods. We will compare the estimated treatment effects, their standard errors, treatment hierarchy based on the surface under the cumulative ranking (SUCRA) curve, the SUCRA value, and the between-trial heterogeneity variance across the network meta-analysis methods. We will investigate whether differences in the results are affected by network characteristics and baseline risk. DISCUSSION: The results of this study will inform whether, in practice, the choice of network meta-analysis method matters, and if it does, in what situations differences in the results between methods might arise. The results from this research might also inform future simulation studies

Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement

Objectives: To describe the processes used to update the PRISMA 2009 statement for reporting systematic reviews, present results of a survey conducted to inform the update, summarize decisions made at the PRISMA update meeting, and describe and justify changes made to the guideline. Methods: We reviewed 60 documents with reporting guidance for systematic reviews to generate suggested modifications to the PRISMA 2009 statement. We invited 220 systematic review methodologists and journal editors to complete a survey about the suggested modifications. The results of these projects were discussed at a 21-member in-person meeting. Following the meeting, we drafted the PRISMA 2020 statement and refined it based on feedback from co-authors and a convenience sample of 15 systematic reviewers. Results: The review of 60 documents revealed that all topics addressed by the PRISMA 2009 statement could be modified. Of the 110 survey respondents, more than 66% recommended keeping six of the original checklist items as they were and modifying 15 of them using wording suggested by us. Attendees at the in-person meeting supported the revised wording for several items but suggested rewording for most to enhance clarity, and further refinements were made over six drafts of the guideline. Conclusions: The PRISMA 2020 statement consists of updated reporting guidance for systematic reviews. We hope that providing this detailed description of the development process will enhance the acceptance and uptake of the guideline and assist those developing and updating future reporting guidelines