Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions

Abstract Background Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. It is recommended that authors assess and report the methodological quality of SRs in overviews—for example, using A MeaSurement Tool to Assess systematic Reviews (AMSTAR). Currently, there is variation in whether and how overview authors assess and report SR quality, and limited guidance is available. Our objectives were to: examine methodological considerations involved in using AMSTAR to assess the quality of Cochrane and non-Cochrane SRs in overviews of healthcare interventions; identify challenges (and develop potential decision rules) when using AMSTAR in overviews; and examine the potential impact of considering methodological quality when making inclusion decisions in overviews. Methods We selected seven overviews of healthcare interventions and included all SRs meeting each overview’s inclusion criteria. For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also examined the correlation between AMSTAR assessments and SR results/conclusions. Results Ninety-five SRs were included (30 Cochrane, 65 non-Cochrane). Mean AMSTAR assessments (9.6/11 vs. 5.5/11; p < 0.001) and inter-rater reliability (AC1 statistic: 0.84 vs. 0.69; “almost perfect” vs. “substantial” using the Landis & Koch criteria) were higher for Cochrane compared to non-Cochrane SRs. Four challenges were identified when applying AMSTAR in overviews: the scope of the SRs and overviews often differed; SRs examining similar topics sometimes made different methodological decisions; reporting of non-Cochrane SRs was sometimes poor; and some non-Cochrane SRs included other SRs as well as primary studies. Decision rules were developed to address each challenge. We found no evidence that AMSTAR assessments were correlated with SR results/conclusions. Conclusions Results indicate that the AMSTAR tool can be used successfully in overviews that include Cochrane and non-Cochrane SRs, though decision rules may be useful to circumvent common challenges. Findings support existing recommendations that quality assessments of SRs in overviews be conducted independently, in duplicate, with a process for consensus. Results also suggest that using methodological quality to guide inclusion decisions (e.g., to exclude poorly conducted and reported SRs) may not introduce bias into the overview process

Treatment of Hepatitis C in Children: A Systematic Review

Background: Current guidelines recommend children be treated for hepatitis C virus (HCV) using the same principles applied in adults. There are however few published studies which assess the efficacy and safety of HCV therapy in children. Methodology/Principal Findings: A systematic review of the literature was completed for studies of any design that evaluated HCV therapy in children. The primary outcome was sustained virologic response (SVR), with sub-group analysis of response rates by genotype. There were 4 randomized controlled trials (RCTs) and 31 non-randomized studies, all involving interferon, pegylated interferon (PEG-IFN), or combinations of these drugs with ribavirin. The SVR rate could not be directly compared as the populations and interventions differed across studies. Genotype was not reported or differed substantially from study to study. The overall SVR rate for PEG-IFN and ribavirin ranged from 30 to 100 % which is comparable to the rate in adults. Similar to adults, the SVR rates were significantly higher in children with genotype 2 or 3 compared to genotype 1. Adverse effects were primarily flu-like symptoms and neutropenia. There were insufficient data to assess the applicability of the week 12 stop rule (stopping therapy at week 12 if there is less than a 2 log drop in HCV RNA) or the efficacy of shortening therapy to 24 weeks in children with genotype 2 and 3. Conclusions/Significance: Current guidelines for the treatment of HCV in children are based on limited data. Furthe

Evaluating a Social Media Campaign for a Parent Educational Video on Bronchiolitis

Bronchiolitis, or lower airway swelling, is a common cause of pediatric hospital admissions. Parents have expressed wishes for more information regarding bronchiolitis but had difficulty finding reliable information, suggesting the need for more effective and easily accessible information resources. Knowledge translation (KT) tools like videos provide research-based information and may be conveniently disseminated to large audiences through social media. The purpose of this project was to evaluate the effectiveness of a social media campaign to promote a video on bronchiolitis. A social media campaign was conducted from 14 October to 30 November 2019. User interactions were recorded for the Facebook and Twitter accounts, website, and YouTube of Evidence in Child Health to Enhance Outcomes (ECHO), Alberta Research Centre for Health Evidence (ARCHE), and Translating Emergency Knowledge for Kids (TREKK). Baseline metrics were collected from 1 August to 30 September 2019 and post-campaign metrics were collected from 1 December 2019 to 31 March 2020. Mean monthly changes, standard deviations, and percent changes between periods were generated for the baseline, campaign, and post-campaign periods. Overall, there was a visible increase in user interactions throughout the campaign period. There was an overall downward trend in user interactions following the campaign. These findings suggest that social media may be a useful method of KT tool dissemination when consistently used. The downward trend post-campaign highlights the need for further research to investigate methods to maintain continuous interaction following a campaign

Controlled Trials in Children: Quantity, Methodological Quality and Descriptive Characteristics of Pediatric Controlled Trials Published 1948-2006

BACKGROUND: The objective of this study was to describe randomized controlled trials (RCTs) and controlled clinical trials (CCTs) in child health published between 1948 and 2006, in terms of quantity, methodological quality, and publication and trial characteristics. We used the Trials Register of the Cochrane Child Health Field for overall trends and a sample from this to explore trial characteristics in more detail. METHODOLOGY/PRINCIPAL FINDINGS: We extracted descriptive data on a random sample of 578 trials. Ninety-six percent of the trials were published in English; the percentage of child-only trials was 90.5%. The most frequent diagnostic categories were infectious diseases (13.2%), behavioural and psychiatric disorders (11.6%), neonatal critical care (11.4%), respiratory disorders (8.9%), non-critical neonatology (7.9%), and anaesthesia (6.5%). There were significantly fewer child-only studies (i.e., more mixed child and adult studies) over time (P = 0.0460). The proportion of RCTs to CCTs increased significantly over time (P<0.0001), as did the proportion of multicentre trials (P = 0.002). Significant increases over time were found in methodological quality (Jadad score) (P<0.0001), the proportion of double-blind studies (P<0.0001), and studies with adequate allocation concealment (P<0.0001). Additionally, we found an improvement in reporting over time: adequate description of withdrawals and losses to follow-up (P<0.0001), sample size calculations (P<0.0001), and intention-to-treat analysis (P<0.0001). However, many trials still do not describe their level of blinding, and allocation concealment was inadequately reported in the majority of studies across the entire time period. The proportion of studies with industry funding decreased slightly over time (P = 0.003), and these studies were more likely to report positive conclusions (P = 0.028). CONCLUSIONS/SIGNIFICANCE: The quantity and quality of pediatric controlled trials has increased over time; however, much work remains to be done, particularly in improving methodological issues around conduct and reporting of trials

Protocol for a systematic review of the use of narrative storytelling and visual-arts-based approaches as knowledge translation tools in healthcare

BACKGROUND: The arts are powerful, accessible forms of communication that have the potential to impart knowledge by attracting interest and developing meaningful connections. Knowledge translation aims to reduce the ‘evidence-practice’ gap by developing, implementing and evaluating strategies designed to enhance awareness and promote behavior change congruent with research evidence. Increasingly, innovative approaches such as narrative storytelling and other arts-based interventions are being investigated to bridge the growing gap between practice and research. This study is the first to systematically identify and synthesize current research on narrative storytelling and visual art to translate and disseminate health research. METHODS: A health research librarian will develop and implement search strategies designed to identify relevant evidence. Studies will be included if they are primary research employing narrative storytelling and/or visual art as a knowledge translation strategy in healthcare. Two reviewers will independently perform study selection, quality assessment, and data extraction using standard forms. Disagreements will be resolved through discussion or third party adjudication. Data will be grouped and analyzed by research design, type of knowledge translation strategy (that is, a narrative or visual-arts-based approach), and target audience. An overall synthesis across all studies will be conducted. DISCUSSION: The findings from this research project will describe the ‘state of the science’ regarding the use of narrative storytelling and visual art as knowledge translation strategies. This systematic review will provide critical information for: (1) researchers conducting knowledge translation intervention studies; (2) nursing, medicine, and allied healthcare professionals; (3) healthcare consumers, including patients and families; and (4) decision makers and knowledge users who are charged to increase use of the latest research in healthcare settings

Certified Computation from Unreliable Datasets

A wide range of learning tasks require human input in labeling massive data. The collected data though are usually low quality and contain inaccuracies and errors. As a result, modern science and business face the problem of learning from unreliable data sets. In this work, we provide a generic approach that is based on \textit{verification} of only few records of the data set to guarantee high quality learning outcomes for various optimization objectives. Our method, identifies small sets of critical records and verifies their validity. We show that many problems only need $\text{poly}(1/\varepsilon)$ verifications, to ensure that the output of the computation is at most a factor of $(1 \pm \varepsilon)$ away from the truth. For any given instance, we provide an \textit{instance optimal} solution that verifies the minimum possible number of records to approximately certify correctness. Then using this instance optimal formulation of the problem we prove our main result: "every function that satisfies some Lipschitz continuity condition can be certified with a small number of verifications". We show that the required Lipschitz continuity condition is satisfied even by some NP-complete problems, which illustrates the generality and importance of this theorem. In case this certification step fails, an invalid record will be identified. Removing these records and repeating until success, guarantees that the result will be accurate and will depend only on the verified records. Surprisingly, as we show, for several computation tasks more efficient methods are possible. These methods always guarantee that the produced result is not affected by the invalid records, since any invalid record that affects the output will be detected and verified

Treatment in the pediatric emergency department is evidence based: a retrospective analysis

BACKGROUND: Our goal was to quantify the evidence that is available to the physicians of a pediatric emergency department (PED) in making treatment decisions. Further, we wished to ascertain what percentage of evidence for treatment provided in the PED comes from pediatric studies. METHODS: We conducted a retrospective chart review of randomly selected patients seen in the PED between January 1 and December 31, 2002. The principal investigator identified a primary diagnosis and primary intervention for each chart. A thorough literature search was then undertaken with respect to the primary intervention. If a randomized control trial (RCT) or a systematic review was found, the intervention was classified as level I evidence. If no RCT was found, the intervention was assessed by an expert committee who determined its appropriateness based on face validity (RCTs were unanimously judged to be both unnecessary and, if a placebo would have been involved, unethical). These interventions were classified as level II evidence. Interventions that did not fall into either above category were classified as level III evidence. RESULTS: Two hundred and sixty-two patient charts were reviewed. Of these, 35.9% did not receive a primary intervention. Of the 168 interventions assessed, 80.4% were evidence-based (level I), 7.1% had face validity (level II) and 12.5% had no supporting evidence (level III). Of the evidence-based interventions, 83.7% were supported by studies with mostly pediatric patients. CONCLUSION: Our study demonstrates that a substantial proportion of PED treatment decisions are evidence-based, with most based on studies in pediatric patients. Also, a large number of patients seen in the PED receive no intervention

An evaluation of harvest plots to display results of meta-analyses in overviews of reviews: a cross-sectional study

Survey for Acute Otitis Media. (PDF 63 kb

Do health care institutions value research? A mixed methods study of barriers and facilitators to methodological rigor in pediatric randomized trials

BACKGROUND: Pediatric randomized controlled trials (RCTs) are susceptible to a high risk of bias. We examined the barriers and facilitators that pediatric trialists face in the design and conduct of unbiased trials. METHODS: We used a mixed methods design, with semi-structured interviews building upon the results of a quantitative survey. We surveyed Canadian (n=253) and international (n=600) pediatric trialists regarding their knowledge and awareness of bias and their perceived barriers and facilitators in conducting clinical trials. We then interviewed 13 participants from different subspecialties and geographic locations to gain a more detailed description of how their experiences and attitudes towards research interacted with trial design and conduct. RESULTS: The survey response rate was 23.0% (186/807). 68.1% of respondents agreed that bias is a problem in pediatric RCTs and 72.0% felt that there is sufficient evidence to support changing some aspects of how trials are conducted. Knowledge related to bias was variable, with inconsistent awareness of study design features that may introduce bias into a study. Interview participants highlighted a lack of formal training in research methods, a negative research culture, and the pragmatics of trial conduct as barriers. Facilitators included contact with knowledgeable and supportive colleagues and infrastructure for research. CONCLUSIONS: A lack of awareness of bias and negative attitudes towards research present significant barriers in terms of conducting methodologically rigorous pediatric RCTs. Knowledge translation efforts must focus on these issues to ensure the relevance and validity of trial results