Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis

BACKGROUND: Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question. METHODS: We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews. RESULTS: Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P < 0.001) and lower precision (P < 0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently. CONCLUSIONS: Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question

Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis

BACKGROUND: Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question. METHODS: We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews. RESULTS: Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P < 0.001) and lower precision (P < 0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently. CONCLUSIONS: Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question

Inter-observer agreement of canine and feline paroxysmal event semiology and classification by veterinary neurology specialists and non-specialists

Background: Advances in mobile technology mean vets are now commonly presented with videos of paroxysmal events by clients, but the consistency of the interpretation of these videos has not been investigated. The objective of this study was to investigate the level of agreement between vets (both neurology specialists and non-specialists) on the description and classification of videos depicting paroxysmal events, without knowing any results of diagnostic workup. An online questionnaire study was conducted, where participants watched 100 videos of dogs and cats exhibiting paroxysmal events and answered questions regarding: epileptic seizure presence (yes/ no), seizure type, consciousness status, and the presence of motor, autonomic and neurobehavioural signs. Agreement statistics (percentage agreement and kappa) calculated for each variable, with prevalence indices calculated to aid their interpretation. Results: Only a fair level of agreement (kappa = 0.40) was found for epileptic seizure presence. Overall agreement of seizure type was moderate (kappa = 0.44), with primary generalised seizures showing the highest level of agreement (kappa = 0.60), and focal the lowest (kappa = 0.31). Fair agreement was found for consciousness status and the presence of autonomic signs (kappa = 0.21-0.40), but poor agreement for neurobehavioral signs (kappa = 0.16). Agreement for motor signs ranged from poor (kappa = <= 0.20) to moderate (kappa = 0.41-0.60). Differences between specialists and non-specialists were identified. Conclusions: The relatively low levels of agreement described here highlight the need for further discussions between neurology experts regarding classifying and describing epileptic seizures, and additional training of non-specialists to facilitate accurate diagnosis. There is a need for diagnostic tools (e.g. electroencephalogram) able to differentiate between epileptic and non-epileptic paroxysms

Adjusting COVID-19 reports for countries' age disparities: a comparative framework for reporting performances

Objectives: The COVID-19 outbreak has impacted distinct health care systems differently. While the rate of disease for COVID-19 is highly age-variant, there is no unified and age/gender-inclusive reporting taking place. This renders the comparison of individual countries based on their corresponding metrics, such as CFR difficult. In this paper, we examine cross-country differences, in terms of the age distribution of symptomatic cases, hospitalizations, intensive care unit (ICU) cases, and fatalities. In addition, we propose a new quality measure (called dissonance ratio) to facilitate comparison of countries’ performance in testing and reporting COVID-19 cases (i.e., their reporting quality). Methods: By combining population pyramids with estimated COVID-19 age dependent conditional probabilities, we bridge country-level incidence data gathered from different countries and attribute the variability in data to country demographics. Results: We show that age-adjustment can account for as much as a 22-fold difference in the expected number of fatalities across different countries. We provide case, hospitalization, ICU, and fatality breakdown estimates for a comprehensive list of countries. Also, a comparison is conducted between countries in terms of their performance in reporting COVID-19 cases and fatalities. Conclusions: Our research sheds light on the importance of and propose a methodology to use countries’ population pyramids for obtaining accurate estimates of the healthcare system requirements based on the experience of other, already affected, countries at the time of pandemics

Comparison of established and emerging biodosimetry assays

Rapid biodosimetry tools are required to assist with triage in the case of a large-scale radiation incident. Here, we aimed to determine the dose-assessment accuracy of the well-established dicentric chromosome assay (DCA) and cytokinesis-block micronucleus assay (CBMN) in comparison to the emerging &gamma;-H2AX foci and gene expression assays for triage mode biodosimetry and radiation injury assessment. Coded blood samples exposed to 10 X-ray doses (240 kVp, 1 Gy/min) of up to 6.4 Gy were sent to participants for dose estimation. Report times were documented for each laboratory and assay. The mean absolute difference (MAD) of estimated doses relative to the true doses was calculated. We also merged doses into binary dose categories of clinical relevance and examined accuracy, sensitivity and specificity of the assays. Dose estimates were reported by the first laboratories within 0.3-0.4 days of receipt of samples for the &gamma;-H2AX and gene expression assays compared to 2.4 and 4 days for the DCA and CBMN assays, respectively. Irrespective of the assay we found a 2.5-4-fold variation of interlaboratory accuracy per assay and lowest MAD values for the DCA assay (0.16 Gy) followed by CBMN (0.34 Gy), gene expression (0.34 Gy) and &gamma;-H2AX (0.45 Gy) foci assay. Binary categories of dose estimates could be discriminated with equal efficiency for all assays, but at doses &ge;1.5 Gy a 10% decrease in efficiency was observed for the foci assay, which was still comparable to the CBMN assay. In conclusion, the DCA has been confirmed as the gold standard biodosimetry method, but in situations where speed and throughput are more important than ultimate accuracy, the emerging rapid molecular assays have the potential to become useful triage tools

A yeast fermentate improves gastrointestinal discomfort and constipation by modulation of the gut microbiome : results from a randomized double-blind placebo-controlled pilot trial

Background: Constipation and symptoms of gastrointestinal discomfort such as bloating are common among otherwise healthy individuals, but with significant impact on quality of life. Despite the recognized contribution of the gut microbiome to this pathology, little is known about which group(s) of microorganism(s) are playing a role. A previous study performed in vitro suggests that EpiCor (R) fermentate has prebiotic-like properties, being able to favorably modulate the composition of the gut microbiome. Therefore, the aim of this study was to investigate the effects of EpiCor fermentate in a population with symptoms of gastrointestinal discomfort and reduced bowel movements and to evaluate its effect at the level of the gut microbiome. Methods: This pilot study was performed according to a randomized, double-blind, placebo-controlled parallel design. Eighty subjects with symptoms of gastrointestinal discomfort and constipation were allocated to one of two trial arms (placebo or EpiCor fermentate). Randomization was done in a stratified manner according to symptom severity, resulting in two subgroups of patients: severe and moderate. Daily records of gastrointestinal symptoms were assessed on a 5-point scale, and also stool frequency and consistency were documented during a 2-week run-in and a 6-week intervention phases. Averages over two-week intervals were calculated. Constipation-associated quality of life and general perceived stress were assessed at baseline and after 3 and 6 weeks of intervention. Fecal samples were also collected at these same time points. Results: EpiCor fermentate led to a significant improvement of symptoms such as bloating/distension (p = 0.033 and p = 0.024 after 2 and 4 weeks of intervention, respectively), feeling of fullness (p = 0.004 and p = 0.023 after 2 and 4 weeks of intervention, respectively) and general daily scores (p = 0.046 after 2 weeks of intervention) in the moderate subgroup. A significant improvement in stool consistency was observed for the total population (p = 0.023 after 2 weeks of intervention) as well as for the severe subgroup (p = 0.046 after 2 weeks of intervention), and a nearly significant increase in stool frequency was detected for the total cohort (p = 0.083 and p = 0.090 after 2 and 4 weeks of intervention, respectively). These effects were accompanied by an improvement in constipation-associated quality of life and general perceived stress, particularly in the moderate subgroup. Members of the families Bacteroidaceae and Prevotellaceae, two groups of bacteria that have been previously reported to be deficient in constipated patients, were found to increase with EpiCor fermentate in the severe subgroup. In the moderate subgroup, a significant increase in Akkermansia muciniphila was observed. Conclusions: Despite the relatively low dose administered (500 mg/day), particularly when comparing to the high recommended doses for prebiotic fibers, EpiCor fermentate was able to modulate the composition of the gut microbiome, resulting in improvement of constipation-associated symptoms. Conversely, the reported increase in bowel movements may have altered the gut microbial community by increasing those groups of bacteria that are better adapted to a faster gastrointestinal transit time

Variation in dengue virus plaque reduction neutralization testing: systematic review and pooled analysis.

BackgroundThe plaque reduction neutralization test (PRNT) remains the gold standard for the detection of serologic immune responses to dengue virus (DENV). While the basic concept of the PRNT remains constant, this test has evolved in multiple laboratories, introducing variation in materials and methods. Despite the importance of laboratory-to-laboratory comparability in DENV vaccine development, the effects of differing PRNT techniques on assay results, particularly the use of different dengue strains within a serotype, have not been fully characterized.MethodsWe conducted a systematic review and pooled analysis of published literature reporting individual-level PRNT titers to identify factors associated with heterogeneity in PRNT results and compared variation between strains within DENV serotypes and between articles using hierarchical models.ResultsThe literature search and selection criteria identified 8 vaccine trials and 25 natural exposure studies reporting 4,411 titers from 605 individuals using 4 different neutralization percentages, 3 cell lines, 12 virus concentrations and 51 strains. Of 1,057 titers from primary DENV exposure, titers to the exposure serotype were consistently higher than titers to non-exposure serotypes. In contrast, titers from secondary DENV exposures (n = 628) demonstrated high titers to exposure and non-exposure serotypes. Additionally, PRNT titers from different strains within a serotype varied substantially. A pooled analysis of 1,689 titers demonstrated strain choice accounted for 8.04% (90% credible interval [CrI]: 3.05%, 15.7%) of between-titer variation after adjusting for secondary exposure, time since DENV exposure, vaccination and neutralization percentage. Differences between articles (a proxy for inter-laboratory differences) accounted for 50.7% (90% CrI: 30.8%, 71.6%) of between-titer variance.ConclusionsAs promising vaccine candidates arise, the lack of standardized assays among diagnostic and research laboratories make unbiased inferences about vaccine-induced protection difficult. Clearly defined, widely accessible reference reagents, proficiency testing or algorithms to adjust for protocol differences would be a useful first step in improving dengue PRNT comparability and quality assurance