Statistical reproducibility of meta-analysis research claims for medical mask use in community settings to prevent COVID infection

The coronavirus pandemic (COVID) has been an exceptional test of current scientific evidence that inform and shape policy. Many US states, cities, and counties implemented public orders for mask use on the notion that this intervention would delay and flatten the epidemic peak and largely benefit public health outcomes. P-value plotting was used to evaluate statistical reproducibility of meta-analysis research claims of a benefit for medical (surgical) mask use in community settings to prevent COVID infection. Eight studies (seven meta-analyses, one systematic review) published between 1 January 2020 and 7 December 2022 were evaluated. Base studies were randomized control trials with outcomes of medical diagnosis or laboratory-confirmed diagnosis of viral (Influenza or COVID) illness. Self-reported viral illness outcomes were excluded because of awareness bias. No evidence was observed for a medical mask use benefit to prevent viral infections in six p-value plots (five meta-analyses and one systematic review). Research claims of no benefit in three meta-analyses and the systematic review were reproduced in p-value plots. Research claims of a benefit in two meta-analyses were not reproduced in p-value plots. Insufficient data were available to construct p-value plots for two meta-analyses because of overreliance on self-reported outcomes. These findings suggest a benefit for medical mask use in community settings to prevent viral, including COVID infection, is unproven.Comment: 21 pages, 100 references, 3 appendice

Statistical reliability of meta_analysis research claims for gas stove cooking_childhood respiratory health associations

Odds ratios or p_values from individual observational studies can be combined to examine a common cause_effect research question in meta_analysis. However, reliability of individual studies used in meta_analysis should not be taken for granted as claimed cause_effect associations may not reproduce. An evaluation was undertaken on meta_analysis of base papers examining gas stove cooking, including nitrogen dioxide, NO2, and childhood asthma and wheeze associations. Numbers of hypotheses tested in 14 of 27 base papers, 52 percent, used in meta_analysis of asthma and wheeze were counted. Test statistics used in the meta_analysis, 40 odds ratios with 95 percent confidence limits, were converted to p_values and presented in p_value plots. The median and interquartile range of possible numbers of hypotheses tested in the 14 base papers was 15,360, 6,336_49,152. None of the 14 base papers made mention of correcting for multiple testing, nor was any explanation offered if no multiple testing procedure was used. Given large numbers of hypotheses available, statistics drawn from base papers and used for meta-analysis are likely biased. Even so, p-value plots for gas stove_current asthma and gas stove_current wheeze associations show randomness consistent with unproven gas stove harms. The meta-analysis fails to provide reliable evidence for public health policy making on gas stove harms to children in North America. NO2 is not established as a biologically plausible explanation of a causal link with childhood asthma. Biases_multiple testing and p-hacking_cannot be ruled out as explanations for a gas stove_current asthma association claim. Selective reporting is another bias in published literature of gas stove_childhood respiratory health studies. Keywords gas stove, asthma, meta-analysis, p-value plot, multiple testing, p_hackingComment: International Journal of Statistics and Probability (2023

Evaluation of a Meta-Analysis of Ambient Air Quality as a Risk Factor for Asthma Exacerbation

Background: An irreproducibility crisis currently afflicts a wide range of scientific disciplines, including public health and biomedical science. A study was undertaken to assess the reliability of a meta-analysis examining whether air quality components (carbon monoxide, particulate matter 10 µm and 2.5 µm (PM10 and PM2.5), sulfur dioxide, nitrogen dioxide and ozone) are risk factors for asthma exacerbation. Methods: The number of statistical tests and models were counted in 17 randomly selected base papers from 87 used in the meta-analysis. Confidence intervals from all 87 base papers were converted to p-values. p-value plots for each air component were constructed to evaluate the effect heterogeneity of the p-values. Results: The number of statistical tests possible in the 17 selected base papers was large, median = 15,360 (interquartile range = 1536–40,960), in comparison to results presented. Each p-value plot showed a two-component mixture with small p-values < 0.001 while other p-values appeared random (p-values > 0.05). Given potentially large numbers of statistical tests conducted in the 17 selected base papers, p-hacking cannot be ruled out as explanations for small p-values. Conclusions: Our interpretation of the meta-analysis is that random p-values indicating null associations are more plausible and the meta-analysis is unlikely to replicate in the absence of bias

Air Pollution and Acute Myocardial Infarction Hospital Admission in Alberta, Canada: A Three-Step Procedure Case-Crossover Study.

Adverse associations between air pollution and myocardial infarction (MI) are widely reported in medical literature. However, inconsistency and sensitivity of the findings are still big concerns. An exploratory investigation was undertaken to examine associations between air pollutants and risk of acute MI (AMI) hospitalization in Alberta, Canada. A time stratified case-crossover design was used to assess the transient effect of five air pollutants (carbon monoxide (CO), nitrogen dioxide (NO2), nitric oxide (NO), ozone (O3) and particulate matter with an aerodynamic diameter ≤2.5 (PM2.5)) on the risk of AMI hospitalization over the period 1999-2009. Subgroups were predefined to see if any susceptible group of individuals existed. A three-step procedure, including univariate analysis, multivariate analysis, and bootstrap model averaging, was used. The multivariate analysis was used in an effort to address adjustment uncertainty; whereas the bootstrap technique was used as a way to account for regression model uncertainty. There were 25,894 AMI hospital admissions during the 11-year period. Estimating health effects that are properly adjusted for all possible confounding factors and accounting for model uncertainty are important for making interpretations of air pollution-health effect associations. The most robust findings included: (1) only 1-day lag NO2 concentrations (6-, 12- or 24-hour average), but not those of CO, NO, O3 or PM2.5, were associated with an elevated risk of AMI hospitalization; (2) evidence was suggested for an effect of elevated risk of hospitalization for NSTEMI (Non-ST Segment Elevation Myocardial Infarction), but not for STEMI (ST segment elevation myocardial infarction); and (3) susceptible subgroups included elders (age ≥65) and elders with hypertension. As this was only an exploratory study there is a need to replicate these findings with other methodologies and datasets

Comparison of 1-day lag NO₂ concentration effects in subgroups defined by age and type of AMI.

<p>AGECAT1 = age <65; AGECAT2 = age ≥65; STEMI = ST Segment Elevation Myocardial Infarction; NSTEMI = Non-ST Segment Elevation Myocardial Infarction. OR estimates calculated by bootstrap model averaging of 1000 replications for an inter-quintile range increase of NO2_AVE (28.2 μg/m³), NO2_AVE12 (30.1 μg/m³), NO2_AVE6 (34.2 μg/m³), NO2_MIN (16.9 μg/m³), NO2_MAX (39.5 μg/m³). Freq represents frequency that a variable had p-value ≤0.05 from 1,000 model replications.</p

Step 2 multivariate analysis with full adjustment results (p-value ≤0.05).

<p>Note: AGECAT1 = age <65; AGECAT2 = age ≥65. NSTEMI = Non-ST Segment Elevation Myocardial Infarction; Dysrhy = Dysrhythmia; HTN = Hypertension; AVE = 24-hour average; Ave6 = 6-hour average, AVE12 = 12-hour average, MAX = maximum 1-hour; MIN = minimum 1-hour. Data were calculated for an inter-quartile range increase of CO_AVE (0.35 mg/m³), CO_AVE12 (0.35 mg/m³), CO_AVE6 (0.40 mg/m³), CO_MIN (0.12 mg/m³), CO_MAX (0.81 mg/m³), NO_AVE (23.8 μg/m³), NO_AVE12 (25.2 μg/m³), NO_AVE6 (30.8 μg/m³), NO_MIN (2.5 μg/m³), NO_MAX (85 μg/m³), NO2_AVE (28.2 μg/m³), NO2_AVE12 (30.1 μg/m³), NO2_AVE6 (34.2 μg/m³), NO2_MIN (16.9 μg/m³), NO2_MAX (39.5 μg/m³), O3_AVE (30 μg/m³), O3_AVE12 (37.7 μg/m³), O3_AVE6 (37.7 μg/m³), O3_MIN (14 μg/m³), O3_MAX (36 μg/m³), PM25_AVE (7.7 μg/m³), PM25_AVE12 (8.5 μg/m³), PM25_AVE6 (8.8 μg/m³), PM25_MIN (3.1 μg/m³), PM25_MAX (17 μg/m³).</p><p>Step 2 multivariate analysis with full adjustment results (p-value ≤0.05).</p

Step 3 Bootstrap model averaging results.

<p>Note: AGECAT1 = age <65; AGECAT2 = age ≥65. NSTEMI = Non-ST Segment Elevation Myocardial Infarction; HTN = Hypertension; Dysrhy = Dysrhythmia; AVE = 24-hour average; AVE6 = 6-hour average, AVE12 = 12-hour average, MAX = maximum 1-hour; MIN = minimum 1-hour. Data were calculated for an inter-quartile range increase of CO_AVE (0.35 mg/m³), CO_AVE12 (0.35 mg/m³), CO_AVE6 (0.40 mg/m³), CO_MIN (0.12 mg/m³), CO_MAX (0.81 mg/m³), NO_AVE (23.8 μg/m³), NO_AVE12 (25.2 μg/m³), NO_AVE6 (30.8 μg/m³), NO_MIN (2.5 μg/m³), NO_MAX (85 μg/m³), NO2_AVE (28.2 μg/m³), NO2_AVE12 (30.1 μg/m³), NO2_AVE6 (34.2 μg/m³), NO2_MIN (16.9 μg/m³), NO2_MAX (39.5 μg/m³), O3_AVE (30 μg/m³), O3_AVE12 (37.7 μg/m³), O3_AVE6 (37.7 μg/m³), O3_MIN (14 μg/m³), O3_MAX (36 μg/m³), PM25_AVE (7.7 μg/m³), PM25_AVE12 (8.5 μg/m³), PM25_AVE6 (8.8 μg/m³), PM25_MIN (3.1 μg/m³), PM25_MAX (17 μg/m³).</p><p>^a Median value from 1,000 model replications.</p><p>^b Number of times that a variable was significant (p-value ≤0.05) from 1,000 model replications.</p><p>Step 3 Bootstrap model averaging results.</p

Seasonal trends of monthly average frequency of AMI hospitalizations and monthly average concentrations of air pollutants (April 1999 –March 2010).

<p>Monthly frequency of AMI hospitalizations (1 unit = 100) were the average number of events by month. Monthly average concentration levels of CO (1 unit = 1 mg/m³), NO (1 unit = 10 μg/m³), NO₂ (1 unit = 10 μg/m³), O₃ (1 unit = 10 μg/m³), or PM_2.5 (1 unit = 10 μg /m³) were averaged by month in which the daily mean concentrations linked to the event days.</p