Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks

The United States COVID-19 Forecast Hub dataset

Academic researchers, government agencies, industry groups, and individuals have produced forecasts at an unprecedented scale during the COVID-19 pandemic. To leverage these forecasts, the United States Centers for Disease Control and Prevention (CDC) partnered with an academic research lab at the University of Massachusetts Amherst to create the US COVID-19 Forecast Hub. Launched in April 2020, the Forecast Hub is a dataset with point and probabilistic forecasts of incident cases, incident hospitalizations, incident deaths, and cumulative deaths due to COVID-19 at county, state, and national, levels in the United States. Included forecasts represent a variety of modeling approaches, data sources, and assumptions regarding the spread of COVID-19. The goal of this dataset is to establish a standardized and comparable set of short-term forecasts from modeling teams. These data can be used to develop ensemble models, communicate forecasts to the public, create visualizations, compare models, and inform policies regarding COVID-19 mitigation. These open-source data are available via download from GitHub, through an online API, and through R packages

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks

Publisher Correction: Whole-genome sequencing of a sporadic primary immunodeficiency cohort (Nature, (2020), 583, 7814, (90-95), 10.1038/s41586-020-2265-1)

An amendment to this paper has been published and can be accessed via a link at the top of the paper

Health-status outcomes with invasive or conservative care in coronary disease

BACKGROUND In the ISCHEMIA trial, an invasive strategy with angiographic assessment and revascularization did not reduce clinical events among patients with stable ischemic heart disease and moderate or severe ischemia. A secondary objective of the trial was to assess angina-related health status among these patients. METHODS We assessed angina-related symptoms, function, and quality of life with the Seattle Angina Questionnaire (SAQ) at randomization, at months 1.5, 3, and 6, and every 6 months thereafter in participants who had been randomly assigned to an invasive treatment strategy (2295 participants) or a conservative strategy (2322). Mixed-effects cumulative probability models within a Bayesian framework were used to estimate differences between the treatment groups. The primary outcome of this health-status analysis was the SAQ summary score (scores range from 0 to 100, with higher scores indicating better health status). All analyses were performed in the overall population and according to baseline angina frequency. RESULTS At baseline, 35% of patients reported having no angina in the previous month. SAQ summary scores increased in both treatment groups, with increases at 3, 12, and 36 months that were 4.1 points (95% credible interval, 3.2 to 5.0), 4.2 points (95% credible interval, 3.3 to 5.1), and 2.9 points (95% credible interval, 2.2 to 3.7) higher with the invasive strategy than with the conservative strategy. Differences were larger among participants who had more frequent angina at baseline (8.5 vs. 0.1 points at 3 months and 5.3 vs. 1.2 points at 36 months among participants with daily or weekly angina as compared with no angina). CONCLUSIONS In the overall trial population with moderate or severe ischemia, which included 35% of participants without angina at baseline, patients randomly assigned to the invasive strategy had greater improvement in angina-related health status than those assigned to the conservative strategy. The modest mean differences favoring the invasive strategy in the overall group reflected minimal differences among asymptomatic patients and larger differences among patients who had had angina at baseline

Initial invasive or conservative strategy for stable coronary disease

BACKGROUND Among patients with stable coronary disease and moderate or severe ischemia, whether clinical outcomes are better in those who receive an invasive intervention plus medical therapy than in those who receive medical therapy alone is uncertain. METHODS We randomly assigned 5179 patients with moderate or severe ischemia to an initial invasive strategy (angiography and revascularization when feasible) and medical therapy or to an initial conservative strategy of medical therapy alone and angiography if medical therapy failed. The primary outcome was a composite of death from cardiovascular causes, myocardial infarction, or hospitalization for unstable angina, heart failure, or resuscitated cardiac arrest. A key secondary outcome was death from cardiovascular causes or myocardial infarction. RESULTS Over a median of 3.2 years, 318 primary outcome events occurred in the invasive-strategy group and 352 occurred in the conservative-strategy group. At 6 months, the cumulative event rate was 5.3% in the invasive-strategy group and 3.4% in the conservative-strategy group (difference, 1.9 percentage points; 95% confidence interval [CI], 0.8 to 3.0); at 5 years, the cumulative event rate was 16.4% and 18.2%, respectively (difference, 121.8 percentage points; 95% CI, 124.7 to 1.0). Results were similar with respect to the key secondary outcome. The incidence of the primary outcome was sensitive to the definition of myocardial infarction; a secondary analysis yielded more procedural myocardial infarctions of uncertain clinical importance. There were 145 deaths in the invasive-strategy group and 144 deaths in the conservative-strategy group (hazard ratio, 1.05; 95% CI, 0.83 to 1.32). CONCLUSIONS Among patients with stable coronary disease and moderate or severe ischemia, we did not find evidence that an initial invasive strategy, as compared with an initial conservative strategy, reduced the risk of ischemic cardiovascular events or death from any cause over a median of 3.2 years. The trial findings were sensitive to the definition of myocardial infarction that was used