Scoring epidemiological forecasts on transformed scales

Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence

Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.

BACKGROUND: Isolation of cases and contact tracing is used to control outbreaks of infectious diseases, and has been used for coronavirus disease 2019 (COVID-19). Whether this strategy will achieve control depends on characteristics of both the pathogen and the response. Here we use a mathematical model to assess if isolation and contact tracing are able to control onwards transmission from imported cases of COVID-19. METHODS: We developed a stochastic transmission model, parameterised to the COVID-19 outbreak. We used the model to quantify the potential effectiveness of contact tracing and isolation of cases at controlling a severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)-like pathogen. We considered scenarios that varied in the number of initial cases, the basic reproduction number (R0), the delay from symptom onset to isolation, the probability that contacts were traced, the proportion of transmission that occurred before symptom onset, and the proportion of subclinical infections. We assumed isolation prevented all further transmission in the model. Outbreaks were deemed controlled if transmission ended within 12 weeks or before 5000 cases in total. We measured the success of controlling outbreaks using isolation and contact tracing, and quantified the weekly maximum number of cases traced to measure feasibility of public health effort. FINDINGS: Simulated outbreaks starting with five initial cases, an R0 of 1·5, and 0% transmission before symptom onset could be controlled even with low contact tracing probability; however, the probability of controlling an outbreak decreased with the number of initial cases, when R0 was 2·5 or 3·5 and with more transmission before symptom onset. Across different initial numbers of cases, the majority of scenarios with an R0 of 1·5 were controllable with less than 50% of contacts successfully traced. To control the majority of outbreaks, for R0 of 2·5 more than 70% of contacts had to be traced, and for an R0 of 3·5 more than 90% of contacts had to be traced. The delay between symptom onset and isolation had the largest role in determining whether an outbreak was controllable when R0 was 1·5. For R0 values of 2·5 or 3·5, if there were 40 initial cases, contact tracing and isolation were only potentially feasible when less than 1% of transmission occurred before symptom onset. INTERPRETATION: In most scenarios, highly effective contact tracing and case isolation is enough to control a new outbreak of COVID-19 within 3 months. The probability of control decreases with long delays from symptom onset to isolation, fewer cases ascertained by contact tracing, and increasing transmission before symptoms. This model can be modified to reflect updated transmission characteristics and more specific definitions of outbreak control to assess the potential success of local response efforts. FUNDING: Wellcome Trust, Global Challenges Research Fund, and Health Data Research UK

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways

Comparing human and model-based forecasts of COVID-19 in Germany and Poland.

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways

Implications of the school-household network structure on SARS-CoV-2 transmission under school reopening strategies in England.

In early 2020 many countries closed schools to mitigate the spread of SARS-CoV-2. Since then, governments have sought to relax the closures, engendering a need to understand associated risks. Using address records, we construct a network of schools in England connected through pupils who share households. We evaluate the risk of transmission between schools under different reopening scenarios. We show that whilst reopening select year-groups causes low risk of large-scale transmission, reopening secondary schools could result in outbreaks affecting up to 2.5 million households if unmitigated, highlighting the importance of careful monitoring and within-school infection control to avoid further school closures or other restrictions

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

We compare forecasts of weekly case and death numbers for COVID-19 in Germany and Poland based on 15 different modelling approaches. These cover the period from January to April 2021 and address numbers of cases and deaths one and two weeks into the future, along with the respective uncertainties. We find that combining different forecasts into one forecast can enable better predictions. However, case numbers over longer periods were challenging to predict. Additional data sources, such as information about different versions of the SARS-CoV-2 virus present in the population, might improve forecasts in the future

Inferring the number of COVID-19 cases from recently reported deaths.

We estimate the number of COVID-19 cases from newly reported deaths in a population without previous reports. Our results suggest that by the time a single death occurs, hundreds to thousands of cases are likely to be present in that population. This suggests containment via contact tracing will be challenging at this point, and other response strategies should be considered. Our approach is implemented in a publicly available, user-friendly, online tool

Estimating the time-varying reproduction number of SARS-CoV-2 using national and subnational case counts

Background: Assessing temporal variations in transmission in different countries is essential for monitoring the epidemic, evaluating the effectiveness of public health interventions and estimating the impact of changes in policy. Methods: We use case and death notification data to generate daily estimates of the time-varying reproduction number globally, regionally, nationally, and subnationally over a 12-week rolling window. Our modelling framework, based on open source tooling, accounts for uncertainty in reporting delays, so that the reproduction number is estimated based on underlying latent infections. Results: Estimates of the reproduction number, trajectories of infections, and forecasts are displayed on a dedicated website as both maps and time series, and made available to download in tabular form. Conclusions:  This decision-support tool can be used to assess changes in virus transmission both globally, regionally, nationally, and subnationally. This allows public health officials and policymakers to track the progress of the outbreak in near real-time using an epidemiologically valid measure. As well as providing regular updates on our website, we also provide an open source tool-set so that our approach can be used directly by researchers and policymakers on confidential data-sets. We hope that our tool will be used to support decisions in countries worldwide throughout the ongoing COVID-19 pandemic.</ns4:p

Global, regional, and national estimates of the population at increased risk of severe COVID-19 due to underlying health conditions in 2020: a modelling study

Background: The risk of severe COVID-19 if an individual becomes infected is known to be higher in older individuals and those with underlying health conditions. Understanding the number of individuals at increased risk of severe COVID-19 and how this varies between countries should inform the design of possible strategies to shield or vaccinate those at highest risk. Methods: We estimated the number of individuals at increased risk of severe disease (defined as those with at least one condition listed as “at increased risk of severe COVID-19” in current guidelines) by age (5-year age groups), sex, and country for 188 countries using prevalence data from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2017 and UN population estimates for 2020. The list of underlying conditions relevant to COVID-19 was determined by mapping the conditions listed in GBD 2017 to those listed in guidelines published by WHO and public health agencies in the UK and the USA. We analysed data from two large multimorbidity studies to determine appropriate adjustment factors for clustering and multimorbidity. To help interpretation of the degree of risk among those at increased risk, we also estimated the number of individuals at high risk (defined as those that would require hospital admission if infected) using age-specific infection–hospitalisation ratios for COVID-19 estimated for mainland China and making adjustments to reflect country-specific differences in the prevalence of underlying conditions and frailty. We assumed males were twice at likely as females to be at high risk. We also calculated the number of individuals without an underlying condition that could be considered at increased risk because of their age, using minimum ages from 50 to 70 years. We generated uncertainty intervals (UIs) for our estimates by running low and high scenarios using the lower and upper 95% confidence limits for country population size, disease prevalences, multimorbidity fractions, and infection–hospitalisation ratios, and plausible low and high estimates for the degree of clustering, informed by multimorbidity studies. Findings: We estimated that 1·7 billion (UI 1·0–2·4) people, comprising 22% (UI 15–28) of the global population, have at least one underlying condition that puts them at increased risk of severe COVID-19 if infected (ranging from &lt;5% of those younger than 20 years to &gt;66% of those aged 70 years or older). We estimated that 349 million (186–787) people (4% [3–9] of the global population) are at high risk of severe COVID-19 and would require hospital admission if infected (ranging from &lt;1% of those younger than 20 years to approximately 20% of those aged 70 years or older). We estimated 6% (3–12) of males to be at high risk compared with 3% (2–7) of females. The share of the population at increased risk was highest in countries with older populations, African countries with high HIV/AIDS prevalence, and small island nations with high diabetes prevalence. Estimates of the number of individuals at increased risk were most sensitive to the prevalence of chronic kidney disease, diabetes, cardiovascular disease, and chronic respiratory disease. Interpretation: About one in five individuals worldwide could be at increased risk of severe COVID-19, should they become infected, due to underlying health conditions, but this risk varies considerably by age. Our estimates are uncertain, and focus on underlying conditions rather than other risk factors such as ethnicity, socioeconomic deprivation, and obesity, but provide a starting point for considering the number of individuals that might need to be shielded or vaccinated as the global pandemic unfolds. Funding: UK Department for International Development, Wellcome Trust, Health Data Research UK, Medical Research Council, and National Institute for Health Research

Practical considerations for measuring the effective reproductive number, Rt.

Estimation of the effective reproductive number Rt is important for detecting changes in disease transmission over time. During the Coronavirus Disease 2019 (COVID-19) pandemic, policy makers and public health officials are using Rt to assess the effectiveness of interventions and to inform policy. However, estimation of Rt from available data presents several challenges, with critical implications for the interpretation of the course of the pandemic. The purpose of this document is to summarize these challenges, illustrate them with examples from synthetic data, and, where possible, make recommendations. For near real-time estimation of Rt, we recommend the approach of Cori and colleagues, which uses data from before time t and empirical estimates of the distribution of time between infections. Methods that require data from after time t, such as Wallinga and Teunis, are conceptually and methodologically less suited for near real-time estimation, but may be appropriate for retrospective analyses of how individuals infected at different time points contributed to the spread. We advise caution when using methods derived from the approach of Bettencourt and Ribeiro, as the resulting Rt estimates may be biased if the underlying structural assumptions are not met. Two key challenges common to all approaches are accurate specification of the generation interval and reconstruction of the time series of new infections from observations occurring long after the moment of transmission. Naive approaches for dealing with observation delays, such as subtracting delays sampled from a distribution, can introduce bias. We provide suggestions for how to mitigate this and other technical challenges and highlight open problems in Rt estimation