Scoring epidemiological forecasts on transformed scales

Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence

Case studies of personalized learning

Deliverable 4.1, Literature review of personalised learning and the Cloud, started with an evaluation and synthesis of the definitions of personalized learning, followed by an analysis of how this is implemented in a method (e-learning vs. i-learning, m-learning and u-learning), learning approach and the appropriate didactic process, based on adapted didactic theories. From this research a list of criteria was created needed to implement personalised learning onto the learner of the future. This list of criteria is the basis for the analysis of all case studies investigated. – as well to the learning process as the learning place. In total 60 case studies (all 59 case studies mentioned in D6.4 Education on the Cloud 2015 + one extra) were analysed. The case studies were compared with the list of criteria, and a score was calculated. As a result, the best examples could be retained. On average most case studies were good on: taking different learning methods into account, interactivity and accessibility and usability of learning materials for everyone. All had a real formal education content, thus aiming at the core-curriculum, valuing previous knowledge, competences, life and work skills, also informal. Also the availability of an instructor / tutor or other network of peers, experts and teachers to guide and support the learning is common. On the other hand, most case studies lack diagnostics tests as well at the start (diagnostic entry test), during the personalized learning trajectory and at the end (assessment at the end). Also most do not include non-formal and informal learning aspects. And the ownership of personalized learning is not in the hands of the learner. Five of the 60 case studies can as a result be considered as very good examples of real personalized learning

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways

Comparative assessment of methods for short-term forecasts of COVID-19 hospital admissions in England at the local level.

BACKGROUND: Forecasting healthcare demand is essential in epidemic settings, both to inform situational awareness and facilitate resource planning. Ideally, forecasts should be robust across time and locations. During the COVID-19 pandemic in England, it is an ongoing concern that demand for hospital care for COVID-19 patients in England will exceed available resources. METHODS: We made weekly forecasts of daily COVID-19 hospital admissions for National Health Service (NHS) Trusts in England between August 2020 and April 2021 using three disease-agnostic forecasting models: a mean ensemble of autoregressive time series models, a linear regression model with 7-day-lagged local cases as a predictor, and a scaled convolution of local cases and a delay distribution. We compared their point and probabilistic accuracy to a mean-ensemble of them all and to a simple baseline model of no change from the last day of admissions. We measured predictive performance using the weighted interval score (WIS) and considered how this changed in different scenarios (the length of the predictive horizon, the date on which the forecast was made, and by location), as well as how much admissions forecasts improved when future cases were known. RESULTS: All models outperformed the baseline in the majority of scenarios. Forecasting accuracy varied by forecast date and location, depending on the trajectory of the outbreak, and all individual models had instances where they were the top- or bottom-ranked model. Forecasts produced by the mean-ensemble were both the most accurate and most consistently accurate forecasts amongst all the models considered. Forecasting accuracy was improved when using future observed, rather than forecast, cases, especially at longer forecast horizons. CONCLUSIONS: Assuming no change in current admissions is rarely better than including at least a trend. Using confirmed COVID-19 cases as a predictor can improve admissions forecasts in some scenarios, but this is variable and depends on the ability to make consistently good case forecasts. However, ensemble forecasts can make forecasts that make consistently more accurate forecasts across time and locations. Given minimal requirements on data and computation, our admissions forecasting ensemble could be used to anticipate healthcare needs in future epidemic or pandemic settings

Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts.

BACKGROUND: Isolation of cases and contact tracing is used to control outbreaks of infectious diseases, and has been used for coronavirus disease 2019 (COVID-19). Whether this strategy will achieve control depends on characteristics of both the pathogen and the response. Here we use a mathematical model to assess if isolation and contact tracing are able to control onwards transmission from imported cases of COVID-19. METHODS: We developed a stochastic transmission model, parameterised to the COVID-19 outbreak. We used the model to quantify the potential effectiveness of contact tracing and isolation of cases at controlling a severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)-like pathogen. We considered scenarios that varied in the number of initial cases, the basic reproduction number (R0), the delay from symptom onset to isolation, the probability that contacts were traced, the proportion of transmission that occurred before symptom onset, and the proportion of subclinical infections. We assumed isolation prevented all further transmission in the model. Outbreaks were deemed controlled if transmission ended within 12 weeks or before 5000 cases in total. We measured the success of controlling outbreaks using isolation and contact tracing, and quantified the weekly maximum number of cases traced to measure feasibility of public health effort. FINDINGS: Simulated outbreaks starting with five initial cases, an R0 of 1·5, and 0% transmission before symptom onset could be controlled even with low contact tracing probability; however, the probability of controlling an outbreak decreased with the number of initial cases, when R0 was 2·5 or 3·5 and with more transmission before symptom onset. Across different initial numbers of cases, the majority of scenarios with an R0 of 1·5 were controllable with less than 50% of contacts successfully traced. To control the majority of outbreaks, for R0 of 2·5 more than 70% of contacts had to be traced, and for an R0 of 3·5 more than 90% of contacts had to be traced. The delay between symptom onset and isolation had the largest role in determining whether an outbreak was controllable when R0 was 1·5. For R0 values of 2·5 or 3·5, if there were 40 initial cases, contact tracing and isolation were only potentially feasible when less than 1% of transmission occurred before symptom onset. INTERPRETATION: In most scenarios, highly effective contact tracing and case isolation is enough to control a new outbreak of COVID-19 within 3 months. The probability of control decreases with long delays from symptom onset to isolation, fewer cases ascertained by contact tracing, and increasing transmission before symptoms. This model can be modified to reflect updated transmission characteristics and more specific definitions of outbreak control to assess the potential success of local response efforts. FUNDING: Wellcome Trust, Global Challenges Research Fund, and Health Data Research UK

Exploring surveillance data biases when estimating the reproduction number: with insights into subpopulation transmission of COVID-19 in England.

The time-varying reproduction number (Rt: the average number of secondary infections caused by each infected person) may be used to assess changes in transmission potential during an epidemic. While new infections are not usually observed directly, they can be estimated from data. However, data may be delayed and potentially biased. We investigated the sensitivity of Rt estimates to different data sources representing COVID-19 in England, and we explored how this sensitivity could track epidemic dynamics in population sub-groups. We sourced public data on test-positive cases, hospital admissions and deaths with confirmed COVID-19 in seven regions of England over March through August 2020. We estimated Rt using a model that mapped unobserved infections to each data source. We then compared differences in Rt with the demographic and social context of surveillance data over time. Our estimates of transmission potential varied for each data source, with the relative inconsistency of estimates varying across regions and over time. Rt estimates based on hospital admissions and deaths were more spatio-temporally synchronous than when compared to estimates from all test positives. We found these differences may be linked to biased representations of subpopulations in each data source. These included spatially clustered testing, and where outbreaks in hospitals, care homes, and young age groups reflected the link between age and severity of the disease. We highlight that policy makers could better target interventions by considering the source populations of Rt estimates. Further work should clarify the best way to combine and interpret Rt estimates from different data sources based on the desired use. This article is part of the theme issue 'Modelling that shaped the early COVID-19 pandemic response in the UK'

A literature review of personalized learning and the Cloud

In order to provide effective application of the Cloud in education it is essential to know how the learning should and could – if needed – be adapted. In this respect the concept of ‘personalising learning’ is frequently used. But what exactly is personalising learning. And how can it be implemented in using the cloud? The aim of WG3 i-Learner of the School on the Cloud network is to investigate this from the point of view of the learner, whereas WG2 i-Teacher looks on the role of the educators, and WG4 i-Future on the technology. The document has two parts: - The first part starts with an evaluation and synthesis of the definitions of personalized learning (Ch. 3), followed by an analysis of how this is implemented in learning style (e-learning vs. i-learning, m-learning and u-learning, Ch. 4) and learning approach (Ch. 5). To implement this an appropriate pedagogy (Ch. 6) is needed. - The second part is an attempt on how to implement this onto the learner of the future (Ch. 7), as well to the learning process and to the learning place. Recommendations are made in Ch. 8

Comparing human and model-based forecasts of COVID-19 in Germany and Poland.

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways

Implications of the school-household network structure on SARS-CoV-2 transmission under school reopening strategies in England.

In early 2020 many countries closed schools to mitigate the spread of SARS-CoV-2. Since then, governments have sought to relax the closures, engendering a need to understand associated risks. Using address records, we construct a network of schools in England connected through pupils who share households. We evaluate the risk of transmission between schools under different reopening scenarios. We show that whilst reopening select year-groups causes low risk of large-scale transmission, reopening secondary schools could result in outbreaks affecting up to 2.5 million households if unmitigated, highlighting the importance of careful monitoring and within-school infection control to avoid further school closures or other restrictions

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

We compare forecasts of weekly case and death numbers for COVID-19 in Germany and Poland based on 15 different modelling approaches. These cover the period from January to April 2021 and address numbers of cases and deaths one and two weeks into the future, along with the respective uncertainties. We find that combining different forecasts into one forecast can enable better predictions. However, case numbers over longer periods were challenging to predict. Additional data sources, such as information about different versions of the SARS-CoV-2 virus present in the population, might improve forecasts in the future