On the use of multi-model ensemble techniques for ionospheric and thermospheric characterisation

Space weather can have a negative impact on a number of radio frequency (RF) systems, with mitigation by ionospheric and thermospheric modelling one approach to improving system performance. However, before a model can be adopted operationally its performance must be quantified. Taylor diagrams, which show a model’s standard deviation and correlation, have been extended to further illustrate the model’s bias, standard deviation of error and mean square error in comparison to observational data. By normalising the statistics, multiple parameters can be shown simultaneously for a number of models. Using these modified Taylor diagrams, the first known long term (one month) comparison of three model types – empirical, physics and data assimilation - has been performed. The data assimilation models performed best, offering a statistically significant improvement in performance. One physics model performed sufficiently well that it is a viable background model option in future data assimilation schemes. Finally, multi-model thermospheric ensembles (MMEs) have been constructed from which the thermospheric forecasts exhibited a reduced root mean square error compared to non-ensemble approaches. Using an equally weighted MME the reduction was 55% and using a mean square error weighted approach the reduction was 48%

Bootstrapping to Assess and Improve Atmospheric Prediction Models

Bootstrapping is a simple technique typically used to assess accuracy of estimates of model parameters by using simple plug-in principles and replacing sometimes unwieldy theory by computer simulation. Common uses include variance estimation and confidence interval construction of model parameters. It also provides a way to estimate prediction accuracy of continuous and class-valued outcomes regression models. In this paper we will overview some of these applications of the bootstrap focusing on bootstrap estimates of prediction error, and also explore how the bootstrap can be used to improve prediction accuracy of unstable models like tree-structured classifiers through aggregation. The improvements can typically be attributed to variance reduction in the classical regression setting and more generally a smoothing of decision boundaries for the classification setting. These advancements have important implications in the way that atmospheric prediction models can be improved, and illustrations of this will be shown. For class-valued outcomes, an interesting graphic known as the CAT scan can be constructed to help understand the aggregated decision boundary. This will be illustrated using simulated data