Detecting improvements in forecast correlation skill: Statistical testing and power analysis

This is the final version. Available from the American Meteorological Society via the DOI in this recordThe skill of weather and climate forecast systems is often assessed by calculating the correlation coefficient between past forecasts and their verifying observations. Improvements in forecast skill can thus be quantified by correlation differences. The uncertainty in the correlation difference needs to be assessed to judge whether the observed difference constitutes a genuine improvement, or is compatible with random sampling variations. A widely used statistical test for correlation difference is known to be unsuitable, because it assumes that the competing forecasting systems are independent. In this paper, appropriate statistical methods are reviewed to assess correlation differences when the competing forecasting systems are strongly correlated with one another. The methods are used to compare correlation skill between seasonal temperature forecasts that differ in initialization scheme and model resolution. A simple power analysis framework is proposed to estimate the probability of correctly detecting skill improvements, and to determine the minimum number of samples required to reliably detect improvements. The proposed statistical test has a higher power of detecting improvements than the traditional test. The main examples suggest that sample sizes of climate hindcasts should be increased to about 40 years to ensure sufficiently high power. It is found that seasonal temperature forecasts are significantly improved by using realistic land surface initial conditions.The authors acknowledge support by the European Union Program FP7/2007-13 under Grant Agreement 3038378 (SPECS). The work of O. Bellprat was funded by ESA under the Climate Change Initiative (CCI) Living Planet Fellowship VERITAS-CCI

Bias adjustment and ensemble recalibration methods for seasonal forecasting: a comprehensive intercomparison using the C3S dataset

This work presents a comprehensive intercomparison of diferent alternatives for the calibration of seasonal forecasts, ranging from simple bias adjustment (BA)-e.g. quantile mapping-to more sophisticated ensemble recalibration (RC) methods- e.g. non-homogeneous Gaussian regression, which build on the temporal correspondence between the climate model and the corresponding observations to generate reliable predictions. To be as critical as possible, we validate the raw model and the calibrated forecasts in terms of a number of metrics which take into account diferent aspects of forecast quality (association, accuracy, discrimination and reliability). We focus on one-month lead forecasts of precipitation and temperature from four state-of-the-art seasonal forecasting systems, three of them included in the Copernicus Climate Change Service dataset (ECMWF-SEAS5, UK Met Ofce-GloSea5 and Météo France-System5) for boreal winter and summer over two illustrative regions with diferent skill characteristics (Europe and Southeast Asia). Our results indicate that both BA and RC methods efectively correct the large raw model biases, which is of paramount importance for users, particularly when directly using the climate model outputs to run impact models, or when computing climate indices depending on absolute values/thresholds. However, except for particular regions and/or seasons (typically with high skill), there is only marginal added value-with respect to the raw model outputs-beyond this bias removal. For those cases, RC methods can outperform BA ones, mostly due to an improvement in reliability. Finally, we also show that whereas an increase in the number of members only modestly afects the results obtained from calibration, longer hindcast periods lead to improved forecast quality, particularly for RC methods.This work has been funded by the C3S activity on Evaluation and Quality Control for seasonal forecasts. JMG was partially supported by the project MULTI-SDM (CGL2015-66583-R, MINECO/FEDER). FJDR was partially funded by the H2020 EUCP project (GA 776613)

Process-conditioned bias correction for seasonal forecasting: a case-study with ENSO in Peru

This work assesses the suitability of a first simple attempt for process-conditioned bias correction in the context of seasonal forecasting. To do this, we focus on the northwestern part of Peru and bias correct 1- and 4-month lead seasonal predictions of boreal winter (DJF) precipitation from the ECMWF System4 forecasting system for the period 1981–2010. In order to include information about the underlying large-scale circulation which may help to discriminate between precipitation affected by different processes, we introduce here an empirical quantile–quantile mapping method which runs conditioned on the state of the Southern Oscillation Index (SOI), which is accurately predicted by System4 and is known to affect the local climate. Beyond the reduction of model biases, our results show that the SOI-conditioned method yields better ROC skill scores and reliability than the raw model output over the entire region of study, whereas the standard unconditioned implementation provides no added value for any of these metrics. This suggests that conditioning the bias correction on simple but well-simulated large-scale processes relevant to the local climate may be a suitable approach for seasonal forecasting. Yet, further research on the suitability of the application of similar approaches to the one considered here for other regions, seasons and/or variables is needed.This work has received funding from the MULTI-SDM project (MINECO/FEDER, CGL2015-66583-R). The authors are grateful to SENAMHI for the observational data, which are publicly available from http://www.senamhi.gob.pe/?p=data-historica, and to the European Center for Medium-Range Weather Forecast (ECMWF), for the access to the System4 seasonal forecasting hindcast

Decadal predictions with the HiGEM high resolution global coupled climate model: description and basic evaluation

This paper describes the development and basic evaluation of decadal predictions produced using the HiGEM coupled climate model. HiGEM is a higher resolution version of the HadGEM1 Met Office Unified Model. The horizontal resolution in HiGEM has been increased to 1.25◦ × 0.83◦ in longitude and latitude for the atmosphere, and 1/3◦ × 1/3◦ globally for the ocean. The HiGEM decadal predictions are initialised using an anomaly assimilation scheme that relaxes anomalies of ocean temperature and salinity to observed anomalies. 10 year hindcasts are produced for 10 start dates (1960, 1965,..., 2000, 2005). To determine the relative contributions to prediction skill from initial conditions and external forcing, the HiGEM decadal predictions are compared to uninitialised HiGEM transient experiments. The HiGEM decadal predictions have substantial skill for predictions of annual mean surface air temperature and 100 m upper ocean temperature. For lead times up to 10 years, anomaly correlations (ACC) over large areas of the North Atlantic Ocean, the Western Pacific Ocean and the Indian Ocean exceed values of 0.6. Initialisation of the HiGEM decadal predictions significantly increases skill over regions of the Atlantic Ocean,the Maritime Continent and regions of the subtropical North and South Pacific Ocean. In particular, HiGEM produces skillful predictions of the North Atlantic subpolar gyre for up to 4 years lead time (with ACC > 0.7), which are significantly larger than the uninitialised HiGEM transient experiments

Climate model forecast biases assessed with a perturbed physics ensemble

Perturbed physics ensembles have often been used to analyse long-timescale climate model behaviour, but have been used less often to study model processes on shorter timescales. We combine a transient perturbed physics ensemble with a set of initialised forecasts to deduce regional process errors present in the standard HadCM3 model, which cause the model to drift in the early stages of the forecast. First, it is shown that the transient drifts in the perturbed physics ensembles can be used to recover quantitatively the parameters that were perturbed. The parameters which exert most influence on the drifts vary regionally, but upper ocean mixing and atmospheric convective processes are particularly important on the 1-month timescale. Drifts in the initialised forecasts are then used to recover the ‘equivalent parameter perturbations’, which allow identification of the physical processes that may be at fault in the HadCM3 representation of the real world. Most parameters show positive and negative adjustments in different regions, indicating that standard HadCM3 values represent a global compromise. The method is verified by correcting an unusually widespread positive bias in the strength of wind-driven ocean mixing, with forecast drifts reduced in a large number of areas as a result. This method could therefore be used to improve the skill of initialised climate model forecasts by reducing model biases through regional adjustments to physical processes, either by tuning or targeted parametrisation refinement. Further, such regionally tuned models might also significantly outperform standard climate models, with global parameter configurations, in longer-term climate studies

An empirical model for probabilistic decadal prediction: global attribution and regional hindcasts

Empirical models, designed to predict surface variables over seasons to decades ahead, provide useful benchmarks for comparison against the performance of dynamical forecast systems; they may also be employable as predictive tools for use by climate services in their own right. A new global empirical decadal prediction system is presented, based on a multiple linear regression approach designed to produce probabilistic output for comparison against dynamical models. A global attribution is performed initially to identify the important forcing and predictor components of the model . Ensemble hindcasts of surface air temperature anomaly fields are then generated, based on the forcings and predictors identified as important, under a series of different prediction ‘modes’ and their performance is evaluated. The modes include a real-time setting, a scenario in which future volcanic forcings are prescribed during the hindcasts, and an approach which exploits knowledge of the forced trend. A two-tier prediction system, which uses knowledge of future sea surface temperatures in the Pacific and Atlantic Oceans, is also tested, but within a perfect knowledge framework. Each mode is designed to identify sources of predictability and uncertainty, as well as investigate different approaches to the design of decadal prediction systems for operational use. It is found that the empirical model shows skill above that of persistence hindcasts for annual means at lead times of up to 10 years ahead in all of the prediction modes investigated. It is suggested that hindcasts which exploit full knowledge of the forced trend due to increasing greenhouse gases throughout the hindcast period can provide more robust estimates of model bias for the calibration of the empirical model in an operational setting. The two-tier system shows potential for improved real-time prediction, given the assumption that skilful predictions of large-scale modes of variability are available. The empirical model framework has been designed with enough flexibility to facilitate further developments, including the prediction of other surface variables and the ability to incorporate additional predictors within the model that are shown to contribute significantly to variability at the local scale. It is also semi-operational in the sense that forecasts have been produced for the coming decade and can be updated when additional data becomes available

Development of a European Multi-Model Ensemble System for Seasonal to Inter-Annual Prediction (DEMETER)

A multi-model ensemble-based system for seasonal-to-interannual prediction has been developed in a joint European project known as DEMETER (Development of a European Multimodel Ensemble Prediction System for Seasonal to Interannual Prediction). The DEMETER system comprises seven global atmosphere–ocean coupled models, each running from an ensemble of initial conditions. Comprehensive hindcast evaluation demonstrates the enhanced reliability and skill of the multimodel ensemble over a more conventional single-model ensemble approach. In addition, innovative examples of the application of seasonal ensemble forecasts in malaria and crop yield prediction are discussed. The strategy followed in DEMETER deals with important problems such as communication across disciplines, downscaling of climate simulations, and use of probabilistic forecast information in the applications sector, illustrating the economic value of seasonal-to-interannual prediction for society as a whole

How to create an operational multi-model of seasonal forecasts?

Seasonal forecasts of variables like near-surface temperature or precipitation are becoming increasingly important for a wide range of stakeholders. Due to the many possibilities of recalibrating, combining, and verifying ensemble forecasts, there are ambiguities of which methods are most suitable. To address this we compare approaches how to process and verify multi-model seasonal forecasts based on a scientific assessment performed within the framework of the EU Copernicus Climate Change Service (C3S) Quality Assurance for Multi-model Seasonal Forecast Products (QA4Seas) contract C3S 51 lot 3. Our results underpin the importance of processing raw ensemble forecasts differently depending on the final forecast product needed. While ensemble forecasts benefit a lot from bias correction using climate conserving recalibration, this is not the case for the intrinsically bias adjusted multi-category probability forecasts. The same applies for multi-model combination. In this paper, we apply simple, but effective, approaches for multi-model combination of both forecast formats. Further, based on existing literature we recommend to use proper scoring rules like a sample version of the continuous ranked probability score and the ranked probability score for the verification of ensemble forecasts and multi-category probability forecasts, respectively. For a detailed global visualization of calibration as well as bias and dispersion errors, using the Chi-square decomposition of rank histograms proved to be appropriate for the analysis performed within QA4Seas.The research leading to these results is part of the Copernicus Climate Change Service (C3S) (Framework Agreement number C3S_51_Lot3_BSC), a program being implemented by the European Centre for Medium-Range Weather Forecasts (ECMWF) on behalf of the European Commission. Francisco Doblas-Reyes acknowledges the support by the H2020 EUCP project (GA 776613) and the MINECO-funded CLINSA project (CGL2017-85791-R)

Predicting the seasonal evolution of southern African summer precipitation in the DePreSys3 prediction system

We assess the ability of the DePreSys3 prediction system to predict austral summer precipitation (DJF) over southern Africa, defined as the African continent south of 15°S. DePresys3 is a high resolution prediction system (at a horizontal resolution of ~ 60 km in the atmosphere in mid-latitudes and of the quarter degree in the Ocean) and spans the long period 1959–2016. We find skill in predicting interannual precipitation variability, relative to a long-term trend; the anomaly correlation skill score over southern Africa is greater than 0.45 for the first summer (i.e. lead month 2–4), and 0.37 over Mozambique, Zimbabwe and Zambia for the second summer (i.e. lead month 14–16). The skill is related to the successful prediction of the El-Nino Southern Oscillation (ENSO), and the successful simulation of ENSO teleconnections to southern Africa. However, overall skill is sensitive to the inclusion of strong La-Nina events and also appears to change with forecast epoch. For example, the skill in predicting precipitation over Mozambique is significantly larger for the first summer in the 1990–2016 period, compared to the 1959–1985 period. The difference in skill in predicting interannual precipitation variability over southern Africa in different epochs is consistent with a change in the strength of the observed teleconnections of ENSO. After 1990, and consistent with the increased skill, the observed impact of ENSO appears to strengthen over west Mozambique, in association with changes in ENSO related atmospheric convergence anomalies. However, these apparent changes in teleconnections are not captured by the ensemble-mean predictions using DePreSys3. The changes in the ENSO teleconnection are consistent with a warming over the Indian Ocean and modulation of ENSO properties between the different epochs, but may also be associated with unpredictable atmospheric variability

A role of the Atlantic Ocean in predicting summer surface air temperature over North East Asia?

We assess the ability of the DePreSys3 prediction system to predict the summer (JJAS) surface-air temperature over North East Asia. DePreSys3 is based on a high resolution ocean–atmosphere coupled climate prediction system (~ 60 km in the atmosphere and ~ 25 km in the ocean), which is full-field initialized from 1960 to 2014 (26 start-dates). We find skill in predicting surface-air temperature, relative to a long-term trend, for 1 and 2–5 year leadtimes over North East Asia, the North Atlantic Ocean and Eastern Europe. DePreSys3 also reproduces the interdecadal evolution of surface-air temperature over the North Atlantic subpolar gyre and North East Asia for both lead times, along with the strong warming that occurred in the mid-1990s over both areas. Composite analysis reveals that the skill at capturing interdecadal changes in North East Asia is associated with the propagation of an atmospheric Rossby wave, which follows the subtropical jet and modulates surface-air temperature from Europe to Eastern Asia. We hypothesise that this ‘circumglobal teleconnection’ pattern is excited over the Atlantic Ocean and is related to Atlantic multi-decadal variability and the associated changes in precipitation over the Sahel and the subtropical Atlantic Ocean. This mechanism is robust for the 2–5 year lead-time. For the 1 year lead-time the Pacific Ocean also plays an important role in leading to skill in predicting SAT over Northeast Asia. Increased temperatures and precipitation over the western Pacific Ocean was found to be associated with a Pacific-Japan like-pattern, which can affect East Asia’s climate