Detecting improvements in forecast correlation skill: Statistical testing and power analysis

This is the final version. Available from the American Meteorological Society via the DOI in this recordThe skill of weather and climate forecast systems is often assessed by calculating the correlation coefficient between past forecasts and their verifying observations. Improvements in forecast skill can thus be quantified by correlation differences. The uncertainty in the correlation difference needs to be assessed to judge whether the observed difference constitutes a genuine improvement, or is compatible with random sampling variations. A widely used statistical test for correlation difference is known to be unsuitable, because it assumes that the competing forecasting systems are independent. In this paper, appropriate statistical methods are reviewed to assess correlation differences when the competing forecasting systems are strongly correlated with one another. The methods are used to compare correlation skill between seasonal temperature forecasts that differ in initialization scheme and model resolution. A simple power analysis framework is proposed to estimate the probability of correctly detecting skill improvements, and to determine the minimum number of samples required to reliably detect improvements. The proposed statistical test has a higher power of detecting improvements than the traditional test. The main examples suggest that sample sizes of climate hindcasts should be increased to about 40 years to ensure sufficiently high power. It is found that seasonal temperature forecasts are significantly improved by using realistic land surface initial conditions.The authors acknowledge support by the European Union Program FP7/2007-13 under Grant Agreement 3038378 (SPECS). The work of O. Bellprat was funded by ESA under the Climate Change Initiative (CCI) Living Planet Fellowship VERITAS-CCI

Unusual past dry and wet rainy seasons over Southern Africa and South America from a climate perspective

Southern Africa and Southern South America have experienced recent extremes in dry and wet rainy seasons which have caused severe socio-economic damages. Selected past extreme events are here studied, to estimate how human activity has changed the risk of the occurrence of such events, by applying an event attribution approach (Stott et al., 2004)comprising global climate models of Coupled Model Intercomparison Project 5 (CMIP5). Our assessment shows that models' representation of mean precipitation variability over Southern South America is not adequate to make a robust attribution statement about seasonal rainfall extremes in this region. Over Southern Africa, we show that unusually dry austral summers as occurred during 2002/2003 have become more likely, whereas unusually wet austral summers like that of 1999/2000 have become less likely due to anthropogenic climate change. There is some tentative evidence that the risk of extreme high 5-day precipitation totals (as observed in 1999/2000) have increased in the region. These results are consistent with CMIP5 models projecting a general drying trend over SAF during December–January–February (DJF) but also an increase in atmospheric moisture availability to feed heavy rainfall events when they do occur. Bootstrapping the confidence intervals of the fraction of attributable risk has demonstrated estimates of attributable risk are very uncertain, if the events are very rare. The study highlights some of the challenges in making an event attribution study for precipitation using seasonal precipitation and extreme 5-day precipitation totals and considering natural drivers such as ENSO in coupled ocean–atmosphere models

Evaluation of the HadGEM3-A simulations in view of detection and attribution of human influence on extreme events in Europe

A detailed analysis is carried out to assess the HadGEM3-A global atmospheric model skill in simulating extreme temperatures, precipitation and storm surges in Europe in the view of their attribution to human influence. The analysis is performed based on an ensemble of 15 atmospheric simulations forced with observed Sea Surface Temperature of the 54 year period 1960-2013. These simulations, together with dual simulations without human influence in the forcing, are intended to be used in weather and climate event attribution. The analysis investigates the main processes leading to extreme events, including atmospheric circulation patterns, their links with temperature extremes, land-atmosphere and troposphere-stratosphere interactions. It also compares observed and simulated variability, trends and generalized extreme value theory parameters for temperature and precipitation. One of the most striking findings is the ability of the model to capture North Atlantic atmospheric weather regimes as obtained from a cluster analysis of sea level pressure fields. The model also reproduces the main observed weather patterns responsible for temperature and precipitation extreme events. However, biases are found in many physical processes. Slightly excessive drying may be the cause of an overestimated summer interannual variability and too intense heat waves, especially in central/northern Europe. However, this does not seem to hinder proper simulation of summer temperature trends. Cold extremes appear well simulated, as well as the underlying blocking frequency and stratosphere-troposphere interactions. Extreme precipitation amounts are overestimated and too variable. The atmospheric conditions leading to storm surges were also examined in the Baltics region. There, simulated weather conditions appear not to be leading to strong enough storm surges, but winds were found in very good agreement with reanalyses. The performance in reproducing atmospheric weather patterns indicates that biases mainly originate from local and regional physical processes. This makes local bias adjustment meaningful for climate change attribution

Process-conditioned bias correction for seasonal forecasting: a case-study with ENSO in Peru

This work assesses the suitability of a first simple attempt for process-conditioned bias correction in the context of seasonal forecasting. To do this, we focus on the northwestern part of Peru and bias correct 1- and 4-month lead seasonal predictions of boreal winter (DJF) precipitation from the ECMWF System4 forecasting system for the period 1981–2010. In order to include information about the underlying large-scale circulation which may help to discriminate between precipitation affected by different processes, we introduce here an empirical quantile–quantile mapping method which runs conditioned on the state of the Southern Oscillation Index (SOI), which is accurately predicted by System4 and is known to affect the local climate. Beyond the reduction of model biases, our results show that the SOI-conditioned method yields better ROC skill scores and reliability than the raw model output over the entire region of study, whereas the standard unconditioned implementation provides no added value for any of these metrics. This suggests that conditioning the bias correction on simple but well-simulated large-scale processes relevant to the local climate may be a suitable approach for seasonal forecasting. Yet, further research on the suitability of the application of similar approaches to the one considered here for other regions, seasons and/or variables is needed.This work has received funding from the MULTI-SDM project (MINECO/FEDER, CGL2015-66583-R). The authors are grateful to SENAMHI for the observational data, which are publicly available from http://www.senamhi.gob.pe/?p=data-historica, and to the European Center for Medium-Range Weather Forecast (ECMWF), for the access to the System4 seasonal forecasting hindcast

Earth System Model Evaluation Tool (ESMValTool) v2.0 - An extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP

The Earth System Model Evaluation Tool (ESMValTool) is a community diagnostics and performance metrics tool designed to improve comprehensive and routine evaluation of Earth system models (ESMs) participating in the Coupled Model Intercomparison Project (CMIP). It has undergone rapid development since the first release in 2016 and is now a well-tested tool that provides end-to-end provenance tracking to ensure reproducibility. It consists of (1) an easy-to-install, well-documented Python package providing the core functionalities (ESMValCore) that performs common preprocessing operations and (2) a diagnostic part that includes tailored diagnostics and performance metrics for specific scientific applications. Here we describe large-scale diagnostics of the second major release of the tool that supports the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). ESMValTool v2.0 includes a large collection of diagnostics and performance metrics for atmospheric, oceanic, and terrestrial variables for the mean state, trends, and variability. ESMValTool v2.0 also successfully reproduces figures from the evaluation and projections chapters of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and incorporates updates from targeted analysis packages, such as the NCAR Climate Variability Diagnostics Package for the evaluation of modes of variability, the Thermodynamic Diagnostic Tool (TheDiaTo) to evaluate the energetics of the climate system, as well as parts of AutoAssess that contains a mix of top-down performance metrics. The tool has been fully integrated into the Earth System Grid Federation (ESGF) infrastructure at the Deutsches Klimarechenzentrum (DKRZ) to provide evaluation results from CMIP6 model simulations shortly after the output is published to the CMIP archive. A result browser has been implemented that enables advanced monitoring of the evaluation results by a broad user community at much faster timescales than what was possible in CMIP5

Multiple perspectives on the attribution of the extreme European summer of 2012 to climate change

Summer 2012 was very wet in northern Europe, and unusually dry and hot in southern Europe. We use multiple approaches to determine whether anthropogenic forcing made the extreme European summer of 2012 more likely. Using a number of observation- and model-based methods, we find that there was an anthropogenic contribution to the extremes in southern Europe, with a qualitative consensus across all methodologies. There was a consensus across the methodologies that there has been a significant increase in the risk of hot summers in southern Europe with climate change. Most approaches also suggested a slight drying, but none of the results were statistically significant. The unusually wet summer in northern Europe was made more likely by the observed atmospheric circulation pattern in 2012, but no evidence was found for a long-term trend in circulation

Daily precipitation statistics in a EURO-CORDEX RCM ensemble: added value of raw and bias-corrected high-resolution simulations

Daily precipitation statistics as simulated by the ERA-Interim-driven EURO-CORDEX regional climate model (RCM) ensemble are evaluated over two distinct regions of the European continent, namely the European Alps and Spain. The potential added value of the high-resolution 12 km experiments with respect to their 50 km resolution counterparts is investigated. The statistics considered consist of wet-day intensity and precipitation frequency as a measure of mean precipitation, and three precipitation-derived indicators (90th percentile on wet days?90pWET, contribution of the very wet days to total precipitation?R95pTOT and number of consecutive dry days?CDD). As reference for model evaluation high resolution gridded observational data over continental Spain (Spain011/044) and the Alpine region (EURO4M-APGD) are used. The assessment and comparison of the two resolutions is accomplished not only on their original horizontal grids (approximately 12 and 50 km), but the high-resolution RCMs are additionally regridded onto the coarse 50 km grid by grid cell aggregation for the direct comparison with the low resolution simulations. The direct application of RCMs e.g. in many impact modelling studies is hampered by model biases. Therefore bias correction (BC) techniques are needed at both resolutions to ensure a better agreement between models and observations. In this work, the added value of the high resolution (before and after the bias correction) is assessed and the suitability of these BC methods is also discussed. Three basic BC methods are applied to isolate the effect of biases in mean precipitation, wet-day intensity and wet-day frequency on the derived indicators. Daily precipitation percentiles are strongly affected by biases in the wet-day intensity, whereas the dry spells are better represented when the simulated precipitation frequency is adjusted to the observed one. This confirms that there is no single optimal way to correct for RCM biases, since correcting some distributional features typically leads to an improvement of some aspects but to a deterioration of others. Regarding mean seasonal biases before the BC, we find only limited evidence for an added value of the higher resolution in the precipitation intensity and frequency or in the derived indicators. Thereby, evaluation results considerably depend on the RCM, season and indicator considered. High resolution simulations better reproduce the indicators? spatial patterns, especially in terms of spatial correlation. However, this improvement is not statistically significant after applying specific BC methods.The authors are grateful to Prof. C. Schär for his helpful comments and E. van Meijgaard for making available the RACMO model data. We acknowledge the observational data providers. Calculations for WRF311F were made using the TGCC super computers under the GENCI time allocation GEN6877. The WRF331A from CRP-GL (now LIST) was funded by the Luxembourg National Research Fund (FNR) through grant FNR C09/SR/16 (CLIMPACT). The KNMI-RACMO2 simulations were supported by the Dutch Ministry of Infrastructure and the Environment. The CCLM and REMO simulations were supported by the Federal Ministry of Education and Research (BMBF) and performed under the Konsortial share at the German Climate Computing Centre (DKRZ). The CCLM simulations were furthermore supported by the Swiss National Supercomputing Centre (CSCS) under project ID s78. Part of the SMHI contribution was carried out in the Swedish Mistra-SWECIA programme founded by Mistra (the Foundation for Strategic Environmental Research). This work is supported by CORWES (CGL2010-22158-C02) and EXTREMBLES (CGL2010-21869) projects funded by the Spanish R&D programme and the European COST ACTION VALUE (ES1102). A. C. thanks the Spanish Ministry of Economy and Competitiveness for the funding provided within the FPI programme (BES-2011-047612 and EEBB-I-13-06354). We also thank two anonymous referees for their useful comments that helped to improve the original manuscript

Using climate models to estimate the quality of global observational data sets

International audienceObservational estimates of the climate system are essential to monitoring and understanding ongoing climate change and to assessing the quality of climate models used to produce near- and long-term climate information. This study poses the dual and unconventional question: Can climate models be used to assess the quality of observational references? We show that this question not only rests on solid theoretical grounds but also offers insightful applications in practice. By comparing four observational products of sea surface temperature with a large multimodel climate forecast ensemble, we find compelling evidence that models systematically score better against the most recent, advanced, but also most independent product. These results call for generalized procedures of model-observation comparison and provide guidance for a more objective observational data set selection