Predictive verification for the design of partially exchangeable multi-model ensembles

This is the final version. Available on open access from Taylor & Francis via the DOI in this recordThe performance of an ensemble forecast, as measured by scoring rules, depends on its number of members. Under the assumption of ensemble member exchangeability, ensemble-adjusted scores provide unbiased estimates of the ensemble-size effect. In this study, the concept of ensemble-adjusted scores is revisited and exploited in the general context of multi-model ensemble forecasting. In particular, an ensemblesize adjustment is proposed for the continuous ranked probability score in a multi-model ensemble setting. The method requires that the ensemble forecasts satisfy generalized multi-model exchangeability conditions. These conditions do not require the models themselves to be exchangeable. The adjusted scores are tested here on a dual-resolution ensemble, an ensemble which combines members drawn from the same numerical model but run at two different grid resolutions. It is shown that performance of different ensemble combinations can be robustly estimated based on a small subset of members from each model. At no additional cost, the ensemble-size effect is investigated not only considering the pooling of potential extra-members but also including the impact of optimal weighting strategies. With simple and efficient tools, the proposed methodology paves the way for predictive verification of multi-model ensemble forecasts; the derived statistics can provide guidance for the design of future operational ensemble configurations without having to run additional ensemble forecast experiments for all the potential configurations

Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model

Stochastic parameterizations account for uncertainty in the representation of unresolved sub-grid processes by sampling from the distribution of possible sub-grid forcings. Some existing stochastic parameterizations utilize data-driven approaches to characterize uncertainty, but these approaches require significant structural assumptions that can limit their scalability. Machine learning models, including neural networks, are able to represent a wide range of distributions and build optimized mappings between a large number of inputs and sub-grid forcings. Recent research on machine learning parameterizations has focused only on deterministic parameterizations. In this study, we develop a stochastic parameterization using the generative adversarial network (GAN) machine learning framework. The GAN stochastic parameterization is trained and evaluated on output from the Lorenz '96 model, which is a common baseline model for evaluating both parameterization and data assimilation techniques. We evaluate different ways of characterizing the input noise for the model and perform model runs with the GAN parameterization at weather and climate timescales. Some of the GAN configurations perform better than a baseline bespoke parameterization at both timescales, and the networks closely reproduce the spatio-temporal correlations and regimes of the Lorenz '96 system. We also find that in general those models which produce skillful forecasts are also associated with the best climate simulations.Comment: Submitted to Journal of Advances in Modeling Earth Systems (JAMES

Impact of the mesoscale range on error growth and the limits to atmospheric predictability

Global numerical weather prediction (NWP) models have begun to resolve the mesoscale k^{-5/3} range of the energy spectrum, which is known to impose an inherently finite range of deterministic predictability per se as errors develop more rapidly on these scales than on the larger scales. However, the dynamics of these errors under the influence of the synoptic-scale k^{-3} range is little studied. Within a perfect-model context, the present work examines the error growth behavior under such a hybrid spectrum in Lorenz’s original model of 1969, and in a series of identical-twin perturbation experiments using an idealized two-dimensional barotropic turbulence model at a range of resolutions. With the typical resolution of today’s global NWP ensembles, error growth remains largely uniform across scales. The theoretically expected fast error growth characteristic of a k^{-5/3} spectrum is seen to be largely suppressed in the first decade of the mesoscale range by the synoptic-scale k^{-3} range. However, it emerges once models become fully able to resolve features on something like a 20-kilometer scale, which corresponds to a grid resolution on the order of a few kilometers

Parameter variations in prediction skill optimization at ECMWF

Peer reviewe

Working group 3: What are and how do we measure the pros and cons of existing approaches?

WG3 discussed both the pros and cons of existing schemes as well as metrics to measure relative advantages and disadvantages. We first provide a list of the current operational techniques and their respective advantages and disadvantages that were discussed in the WG. We do not claim that the list is complete, and we note that the pros and cons are neither exhaustive nor quantitative. Nevertheless, it may be useful to note the WG’s consensus on the general advantages and disadvantages of the most commonly-used schemes. We then list our recommendations for evaluating model uncertainty schemes. At the end is a short list pertaining to recommendations for further development of methods to represent model uncertainty

Resonant gravity-wave drag enhancement in linear stratified flow over mountains

High-drag states produced in stratified flow over a 2D ridge and an axisymmetric mountain are investigated using a linear, hydrostatic, analytical model. A wind profile is assumed where the background velocity is constant up to a height z1 and then decreases linearly, and the internal gravity-wave solutions are calculated exactly. In flow over a 2D ridge, the normalized surface drag is given by a closed-form analytical expression, while in flow over an axisymmetric mountain it is given by an expression involving a simple 1D integral. The drag is found to depend on two dimensionless parameters: a dimensionless height formed with z_1, and the Richardson number, Ri, in the shear layer. The drag oscillates as z_1 increases, with a period of half the hydrostatic vertical wavelength of the gravity waves. The amplitude of this modulation increases as Ri decreases. This behaviour is due to wave reflection at z_1. Drag maxima correspond to constructive interference of the upward- and downward-propagating waves in the region z < z_1, while drag minima correspond to destructive interference. The reflection coefficient at the interface z = z_1 increases as Ri decreases. The critical level, z_c, plays no role in the drag amplification. A preliminary numerical treatment of nonlinear effects is presented, where z_c appears to become more relevant, and flow over a 2D ridge qualitatively changes its character. But these effects, and their connection with linear theory, still need to be better understood

Evaluating Data Assimilation Algorithms

Data assimilation leads naturally to a Bayesian formulation in which the posterior probability distribution of the system state, given the observations, plays a central conceptual role. The aim of this paper is to use this Bayesian posterior probability distribution as a gold standard against which to evaluate various commonly used data assimilation algorithms. A key aspect of geophysical data assimilation is the high dimensionality and low predictability of the computational model. With this in mind, yet with the goal of allowing an explicit and accurate computation of the posterior distribution, we study the 2D Navier-Stokes equations in a periodic geometry. We compute the posterior probability distribution by state-of-the-art statistical sampling techniques. The commonly used algorithms that we evaluate against this accurate gold standard, as quantified by comparing the relative error in reproducing its moments, are 4DVAR and a variety of sequential filtering approximations based on 3DVAR and on extended and ensemble Kalman filters. The primary conclusions are that: (i) with appropriate parameter choices, approximate filters can perform well in reproducing the mean of the desired probability distribution; (ii) however they typically perform poorly when attempting to reproduce the covariance; (iii) this poor performance is compounded by the need to modify the covariance, in order to induce stability. Thus, whilst filters can be a useful tool in predicting mean behavior, they should be viewed with caution as predictors of uncertainty. These conclusions are intrinsic to the algorithms and will not change if the model complexity is increased, for example by employing a smaller viscosity, or by using a detailed NWP model

Ensemble downscaling in coupled solar wind-magnetosphere modeling for space weather forecasting

Advanced forecasting of space weather requires simulation of the whole Sun-to-Earth system, which necessitates driving magnetospheric models with the outputs from solar wind models. This presents a fundamental difficulty, as the magnetosphere is sensitive to both large-scale solar wind structures, which can be captured by solar wind models, and small-scale solar wind “noise,” which is far below typical solar wind model resolution and results primarily from stochastic processes. Following similar approaches in terrestrial climate modeling, we propose statistical “downscaling” of solar wind model results prior to their use as input to a magnetospheric model. As magnetospheric response can be highly nonlinear, this is preferable to downscaling the results of magnetospheric modeling. To demonstrate the benefit of this approach, we first approximate solar wind model output by smoothing solar wind observations with an 8 h filter, then add small-scale structure back in through the addition of random noise with the observed spectral characteristics. Here we use a very simple parameterization of noise based upon the observed probability distribution functions of solar wind parameters, but more sophisticated methods will be developed in the future. An ensemble of results from the simple downscaling scheme are tested using a model-independent method and shown to add value to the magnetospheric forecast, both improving the best estimate and quantifying the uncertainty. We suggest a number of features desirable in an operational solar wind downscaling scheme

Probabilistic solar wind and geomagnetic forecasting using an analogue ensemble or "Similar Day" approach

Effective space-weather prediction and mitigation requires accurate forecasting of near-Earth solar-wind conditions. Numerical magnetohydrodynamic models of the solar wind, driven by remote solar observations, are gaining skill at forecasting the large-scale solar-wind features that give rise to near-Earth variations over days and weeks. There remains a need for accurate short-term (hours to days) solar-wind forecasts, however. In this study we investigate the analogue ensemble (AnEn), or “similar day”, approach that was developed for atmospheric weather forecasting. The central premise of the AnEn is that past variations that are analogous or similar to current conditions can be used to provide a good estimate of future variations. By considering an ensemble of past analogues, the AnEn forecast is inherently probabilistic and provides a measure of the forecast uncertainty. We show that forecasts of solar-wind speed can be improved by considering both speed and density when determining past analogues, whereas forecasts of the out-of-ecliptic magnetic field [ BNBN ] are improved by also considering the in-ecliptic magnetic-field components. In general, the best forecasts are found by considering only the previous 6 – 12 hours of observations. Using these parameters, the AnEn provides a valuable probabilistic forecast for solar-wind speed, density, and in-ecliptic magnetic field over lead times from a few hours to around four days. For BNBN , which is central to space-weather disturbance, the AnEn only provides a valuable forecast out to around six to seven hours. As the inherent predictability of this parameter is low, this is still likely a marked improvement over other forecast methods. We also investigate the use of the AnEn in forecasting geomagnetic indices Dst and Kp. The AnEn provides a valuable probabilistic forecast of both indices out to around four days. We outline a number of future improvements to AnEn forecasts of near-Earth solar-wind and geomagnetic conditions