Hydrologic verification: A call for action and collaboration

Traditionally, little attention has been focused on the systematic verification of operational hydrologic forecasts. This paper summarizes the results of forecasts verification from 15 river basins in the United States. The verification scores for these forecast locations do not show improvement over the periods of record despite a number of forecast process improvements. In considering a root cause for these results, the authors note that the current paradigm for designing hydrologic forecast process improvements is driven by expert opinion and not by objective verification measures. The authors suggest that this paradigm should be modified and objective verification metrics should become the primary driver for hydrologic forecast process improvements. ©2007 American Meteorological Society

Verification of operational solar flare forecast: Case of Regional Warning Center Japan

In this article, we discuss a verification study of an operational solar flare forecast in the Regional Warning Center (RWC) Japan. The RWC Japan has been issuing four-categorical deterministic solar flare forecasts for a long time. In this forecast verification study, we used solar flare forecast data accumulated over 16 years (from 2000 to 2015). We compiled the forecast data together with solar flare data obtained with the Geostationary Operational Environmental Satellites (GOES). Using the compiled data sets, we estimated some conventional scalar verification measures with 95% confidence intervals. We also estimated a multi-categorical scalar verification measure. These scalar verification measures were compared with those obtained by the persistence method and recurrence method. As solar activity varied during the 16 years, we also applied verification analyses to four subsets of forecast-observation pair data with different solar activity levels. We cannot conclude definitely that there are significant performance difference between the forecasts of RWC Japan and the persistence method, although a slightly significant difference is found for some event definitions. We propose to use a scalar verification measure to assess the judgment skill of the operational solar flare forecast. Finally, we propose a verification strategy for deterministic operational solar flare forecasting.Comment: 29 pages, 7 figures and 6 tables. Accepted for publication in Journal of Space Weather and Space Climate (SWSC

Quantile forecast discrimination ability and value

While probabilistic forecast verification for categorical forecasts is well established, some of the existing concepts and methods have not found their equivalent for the case of continuous variables. New tools dedicated to the assessment of forecast discrimination ability and forecast value are introduced here, based on quantile forecasts being the base product for the continuous case (hence in a nonparametric framework). The relative user characteristic (RUC) curve and the quantile value plot allow analysing the performance of a forecast for a specific user in a decision-making framework. The RUC curve is designed as a user-based discrimination tool and the quantile value plot translates forecast discrimination ability in terms of economic value. The relationship between the overall value of a quantile forecast and the respective quantile skill score is also discussed. The application of these new verification approaches and tools is illustrated based on synthetic datasets, as well as for the case of global radiation forecasts from the high resolution ensemble COSMO-DE-EPS of the German Weather Service

Spatial-temporal fractions verification for high-resolution ensemble forecasts

Experiments with two ensemble systems of the resolutions of 10 km (MF10km) and 2 km (MF2km) were designed to examine the value of cloud-resolving ensemble forecast in predicting small spatiotemporal-scale precipitation. Since the verification was performed on short-term precipitation at high resolution, uncertainties from small-scale processes caused the traditional verification methods inconsistent with the subjective evaluation. An extended verification method based on the Fractions Skill Score (FSS) was introduced to account for these uncertainties. The main idea is to extend the concept of spatial neighborhood in FSS to the time and ensemble dimension. The extension was carried out by recognizing that even if ensemble forecast is used, small-scale variability still exists in forecasts and influences verification results. In addition to FSS, the neighborhood concept was also incorporated into reliability diagrams and relative operating characteristics to verify the reliability and resolution of two systems. The extension of FSS in time dimension demonstrates the important role of temporal scales in short-term precipitation verification at small spatial scales. The extension of FSS in ensemble space is called ensemble FSS, which is a good representative of FSS in ensemble forecast in comparison with FSS of ensemble mean. The verification results show that MF2km outperforms MF10km in heavy rain forecasts. In contrast, MF10km was slightly better than MF2km in predicting light rain, suggesting that the horizontal resolution of 2 km is not necessarily enough to completely resolve convective cells

OPERATIONAL VALIDATION AND VERIFICATION OF ALADIN FORECAST IN METEOROLOGICAL AND HYDROLOGICAL SERVICE OF CROATIA

The numerical forecast using ALADIN model in Meteorological and Hydrological Service of Croatia is run operationally since July 2000. Over the years, various methods of validation and verification of the operational forecast have been applied. The classical methods using root mean square error and mean absolute error would often renalize the high resolution ALADIN when compared to a low resolution global model forecast due to double penalty paradigm. Therefore, the model was mostly evaluated by plotting the forecast and the measurements to allow subjective comparison, especially in weather situations that have high impact on the living and traffic conditions in Croatia. Here we show an overview of validation and verification products created operationally. These products intended for subjective validation in real time can help the forecaster in the decision if to rely on a particular forecast run more or less than to another. Statistical verification scores provide information on model bias and root mean square error but suffer from missing data due to automatic procedures used quality check and filtering of the measured data

Verification tools for probabilistic forecasts of continuous hydrological variables

In the present paper we describe some methods for verifying and evaluating probabilistic forecasts of hydrological variables. We propose an extension to continuous-valued variables of a verification method originated in the meteorological literature for the analysis of binary variables, and based on the use of a suitable cost-loss function to evaluate the quality of the forecasts. We find that this procedure is useful and reliable when it is complemented with other verification tools, borrowed from the economic literature, which are addressed to verify the statistical correctness of the probabilistic forecast. We illustrate our findings with a detailed application to the evaluation of probabilistic and deterministic forecasts of hourly discharge value

A Bayesian Hierarchical Approach to Ensemble Weather Forecasting

In meteorology, the traditional approach to forecasting employs deterministic models mimicking atmospheric dynamics. Forecast uncertainty due to the partial knowledge of initial conditions is tackled by Ensemble Predictions Systems (EPS). Probabilistic forecasting is a relatively new approach which may properly account for all sources of uncertainty. In this work we propose a hierarchical Bayesian model which develops this idea and makes it possible to deal with an EPS with non-identifiable members using a suitable definition of the second level of the model. An application to Italian small-scale temperature data is shown.Ensemble Prediction System, hierarchical Bayesian model, predictive distribution, probabilistic forecast, verification rank histogram.

Valuing information from mesoscale forecasts

The development of meso-gamma scale numerical weather prediction (NWP) models requires a substantial investment in research, development and computational resources. Traditional objective verification of deterministic model output fails to demonstrate the added value of high-resolution forecasts made by such models. It is generally accepted from subjective verification that these models nevertheless have a predictive potential for small-scale weather phenomena and extreme weather events. This has prompted an extensive body of research into new verification techniques and scores aimed at developing mesoscale performance measures that objectively demonstrate the return on investment in meso-gamma NWP. In this article it is argued that the evaluation of the information in mesoscale forecasts should be essentially connected to the method that is used to extract this information from the direct model output (DMO). This could be an evaluation by a forecaster, but, given the probabilistic nature of small-scale weather, is more likely a form of statistical post-processing. Using model output statistics (MOS) and traditional verification scores, the potential of this approach is demonstrated both on an educational abstraction and a real world example. The MOS approach for this article incorporates concepts from fuzzy verification. This MOS approach objectively weighs different forecast quality measures and as such it is an essential extension of fuzzy methods

Calibration of probabilistic quantitative precipitation forecasts with an artificial neural network

A feed-forward neural network is configured to calibrate the bias of a high-resolution probabilistic quantitative precipitation forecast (PQPF) produced by a 12-km version of the NCEP Regional Spectral Model (RSM) ensemble forecast system. Twice-daily forecasts during the 2002-2003 cool season (1 November-31 March, inclusive) are run over four U.S. Geological Survey (USGS) hydrologic unit regions of the southwest United States. Calibration is performed via a cross-validation procedure, where four months are used for training and the excluded month is used for testing. The PQPFs before and after the calibration over a hydrological unit region are evaluated by comparing the joint probability distribution of forecasts and observations. Verification is performed on the 4-km stage IV grid, which is used as "truth." The calibration procedure improves the Brier score (BrS), conditional bias (reliability) and forecast skill, such as the Brier skill score (BrSS) and the ranked probability skill score (RPSS), relative to the sample frequency for all geographic regions and most precipitation thresholds. However, the procedure degrades the resolution of the PQPFs by systematically producing more forecasts with low nonzero forecast probabilities that drive the forecast distribution closer to the climatology of the training sample. The problem of degrading the resolution is most severe over the Colorado River basin and the Great Basin for relatively high precipitation thresholds where the sample of observed events is relatively small. © 2007 American Meteorological Society