Understanding the performance of flood forecasting models for investment and incident management: Final Report - SC130006/R2.

Abstract

Understanding the performance of flood forecasting models - operated in real-time by the Environment Agency, Natural Resources Wales and the national Flood Forecasting Centre - is crucial to their informed use for flood guidance across England & Wales. It is also essential to guide future strategic investment in flood incident management. This report presents the results from the first nationwide analysis of flood forecasting model performance across implementations by local centres of the National Flood Forecasting System (NFFS). It considers regional and model-type differences and presents an overview of the current forecasting capability of models in operational use. Previous forecast performance studies of local models were performed on a grouped-catchment or regional basis and not necessarily using a consistent assessment framework. Spatial analysis of flood forecasting model performance in this report is based on Wales and the English geographical regions that align to the old Environment Agency region names. The report also extends the forecast model performance information available for the Grid-to-Grid (G2G) model, a distributed grid-based hydrological model with rainfall-runoff and flow routing elements. G2G is implemented within the NFFS for the Flood Forecasting Centre as an area-wide national model across England & Wales. The G2G model forecasts are compared with those from the local models which span a variety of model-types: rainfall-runoff models of conceptual and transfer function form, and channel flow routing models of hydrological and hydrodynamic form. The approach taken to performance assessment has been to gather “raw” data (river flow observations, flow forecasts and historical simulation of flows) from previous local model performance studies. While there are significant regional differences in how these data were gathered and in the methodologies used to generate flood forecast model outputs, collation of the underlying “project” datasets has allowed standardisation of the methods of assessment used here. This report presents the background to a flood forecasting model “Performance Summary”, as a template for reporting performance at any site from a given model-type, including the underlying performance measures employed. The Performance Summary takes the form of one A4 page for each model at each site, and contains a variety of different performance measures and graphical displays. Just under 1,800 Performance Summary pages have been produced for those working in an operational setting or in strategic planning. The results contained in the Performance Summary for each site and model combination have been brought together and used as the basis of a national analysis and summary. This constitutes an extensive national evidence base of model performance, stratified by model-type, model-group, geographical region and lead-time. Where there is a choice of model forecast, it also includes information on comparative performance. Recommendations and important findings • There is a need for standardisation across local flood forecasting model performance (FFMP) methodologies. For example, models should use the same rainfall scenarios and the model output time-interval should be standardised to 15 minutes. • The creation of a national FFMP database, with a well-defined submission format and quality control, is urged. All new FFMP studies should be required to provide the project dataset in this standard format. • There is also a need for models that perform better at longer lead-times, particularly for the South West and parts of the North West where Physically Realisable Transfer Function (PRTF) models are used. • On a regional basis, the North East and North West have the strongest forecasting performance. Anglian, Thames and Southern regions do less well. This demarcation is to be expected because of the much more challenging hydrological conditions (flatter catchments, groundwater, river management) in the south and east of England. • Compounded regional and model differences mean that it is not generally possible to know whether one local model outperforms another. Nevertheless, the best performing models revealed by this study are usually the Probability Distributed Model (PDM), the extended Kinematic Wave (KW) model and the hydrodynamic river model ISIS. • As would be expected, the national G2G model performs less well than local models at a number of sites. It also performs better than some local models and for some regions especially at longer lead-times. • Comparison of G2G with local models on a model-by-model basis shows huge variability. For example, on average PDM models outperform G2G, but there are many sites for which the opposite is true. • Local models require at-site calibration and provide forecasts only for this site when they are of the rainfall-runoff type and for locations along a gauged river reach when of the river model type. G2G provides forecasts everywhere across its model domain (the non-tidal river basins of England and Wales) at a 1km grid resolution. In addition, G2G gives a spatially coherent picture of flood evolution, in contrast to the gauging station specific forecasts produced by local rainfall-runoff models. • The Performance Summary provides, on a single A4 page, a concise summary of model performance along with hydrometric details for a given site and model. It should be made available to operational users of the NFFS via the tooltip functionality of NFFS map displays. • An extensive national evidence base of model performance has been created, stratified by model type, model group, geographical region and lead-time. It also includes information on comparative performance where a choice of model forecast exists. This evidence base is seen as of particular value for strategic planning relating to investment in flood forecasting models. • Accessing, viewing and assessing the wealth of model performance information in appropriate ways is challenging and is dependent on the intended use, for example, in supporting real-time decision-making or in guiding offline strategic investment decisions. Interactive and easily accessible methods of viewing the performance information should be considered, such as that offered by the prototype FFMP web portal being developed by the Centre for Ecology & Hydrology. • The Performance Summary framework was designed to be readily refreshed to include new datasets from consultants as they become available or are commissioned. Recommendations are made to make this process more efficient and the model assessments more meaningful and useful, both in incident management and for guiding strategic investment in flood forecasting models