Towards nowcasting in Europe in 2030

The increasing impact of severe weather over Europe on lives and weathersensitive economies can be mitigated by accurate 0–6 h forecasts (nowcasts), supporting a vital ‘last line of defence’ for civil protection and many other applications. Recognizing lack of skill in some complex situations, often at convective and local sub-kilometre scales and associated with rare events, we identify seven recommendations with the aim to improve nowcasting in Europe by the national meteorological and hydrological services (NMHSs) by 2030. These recommendations are based on a review of user needs, the state of the observing system, techniques based on observations and high-resolution numerical weather models, as well as tools, data and infrastructure supporting the nowcasting community in Europe. Denser and more accurate observations are necessary particularly in the boundary layer to better characterize the ingredients of severe storms. A key driver for improvement is next-generation European satellite data becoming available as of 2023. Seamless ensemble prediction methods to produce enhanced weather forecasts with 0–24 h lead times and probabilistic products require further development. Such products need to be understood and interpreted by skilled forecasters operating in an evolving forecasting context

Improving the rainfall nowcast for fine temporal and spatial scales suitable for urban hydrology

Accurate Quantitative Precipitation Forecasts (QPF) at high spatial and temporal resolution are crucial for urban flood prediction. Typically, Lagrangian persistence based on radar data is used to nowcast rainfall intensities with up to 3 hours lead time, but nevertheless is not able to deliver reliable QPFs past 20 min lead time (known as well as the predictability limit). Especially, for extreme events causing pluvial floods, accurate QPFs cannot be achieved past 5 min lead time. Furthermore when compared to gauge recordings, the QPFs are not useful at all. There is an essential need to provide better QPFs by improving the rainfall field supplied to the nowcast and by employing non-linear processes for the extrapolation of rainfall into the future. This study is focused on these two main problems, and it investigates different geostatistical and data-driven methods for the improvement of the QPFs at fine scales. The study was conducted within the Hannover radar range where observations between 2000 to 2018 were available. The skill of the nowcast models was assessed on the point (1 km2 and 5 min) and storm scale, based on continuous criteria comparing both radar and gauge observations. A total of 100 gauge measurements inside the study area were as well employed for the assessment. From the period 2000-2012, 93 events of different properties were distinguished and used as a basis for the method development and assessment. Two state-of-the-art nowcast models (HyRaTrac and Lucas-Kanade) were chosen as reference and used as benchmarks for improvement. To improve the rainfall field, a real time merging between radar and gauge data was investigated. Among different merging techniques (mean field bias, quantile bias correction and kriging interpolation), conditional merging (CM) yielded the best rainfall field. When fed to the reference nowcast models, it led to improvements of up to 1 hour of the predictability limit and of the agreement between radar based QPFs and gauge data. To improve the QPF accuracy even further, two different data driven techniques were developed in order to learn non-linear behaviours from past observed rainfall. First, a nearest neighbour approach (k-NN) was developed and employed instead of Lagrangian Persistence on the HyRaTrac nowcast model. The k-NN method accounts for the non-linearity of the storm evolution by consulting k-similar past storms. A deterministic nowcast issued by averaging the behaviours from the 3 most similar storms yielded the best results, extending the predictability limit at the storm scale to 2-3 hours. Second, an ensemble nowcast accounting for the 10 closest neighbours was generated in order to estimate the uncertainty of the QPF. Third, a deep convolution neural network (CNN) was trained on past merged data, in order to learn the non-linearity of the rainfall process. The network based on the last 15 min of observed radar images proved to successfully capture death and decay and partly birth processes, and extended the rainfall predictability limit at the point scale to 3 hours. Lastly, the methods were tested on 17 convective extreme events, extracted from the period 2013-2018, to compare the tested methods for an urban flood nowcast application. The CNN based on merged data outperformed both reference methods as well as the k-NN based nowcast, with the predictability limit reaching 30 – 40 min. The k-NN, although better than the Lagrangian persistence, suffered greatly from the shortcomings of the storm tracking algorithm present under fast moving and extreme storms. To conclude, even though clear improvements were achieved, there is a clear limit to the data-driven methods that cannot be overcome, unless coupled with the convection initialization from Numerical Weather Prediction (NWP) models. Nevertheless, complex relationships learned from past observed data, together with a better rainfall field as input, were proven to be useful in increasing the QPF accuracy and predictability limits for urban hydrology application.Quantitative Niederschlagsvorhersagen (QPF) in hoher räumlicher und zeitlicher Auflösung sind entscheidend für die Prognose urbaner Sturzfluten. Der auf Radardaten basierende Lagrange Ansatz wird typischerweise für Regenintensitätsvorhersagen mit einem Horizont von 3 Stunden verwendet. Zuverlässig ist dieser allerdings nur bis 20 Minuten (bekanntes Prognoselimit). Bei extremen Niederschlagsereignissen, die urbane Sturzfluten verursachen, ist das Limit sogar bereits bei 5 Minuten erreicht. Außerdem kommt es zu deutlichen Abweichungen zwischen der QPF und den Messdaten an Niederschlagsstationen. Eine Verbesserung der QPF ist demnach zwingend erforderlich. Eine solche Verbesserung kann durch die Anpassung des Eingabe-Niederschlagsfeldes und durch die Anwendung nichtlinearer Prozesse für die Extrapolation des Niederschlags erreicht werden. Die vorliegende Studie konzentriert sich auf diese beiden Hauptprobleme und untersucht verschiedene geostatistische und Data-Mining Methoden zur Verbesserung der QPF auf solchen Skalen. Die Studie wurde im Radarbereich von Hannover durchgeführt, wo Beobachtungsdaten von 2000 bis 2018 verfügbar sind. Die Güte der Nowcast-Modelle wurde auf der Punkteskala (1 km2 und 5 min.) anhand kontinuierlicher Kriterien evaluiert und in Relation zu Radar- und Stationsbeobachtungen gesetzt. Hierfür wurden insgesamt 100 Stationsmessungen innerhalb des Untersuchungsgebietes verwendet. Aus dem Zeitraum 2000 bis 2012 wurden 93 Ereignisse mit unterschiedlichen Eigenschaften als Grundlage für die Methodenentwicklung und -beurteilung ausgewertet. Zwei gängige Nowcast-Modelle (HyRaTrac und Lucas-Kanade) wurden als Referenzmodelle ausgewählt und als Maßstab für Verbesserungen eingesetzt. Um das Niederschlagsfeld zu verbessern, wurden Radar- und Stationsdaten in Echtzeit zusammengeführt. Unter den verschiedenen Methoden (Mean Field Bias, Quantile Mapping Bias, Kriging-Interpolation) ergab das Conditional Merging (CM) das optimalste Niederschlagsfeld. Als Input für die beiden Referenzmodelle verwendet, führte das CM zu einer Verlängerung des Prognoselimits auf bis zu eine Stunde. Auch die Übereinstimmung der radargestützten QPF mit den Stationsdaten verbesserte sich. Um das Prognoselimit noch weiter auszudehnen, wurden zwei verschiedene Data-Mining Techniken entwickelt, um die nichtlinearen Verhaltensweisen aus vergangenen Regenfällen zu erlernen: Zunächst wurde ein Nächster-Nachbar-Ansatz (k-NN) entwickelt und anstelle der Lagrange Persistenz im HyRaTrac-Nowcast-Modell eingesetzt. Die k-NN-Methode berücksichtigt die Nichtlinearität der Regensturmentwicklung, indem k-ähnliche vergangene Stürme herangezogen werden. Ein deterministischer Nowcast, der durch Mittelwertbildung der Verhaltensweisen der drei ähnlichsten Stürme erstellt wurde, lieferte die besten Ergebnisse und verlängerte das Prognoselimit auf bis zu zwei-drei Stunden. Ein Ensemble-Nowcast, bei dem die zehn nächsten Nachbarn berücksichtigt wurden, wurde ebenfalls erstellt, um die Unsicherheit des QPF abzuschätzen. Zudem wurde ein künstliches neuronales Netz (CNN) basierend auf vergangenen Daten entwickelt, um die Nichtlinearität des Niederschlagsprozesses zu berücksichtigen. Das neuronale Netz, das mit den beobachteten Radarbildern der letzten 15 Minuten gefüttert wurde, erwies sich als erfolgreich in der Erfassung von Todes-, Zerfalls- und Geburtsprozessen von Stürmen und konnte das Prognoselimit auf bis zu drei Stunden erweitern. Um die Wirksamkeit der entwickelten Methoden für die Vorhersage urbaner Sturzfluten zu untersuchen, wurden sie auf 17 konvektive Extremereignisse aus dem Zeitraum 2013 bis 2018 angewendet. Der k-NN Ansatz war zwar besser als die Lagrange Persistenz, litt aber stark unter den Fehlern des Sturmverfolgungs-Algorithmus bei schnellen und extremen Stürmen. Das CNN übertraf sowohl die Referenzmethoden als auch den k-NN-basierten Nowcast. Das Prognoselimit konnte so von 5 auf 30 bis 40 Minuten erweitert werden. Für eine weitere Verbesserung zeichnete sich letztlich eine klare Grenze ab, die nur mit der Konvektionsinitialisierung aus Numerischen Wettervorhersagemodellen (NWP-Modellen) überwunden werden kann. Im Vergleich mit den ausgewählten Referenzmodellen, können, durch die hier entwickelten Methoden, die Genauigkeit und das Prognoselimit der QPF in der städtischen Hydrologie erheblich verbessert werden

Feature Papers of Forecasting

Nowadays, forecast applications are receiving unprecedent attention thanks to their capability to improve the decision-making processes by providing useful indications. A large number of forecast approaches related to different forecast horizons and to the specific problem that have to be predicted have been proposed in recent scientific literature, from physical models to data-driven statistic and machine learning approaches. In this Special Issue, the most recent and high-quality researches about forecast are collected. A total of nine papers have been selected to represent a wide range of applications, from weather and environmental predictions to economic and management forecasts. Finally, some applications related to the forecasting of the different phases of COVID in Spain and the photovoltaic power production have been presented

Data Analytics for Automated Near Real Time Detection of Blockages in Smart Wastewater Systems

Blockage events account for a substantial portion of the reported failures in the wastewater network, causing flooding, loss of service, environmental pollution and significant clean-up costs. Increasing telemetry in Combined Sewer Overflows (CSOs) provides the opportunity for near real-time data-driven modelling of the sewer network. The research work presented in this thesis describes the development and testing of a novel system, designed for the automatic detection of blockages and other unusual events in near real-time. The methodology utilises an Evolutionary Artificial Neural Network (EANN) model for short term CSO level predictions and Statistical Process Control (SPC) techniques to analyse unusual CSO level behaviour. The system is designed to mimic the work of a trained, experience human technician in determining if a blockage event has occurred. The detection system has been applied to real blockage events from a UK wastewater network. The results obtained illustrate that the methodology can identify different types of blockage events in a reliable and timely manner, and with a low number of false alarms. In addition, a model has been developed for the prediction of water levels in a CSO chamber and the generation of alerts for upcoming spill events. The model consists of a bi-model committee evolutionary artificial neural network (CEANN), composed of two EANN models optimised for wet and dry weather, respectively. The models are combined using a non-linear weighted averaging approach to overcome bias arising from imbalanced data. Both methodologies are designed to be generic and self-learning, thus they can be applied to any CSO location, without requiring input from a human operator. It is envisioned that the technology will allow utilities to respond proactively to developing blockages events, thus reducing potential harm to the sewer network and the surrounding environment

Forecasting of Medium-term Rainfall Using Artificial Neural Networks: Case Studies from Eastern Australia

The advent of machine learning, of which artificial neural networks (ANN) are a component, has provided an opportunity for improved rainfall forecasts, which is of value for water infrastructure management, agriculture, mining and other industries. In this chapter, ANNs are shown to provide more skillful monthly rainfall forecasts for locations in south-eastern Queensland, Australia, for lead-times of 3–12 months. The skill of the forecasts from the ANNs is highest when the models are individually optimized for each month, and when longer-duration series are used as input. The ANN technique has application where there is temperature and rainfall data extending back at least 50 years. Such datasets exist for much of Europe and North America, though a review of the available literature indicates most research into the application of ANN has focused on China, India and Australia

Nowcasting for a high-resolution weather radar network

2010 Fall.Includes bibliographical references.Short-term prediction (nowcasting) of high-impact weather events can lead to significant improvement in warnings and advisories and is of great practical importance. Nowcasting using weather radar reflectivity data has been shown to be particularly useful. The Collaborative Adaptive Sensing of the Atmosphere (CASA) radar network provides high-resolution reflectivity data amenable to producing valuable nowcasts. The high-resolution nature of CASA data requires the use of an efficient nowcasting approach, which necessitated the development of the Dynamic Adaptive Radar Tracking of Storms (DARTS) and sinc kernel-based advection nowcasting methodology. This methodology was implemented operationally in the CASA Distributed Collaborative Adaptive Sensing (DCAS) system in a robust and efficient manner necessitated by the high-resolution nature of CASA data and distributed nature of the environment in which the nowcasting system operates. Nowcasts up to 10 min to support emergency manager decision-making and 1-5 min to steer the CASA radar nodes to better observe the advecting storm patterns for forecasters and researchers are currently provided by this system. Results of nowcasting performance during the 2009 CASA IP experiment are presented. Additionally, currently state-of-the-art scale-based filtering methods were adapted and evaluated for use in the CASA DCAS to provide a scale-based analysis of nowcasting. DARTS was also incorporated in the Weather Support to Deicing Decision Making system to provide more accurate and efficient snow water equivalent nowcasts for aircraft deicing decision support relative to the radar-based nowcasting method currently used in the operational system. Results of an evaluation using data collected from 2007-2008 by the Weather Service Radar-1988 Doppler (WSR-88D) located near Denver, Colorado, and the National Center for Atmospheric Research Marshall Test Site near Boulder, Colorado, are presented. DARTS was also used to study the short-term predictability of precipitation patterns depicted by high-resolution reflectivity data observed at microalpha (0.2-2 km) to mesobeta (20-200 km) scales by the CASA radar network. Additionally, DARTS was used to investigate the performance of nowcasting rainfall fields derived from specific differential phase estimates, which have been shown to provide more accurate and robust rainfall estimates compared to those made from radar reflectivity data

Aircraft Trajectory Planning Considering Ensemble Forecasting of Thunderstorms

Mención Internacional en el título de doctorConvective weather poses a major threat that compromises the safe operation of flights while inducing delay and cost. The aircraft trajectory planning problem under thunderstorm evolution is addressed in this thesis, proposing two novel heuristic approaches that incorporate uncertainties in the evolution of convective cells. In this context, two additional challenges are faced. On the one hand, studies have demonstrated that given the computational power available nowadays, the best way to characterize weather uncertainties is through ensemble forecasting products, hence compatibility with them is crucial. On the other hand, for the algorithms to be used during a flight, they must be fast and deliver results in a few seconds. As a first methodology, three variants of the Scenario-Based Rapidly-Exploring Random Trees (SB-RRTs) are proposed. Each of them builds a tree to explore the free airspace during an iterative and random process. The so-called SB-RRT, the SB-RRT∗ and the Informed SB-RRT∗ find point-to-point safe trajectories by meeting a user-defined safety threshold. Additionally, the last two techniques converge to solutions of minimum flight length. In a second instance, the Augmented Random Search (ARS) algorithm is used to sample trajectories from a directed graph and deform them iteratively in the search for an optimal path. The aim of such deformations is to adapt the initial graph to the unsafe set and its possible changes. In the end, the ARS determines the population of trajectories that, on average, minimizes a combination of flight time, time in storms, and fuel consumption Both methodologies are tested considering a dynamic model of an aircraft flying between two waypoints at a constant flight level. Test scenarios consist of realistic weather forecasts described by an ensemble of equiprobable members. Moreover, the influence of relevant parameters, such as the maximum number of iterations, safety margin (in SB-RRTs) or relative weights between objectives (in ARS) is analyzed. Since both algorithms and their convergence processes are random, sensitivity analyses are conducted to show that after enough iterations the results match. Finally, through parallelization on graphical processing units, the required computational times are reduced substantially to become compatible with near real-time operation. In either case, results show that the suggested approaches are able to avoid dangerous and uncertain stormy regions, minimize objectives such as time of flight, flown distance or fuel consumption and operate in less than 10 seconds.Los fenómenos convectivos representan una gran amenaza que compromete la seguridad de los vuelos, a la vez que incrementa los retrasos y costes. En esta tesis se aborda el problema de la planificación de vuelos bajo la influencia de tormentas, proponiendo dos nuevos métodos heurísticos que incorporan incertidumbre en la evolución de las células convectivas. En este contexto, se intentará dar respuesta a dos desafíos adicionales. Por un lado, hay estudios que demuestran que, con los recursos computacionales disponibles hoy en día, la mejor manera de caracterizar la incertidumbre meteorológica es mediante productos de tipo “ensemble”. Por tanto, la compatibilidad con ellos es crucial. Por otro lado, para poder emplear los algoritmos durante el vuelo, deben de ser rápidos y obtener resultados en pocos segundos. Como primera aproximación, se proponen tres variantes de los “Scenario-Based Rapidly-Exploring Random Trees” (SB-RRTs). Cada uno de ellos crea un árbol que explora el espacio seguro durante un proceso iterativo y aleatorio. Los denominados SB-RRT, SB-RRT∗ e Informed SB-RRT∗ calculan trayectorias entre dos puntos respetando un margen de seguridad impuesto por el usuario. Además, los dos últimos métodos convergen en soluciones de mínima distancia de vuelo. En segundo lugar, el algoritmo “Augmented Random Search” (ARS) se utiliza para muestrear trajectorias de un grafo dirigido y deformarlas iterativamente en busca del camino óptimo. El fin de tales deformaciones es adaptar el grafo inicial a las zonas peligrosas y a los cambios que puedan sufrir. Finalmente, el ARS calcula aquella población de trayectorias que, de media, minimiza una combinación del tiempo de vuelo, el tiempo en zonas tormentosas y el consumo de combustible. Ambas metodologías se testean considerando un modelo de avión volando punto a punto a altitud constante. Los casos de prueba se basan en datos meteorológicos realistas formados por un grupo de predicciones equiprobables. Además, se analiza la influencia de los parámetros más importantes como el máximo número de iteraciones, el margen de seguridad (en SB-RRTs) o los pesos relativos de cada objetivo (en ARS). Como ambos algoritmos y sus procesos de convergencia son aleatorios, se realizan análisis de sensibilidad para mostrar que, tras suficientes iteraciones, los resultados coinciden. Por último, mediante técnicas de paralelización en procesadores gráficos, se reducen enormemente los tiempos de cálculo, siendo compatibles con una operación en tiempo casi-real. En ambos casos los resultados muestran que los algoritmos son capaces de evitar zonas inciertas de tormenta, minimizar objetivos como el tiempo de vuelo, la distancia recorrida o el consumo de combustible, en menos de 10 segundos de ejecución.Programa de Doctorado en Ingeniería Aeroespacial por la Universidad Carlos III de MadridPresidente: Ernesto Staffetti Giammaria.- Secretario: Alfonso Valenzuela Romero.- Vocal: Valentin Polishchu