Presentación realizada en la 3rd European Nowcasting Conference, celebrada en la sede central de AEMET en Madrid del 24 al 26 de abril de 2019

Beusch, L.

Clementi, L.

Foresti, L.

Germann, Urs

Hamann, U.

Hering, Alessandro

Nerini, D.

Nisi, L.

Sassi, M.

Zeder, J.

English

Hamann, Ulrich

Zeder, Joel

Beusch, Lea

Clementi, Lorenzo

Foresti, Loris

Nerini, Daniele

Nisi, Luca

Sassi, Marco

Agencia Estatal de Meteorología

© Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 1 Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology  MeteoSwiss Nowcasting of thunderstorm severity with Machine Learning  in the Alpine Region   Hamann U., Zeder J., Beusch L., Clementi L., Foresti L.,  Hering A., Nerini D., Nisi L., Sassi M., Germann U.  © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 2 COALITION-3 – Faces Marco Sassi NWC-SAF, deployment Ulrich Hamann COALITION project lead Urs Germann, head of MDR Lorenzo Clementi head of MDRD Elena Leonarduzzi Intern in 2016 COALITION-2 Luca Nisi COALITION-1 forecaster Joel Zeder Intern in 2018/2019 COALITION-3 Alessandro Hering TRT developer Loris Foresti precip. Attractor pySTEPS Daniele Nerini pySTEPS Lea Beusch Intern in 2017 Satellite Rainfall © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 3 Starting point before our study  Example for Thunderstorm Radar Tracking (Hering et al)  11 May 2010 for Switzerland. TRT rank expressed in colours.   Thunderstorm cells are  identified by radar with adaptive reflectivity thresholds.  Thunderstorm intensity is  expressed as heuristic  TRT rank (in colours).  Future position is extrapolated  with the current motion.  TRT rank is kept constant.  Multi-sensor cell parameters  are monitored.   Automatic warning suggestions  are generated for warning regions.  Warning suggestions are modified  by forecasters.  MODERATE SEVERE VERY SEVERE © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 4 Cell severity ranking: Single numerical score [𝟎. 𝟎, 𝟒. 𝟎] based on cell attributes integrated with a weighting scheme (fuzzy logic like approach): • Vertical integrated liquid 𝑽𝑰𝑳 • Median of 45 𝑑𝐵𝑍 Echo Top altitude 𝑬𝑻𝟒𝟓 • Maximum cell reflectivity 𝑴𝒂𝒙𝑬𝒄𝒉𝒐 • Area of cell reflectivity ≥ 57 𝑑𝐵𝑍 𝒂𝒓𝒆𝒂𝟓𝟕𝒅𝑩𝒁  TRT rank (heuristic thunderstorm intensity)   𝑹𝑨𝑵𝑲 =  𝟐. 𝟎 ∗ 𝒈(𝑽𝑰𝑳) + 𝟐. 𝟎 ∗ 𝒈(𝑬𝑻𝟒𝟓𝒎𝒆𝒅) + 𝟏. 𝟎 ∗ 𝒈(𝒅𝑩𝒁𝒎𝒂𝒙) + 𝟐. 𝟎 ∗ 𝒈(𝒂𝒓𝒆𝒂𝟓𝟕𝒅𝑩𝒁) 𝟕   Severity RANK DEVELOPING 𝑹𝑨𝑵𝑲 =  [𝟏. 𝟐 − 𝟏. 𝟓[ MODERATE 𝑹𝑨𝑵𝑲 =  [𝟏. 𝟓 − 𝟐. 𝟓[ SEVERE 𝑹𝑨𝑵𝑲 =  [𝟐. 𝟓 − 𝟑. 𝟓[ VERY SEVERE 𝑹𝑨𝑵𝑲 =  [𝟑. 𝟓 − 𝟒. 𝟎] © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 5 Goals for this study (COALITION-3)   • Improved automatic TRT warning suggestions a) thunderstorm position b) thunderstorm intensity      (expressed as TRT rank) • Multi-sensor retrieval (satellite, radar, lightning, COSMO, meta-data)  • Update cycle 5 min • Nowcast up to 45 min • Long warning lead times • Quantitative, customer oriented output © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 6 Flowchart of COALITION-3 © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 7 • Calculating motion vectors with pySTEPS. Feature selection with Shi and Tomasi (1994). Tracking with pyramidal implementation of  Lucas-Kanade (1981) feature tracker algorithm  • Initial cell identification with TRT algorithm. • Track position 45 min backwards in time to retrieve cell history (predictors). • For each predictor following statistics are calculated in a 23 km diameter circle:     • Monitor the thunderstorm intensity (TRT rank) up to 45 min into the future (truth for training)   Retrieving cell histories 23𝑘𝑚  Mean, Sum, Standard Deviation  Minimum and maximum value  1%, 5%, 25% 50%, 75%, 95%, 99% Quantiles  Number of pixels with certain properties,  e.g. precipitation > 0mm/h © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 8 An example of typical cell histories  © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 9 Overall, the final count of features is composed of: • 68 input variables (RZC, IR 10.8 𝜇𝑚, PV at 500 ℎ𝑃𝑎, Slope …) • 10 time steps (𝑡0, 𝑡−5𝑚𝑖𝑛, … , 𝑡−40𝑚𝑖𝑛, 𝑡−45𝑚𝑖𝑛) • 12 Statistics (Sum, Mean, Quantiles, Pixel counts …) plus some conditional statistics of Radar variables • 68x10x12 = 8’160 possible input parameters • Training period summer 2018, about 10’000 observed cell histories Input data - Cardinality                                 RZC_stat|-15|PERC75  RZC_stat|-15|PERC95  RZC_stat|-15|PERC99  RZC_stat|-15|MAX  RZC_stat|-10|SUM    ...  TOPO_ALTITUDE_stat|-20|PERC01 DATE_TRT_ID  201804291235_2018042910550020           -22.248047             0.000000             0.000000          0.000000      -3424.883789    ...                     -11.875000 201804291235_2018042912000024            -1.099609            44.785156            94.321289         87.427734       1311.467773    ...                     -88.125000 201804291235_2018042912000025            -0.245117            -1.833008           -31.992966        -32.000977       -314.942383    ...                      -5.800003 201804291235_2018042912050022            -0.861328             3.979492            14.365040         23.396484        103.207031    ...                     -16.274994 201804291235_2018042912100033            -8.855469           -68.509766           -41.690041          0.000000      -1397.562500    ...                      -0.375000 201804291235_2018042912200024             0.636719             0.706055             1.180468          0.000000        254.544922    ...                       0.000000 201804291235_2018042912250029             0.804688             0.711914            -1.006836          1.375000        180.148438    ...                     -17.625000 ...                                            ...                  ...                  ...               ...               ...    ...                            ... 201809181900_2018091819000008            -0.435547           -10.399414           -14.435349        -13.000000       -731.588867    ...                       8.199997 201809181900_2018091819000033             0.289062             2.375000             6.605469         -2.296875        208.524414    ...                      57.875000  [34 rows x 10128 columns] TRT Cells identified by (unique) ID and Date/Time © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 10 XGBoost model (gradient boosted trees) • roughly equivalent to random forest with an ensemble of weak-learners (trees) as model • but where during training, trees are added until the objective function converges • where the mean squared error MSE is the loss function, and the regularisation term is the sum of scores at the leaves. • XGBoost also produce an estimate how important an input variable is.  • Number of input parameter could  be reduced to 750. Machine learning nowcasting x = cell history y = TRT rank change © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 11 Top 20 features according to XGB model for 𝑡+20𝑚𝑖𝑛 lead time:  Statistical learning – Feature selection © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 12 Relative importance of feature source: Statistical learning – Feature selection © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 13 Model evaluation • XGBoost nowcasts (XGB, circles) always have the smallest RMSD (skilful up to 𝑡+45𝑚𝑖𝑛) • Probability matching (diamonds) is used to correct for the standard deviation (skilful up until 𝑡+35𝑚𝑖𝑛) • Probability matched results have a smaller RMSD as persistence (triangles) for all lead times.        Forecast times with same RMSD Persistence 5 10 15 20 25 XGBoost, PM 6 14 22 30 42 XGBoost 6 18 40 >45 >45 © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 14 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 15 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 16 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 17 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 18 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 19 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 20 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 21 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 22 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 23 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 24 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 25 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 26 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 27 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 28 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 29 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 30 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 31 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 32 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 33 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 34 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 35 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 36 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 37 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 38 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 39 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 40 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 41 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 42 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 43 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 44 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 45 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 46 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 47 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 48 Visualization of the nowcasting results © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 49 • A multi-sensor thunderstorm nowcasting was developed (COALITION-3) • Thunderstorm cell are tracked with motion vectors • Multi-sensor cell history is monitored  • Gradient boosted trees (XGBoost) is used to nowcast thunderstorm intensity • Feature importance ranking enables reduction to 750 predictors • Nowcasted TRT ranks for all forecast times better than persistence • Nowcasted TRT ranks skilful up to 45 min into the future • Probability matching is used to preserve standard deviation • Easily expandable to more input variables • Strait forward to train ML to nowcast other variables, e.g. lighting activity Summary © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 50 Future work COALITION-3 • Improved TRT rank uncertainty estimates  • Feedback from forecasters • Validation of COALITION-3 in comparison to TRT and operational thunderstorm warnings (POD, FAR, SS for different warning levels) • Operationalisation of COALITION-3 - Implement into automatic warning suggestion system of NinJo • Explore applicability for aviation with European coverage   Future work COALITION-4 • Forecast of specific thunderstorm hazards: heavy precipitation, lightning, hail and wind gusts • Improved thunderstorm motion (right movers, topographic steering)  • Exploiting GPU-powered deep learning technology, e.g. convolutional neural networks to improve further the nowcast quality and lead time • Prioritize warning suggestions by risk & uncertainty   Future Outlook © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 51 1. MTG 4D Weather Cube ©EUMETSAT 2. Advanced Machine Learning 3. Thunderstorm Nowcast Radar NWP model Lightning Rain Hail Lightning Wind Future Outlook – Adaptation to MTG © Madrid, European Nowcasting Conference 25.04.2019                Ulrich Hamann, Joel Zeder et al 52 MeteoSvizzera Via ai Monti 146 CH-6605 Locarno-Monti T +41 58 460 92 22 www.meteosvizzera.ch MétéoSuisse 7bis, av. de la Paix CH-1211 Genève 2 T +41 58 460 98 88 www.meteosuisse.ch MétéoSuisse Chemin de l‘Aérologie CH-1530 Payerne T +41 58 460 94 44 www.meteosuisse.ch MeteoSwiss Operation Center 1  CH-8058 Zurich-Airport  T +41 58 460 91 11 www.meteoswiss.ch Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology  MeteoSwiss 

Nowcasting of thunderstorm severity with Machine Learning in the Alpine Region

https://repositorio.aemet.es/bitstream/20.500.11765/10617/1/NTSP5_Hamann_3ENC2019.pdf

Nowcasting of thunderstorm severity with Machine Learning in the Alpine Region

Abstract

Similar works

Full text

Available Versions

Agencia Estatal de Meteorología