Prediction of severe thunderstorm events with ensemble deep learning and radar data

The problem of nowcasting extreme weather events can be addressed by applying either numerical methods for the solution of dynamic model equations or data-driven artificial intelligence algorithms. Within this latter framework, the most used techniques rely on video prediction deep learning methods which take in input time series of radar reflectivity images to predict the next future sequence of reflectivity images, from which the predicted rainfall quantities are extrapolated. Differently from the previous works, the present paper proposes a deep learning method, exploiting videos of radar reflectivity frames as input and lightning data to realize a warning machine able to sound timely alarms of possible severe thunderstorm events. The problem is recast in a classification one in which the extreme events to be predicted are characterized by a an high level of precipitation and lightning density. From a technical viewpoint, the computational core of this approach is an ensemble learning method based on the recently introduced value-weighted skill scores for both transforming the probabilistic outcomes of the neural network into binary predictions and assessing the forecasting performance. Such value-weighted skill scores are particularly suitable for binary predictions performed over time since they take into account the time evolution of events and predictions paying attention to the value of the prediction for the forecaster. The result of this study is a warning machine validated against weather radar data recorded in the Liguria region, in Italy

Change Detection Techniques with Synthetic Aperture Radar Images: Experiments with Random Forests and Sentinel-1 Observations

This work aims to clarify the potential of incoherent and coherent change detection (CD) approaches for detecting and monitoring ground surface changes using sequences of synthetic aperture radar (SAR) images. Nowadays, the growing availability of remotely sensed data collected by the twin Sentinel-1A/B sensors of the European (EU) Copernicus constellation allows fast mapping of damage after a disastrous event using radar data. In this research, we address the role of SAR (amplitude) backscattered signal variations for CD analyses when a natural (e.g., a fire, a flash flood, etc.) or a human-induced (disastrous) event occurs. Then, we consider the additional pieces of information that can be recovered by comparing interferometric coherence maps related to couples of SAR images collected between a principal disastrous event date. This work is mainly concerned with investigating the capability of different coherent/incoherent change detection indices (CDIs) and their mutual interactions for the rapid mapping of "changed" areas. In this context, artificial intelligence (AI) algorithms have been demonstrated to be beneficial for handling the different information coming from coherent/incoherent CDIs in a unique corpus. Specifically, we used CDIs that synthetically describe ground surface changes associated with a disaster event (i.e., the pre-, cross-, and post-disaster phases), based on the generation of sigma nought and InSAR coherence maps. Then, we trained a random forest (RF) to produce CD maps and study the impact on the final binary decision (changed/unchanged) of the different layers representing the available synthetic CDIs. The proposed strategy was effective for quickly assessing damage using SAR data and can be applied in several contexts. Experiments were conducted to monitor wildfire's effects in the 2021 summer season in Italy, considering two case studies in Sardinia and Sicily. Another experiment was also carried out on the coastal city of Houston, Texas, the US, which was affected by a large flood in 2017; thus, demonstrating the validity of the proposed integrated method for fast mapping of flooded zones using SAR data

Changepoint detection for data intensive settings

Detecting a point in a data sequence where the behaviour alters abruptly, otherwise known as a changepoint, has been an active area of interest for decades. More recently, with the advent of the data intensive era, the need for automated and computationally efficient changepoint methods has grown. We here introduce several new techniques for doing this which address many of the issues inherent in detecting changes in a streaming setting. In short, these new methods, which may be viewed as non-trivial extensions of existing classical procedures, are intended to be as useful in as wide a set of situations as possible, while retaining important theoretical guarantees and ease of implementation. The first novel contribution concerns two methods for parallelising existing dynamic programming based approaches to changepoint detection in the single variate setting. We demonstrate that these methods can result in near quadratic computational gains, while retaining important theoretical guarantees. Our next area of focus is the multivariate setting. We introduce two new methods for data intensive scenarios with a fixed, but possibly large, number of dimensions. The first of these is an offline method which detects one change at a time using a new test statistic. We demonstrate that this test statistic has competitive power in a variety of possible settings for a given changepoint, while allowing the method to be versatile across a range of possible modelling assumptions. The other method we introduce for multivariate data is also suitable in the streaming setting. In addition, it is able to relax many standard modelling assumptions. We discuss the empirical properties of the procedure, especially insofar as they relate to a desired false alarm error rate

Causal Discovery from Temporal Data: An Overview and New Perspectives

Temporal data, representing chronological observations of complex systems, has always been a typical data structure that can be widely generated by many domains, such as industry, medicine and finance. Analyzing this type of data is extremely valuable for various applications. Thus, different temporal data analysis tasks, eg, classification, clustering and prediction, have been proposed in the past decades. Among them, causal discovery, learning the causal relations from temporal data, is considered an interesting yet critical task and has attracted much research attention. Existing casual discovery works can be divided into two highly correlated categories according to whether the temporal data is calibrated, ie, multivariate time series casual discovery, and event sequence casual discovery. However, most previous surveys are only focused on the time series casual discovery and ignore the second category. In this paper, we specify the correlation between the two categories and provide a systematical overview of existing solutions. Furthermore, we provide public datasets, evaluation metrics and new perspectives for temporal data casual discovery.Comment: 52 pages, 6 figure

Sistemas de alarme ótimos e sua aplicação a séries financeiras

Doutoramento em MatemáticaThis thesis focuses on the application of optimal alarm systems to non linear time series models. The most common classes of models in the analysis of real-valued and integer-valued time series are described. The construction of optimal alarm systems is covered and its applications explored. Considering models with conditional heteroscedasticity, particular attention is given to the Fractionally Integrated Asymmetric Power ARCH, FIAPARCH(p; d; q) model and an optimal alarm system is implemented, following both classical and Bayesian methodologies. Taking into consideration the particular characteristics of the APARCH(p; q) representation for financial time series, the introduction of a possible counterpart for modelling time series of counts is proposed: the INteger-valued Asymmetric Power ARCH, INAPARCH(p; q). The probabilistic properties of the INAPARCH(1; 1) model are comprehensively studied, the conditional maximum likelihood (ML) estimation method is applied and the asymptotic properties of the conditional ML estimator are obtained. The final part of the work consists on the implementation of an optimal alarm system to the INAPARCH(1; 1) model. An application is presented to real data series.Esta tese centra-se na aplicação de sistemas de alarme ótimos a modelos de séries temporais não lineares. As classes de modelos mais comuns na análise de séries temporais de valores reais e de valores inteiros são descritas com alguma profundidade. É abordada a construção de sistemas de alarme ótimos e as suas aplicações são exploradas. De entre os modelos com heterocedasticidade condicional é dada especial atenção ao modelo ARCH Fraccionalmente Integrável de Potência Assimétrica, FIAPARCH(p; d; q), e é feita a implementação de um sistema de alarme ótimo, considerando ambas as metodologias clássica e Bayesiana. Tomando em consideração as características particulares do modelo APARCH(p; q) na aplicação a séries de dados financeiros, é proposta a introdução do seu homólogo para a modelação de séries temporais de contagens: o modelo ARCH de valores INteiros e Potência Assimétrica, INAPARCH(p; q). As propriedades probabilísticas do modelo INAPARCH(1; 1) são extensivamente estudadas, é aplicado o método da máxima verosimilhança (MV) condicional para a estimação dos parâmetros do modelo e estudadas as propriedades assintóticas do estimador de MV condicional. Na parte final do trabalho é feita a implementação de um sistema de alarme ótimo ao modelo INAPARCH(1; 1) e apresenta-se uma aplicação a séries de dados reais

Context Exploitation in Data Fusion

Complex and dynamic environments constitute a challenge for existing tracking algorithms. For this reason, modern solutions are trying to utilize any available information which could help to constrain, improve or explain the measurements. So called Context Information (CI) is understood as information that surrounds an element of interest, whose knowledge may help understanding the (estimated) situation and also in reacting to that situation. However, context discovery and exploitation are still largely unexplored research topics. Until now, the context has been extensively exploited as a parameter in system and measurement models which led to the development of numerous approaches for the linear or non-linear constrained estimation and target tracking. More specifically, the spatial or static context is the most common source of the ambient information, i.e. features, utilized for recursive enhancement of the state variables either in the prediction or the measurement update of the filters. In the case of multiple model estimators, context can not only be related to the state but also to a certain mode of the filter. Common practice for multiple model scenarios is to represent states and context as a joint distribution of Gaussian mixtures. These approaches are commonly referred as the join tracking and classification. Alternatively, the usefulness of context was also demonstrated in aiding the measurement data association. Process of formulating a hypothesis, which assigns a particular measurement to the track, is traditionally governed by the empirical knowledge of the noise characteristics of sensors and operating environment, i.e. probability of detection, false alarm, clutter noise, which can be further enhanced by conditioning on context. We believe that interactions between the environment and the object could be classified into actions, activities and intents, and formed into structured graphs with contextual links translated into arcs. By learning the environment model we will be able to make prediction on the target\u2019s future actions based on its past observation. Probability of target future action could be utilized in the fusion process to adjust tracker confidence on measurements. By incorporating contextual knowledge of the environment, in the form of a likelihood function, in the filter measurement update step, we have been able to reduce uncertainties of the tracking solution and improve the consistency of the track. The promising results demonstrate that the fusion of CI brings a significant performance improvement in comparison to the regular tracking approaches

Detection and Localisation of Pipe Bursts in a District Metered Area Using an Online Hydraulic Model

This thesis presents a research work on the development of new methodology for near-real-time detection and localisation of pipe bursts in a Water Distribution System (WDS) at the District Meters Area (DMA) level. The methodology makes use of online hydraulic model coupled with a demand forecasting methodology and several statistical techniques to process the hydraulic meters data (i.e., flows and pressures) coming from the field at regular time intervals (i.e. every 15 minutes). Once the detection part of the methodology identifies a potential burst occurrence in a system it raises an alarm. This is followed by the application of the burst localisation methodology to approximately locate the event within the District Metered Area (DMA). The online hydraulic model is based on data assimilation methodology coupled with a short-term Water Demand Forecasting Model (WDFM) based on Multi-Linear Regression. Three data assimilation methods were tested in the thesis, namely the iterative Kalman Filter method, the Ensemble Kalman Filter method and the Particle Filter method. The iterative Kalman Filter (i-KF) method was eventually chosen for the online hydraulic model based on the best overall trade-off between water system state prediction accuracy and computational efficiency. The online hydraulic model created this way was coupled with the Statistical Process Control (SPC) technique and a newly developed burst detection metric based on the moving average residuals between the predicted and observed hydraulic states (flows/pressures). Two new SPC-based charts with associated generic set of control rules for analysing burst detection metric values over consecutive time steps were introduced to raise burst alarms in a reliable and timely fashion. The SPC rules and relevant thresholds were determined offline by performing appropriate statistical analysis of residuals. The above was followed by the development of the new methodology for online burst localisation. The methodology integrates the information on burst detection metric values obtained during the detection stage with the new sensitivity matrix developed offline and hydraulic model runs used to simulate potential bursts to identify the most likely burst location in the pipe network. A new data algorithm for estimating the ‘normal’ DMA demand and burst flow during the burst period is developed and used for localisation. A new data algorithm for statistical analysis of flow and pressure data was also developed and used to determine the approximate burst area by producing a list of top ten suspected burst location nodes. The above novel methodologies for burst detection and localisation were applied to two real-life District Metred Areas in the United Kingdom (UK) with artificially generated flow and pressure observations and assumed bursts. The results obtained this way show that the developed methodology detects pipe bursts in a reliable and timely fashion, provides good estimate of a burst flow and accurately approximately locates the burst within a DMA. In addition, the results obtained show the potential of the methodology described here for online burst detection and localisation in assisting Water Companies (WCs) to conserve water, save energy and money. It can also enhance the UK WCs’ profile customer satisfaction, improve operational efficiency and improve the OFWAT’s Service Incentive Mechanism (SIM) scores.This STREAM project is funded by the Engineering and Physical Sciences Research Council and Industrial Collaborator, United Utilities

INTRUSION PREDICTION SYSTEM FOR CLOUD COMPUTING AND NETWORK BASED SYSTEMS

Cloud computing offers cost effective computational and storage services with on-demand scalable capacities according to the customers’ needs. These properties encourage organisations and individuals to migrate from classical computing to cloud computing from different disciplines. Although cloud computing is a trendy technology that opens the horizons for many businesses, it is a new paradigm that exploits already existing computing technologies in new framework rather than being a novel technology. This means that cloud computing inherited classical computing problems that are still challenging. Cloud computing security is considered one of the major problems, which require strong security systems to protect the system, and the valuable data stored and processed in it. Intrusion detection systems are one of the important security components and defence layer that detect cyber-attacks and malicious activities in cloud and non-cloud environments. However, there are some limitations such as attacks were detected at the time that the damage of the attack was already done. In recent years, cyber-attacks have increased rapidly in volume and diversity. In 2013, for example, over 552 million customers’ identities and crucial information were revealed through data breaches worldwide [3]. These growing threats are further demonstrated in the 50,000 daily attacks on the London Stock Exchange [4]. It has been predicted that the economic impact of cyber-attacks will cost the global economy $3 trillion on aggregate by 2020 [5]. This thesis focused on proposing an Intrusion Prediction System that is capable of sensing an attack before it happens in cloud or non-cloud environments. The proposed solution is based on assessing the host system vulnerabilities and monitoring the network traffic for attacks preparations. It has three main modules. The monitoring module observes the network for any intrusion preparations. This thesis proposes a new dynamic-selective statistical algorithm for detecting scan activities, which is part of reconnaissance that represents an essential step in network attack preparation. The proposed method performs a statistical selective analysis for network traffic searching for an attack or intrusion indications. This is achieved by exploring and applying different statistical and probabilistic methods that deal with scan detection. The second module of the prediction system is vulnerabilities assessment that evaluates the weaknesses and faults of the system and measures the probability of the system to fall victim to cyber-attack. Finally, the third module is the prediction module that combines the output of the two modules and performs risk assessments of the system security from intrusions prediction. The results of the conducted experiments showed that the suggested system outperforms the analogous methods in regards to performance of network scan detection, which means accordingly a significant improvement to the security of the targeted system. The scanning detection algorithm has achieved high detection accuracy with 0% false negative and 50% false positive. In term of performance, the detection algorithm consumed only 23% of the data needed for analysis compared to the best performed rival detection method