Online network monitoring

An important problem in network analysis is the online detection of anomalous behaviour. In this paper, we introduce a network surveillance method bringing together network modelling and statistical process control. Our approach is to apply multivariate control charts based on exponential smoothing and cumulative sums in order to monitor networks generated by temporal exponential random graph models (TERGM). The latter allows us to account for temporal dependence while simultaneously reducing the number of parameters to be monitored. The performance of the considered charts is evaluated by calculating the average run length and the conditional expected delay for both simulated and real data. To justify the decision of using the TERGM to describe network data, some measures of goodness of fit are inspected. We demonstrate the effectiveness of the proposed approach by an empirical application, monitoring daily flights in the United States to detect anomalous patterns. © 2021, The Author(s)

Online network monitoring

An important problem in network analysis is the online detection of anomalous behaviour. In this paper, we introduce a network surveillance method bringing together network modelling and statistical process control. Our approach is to apply multivariate control charts based on exponential smoothing and cumulative sums in order to monitor networks generated by temporal exponential random graph models (TERGM). The latter allows us to account for temporal dependence while simultaneously reducing the number of parameters to be monitored. The performance of the considered charts is evaluated by calculating the average run length and the conditional expected delay for both simulated and real data. To justify the decision of using the TERGM to describe network data, some measures of goodness of fit are inspected. We demonstrate the effectiveness of the proposed approach by an empirical application, monitoring daily flights in the United States to detect anomalous patterns

Events Recognition System for Water Treatment Works

The supply of drinking water in sufficient quantity and required quality is a challenging task for water companies. Tackling this task successfully depends largely on ensuring a continuous high quality level of water treatment at Water Treatment Works (WTW). Therefore, processes at WTWs are highly automated and controlled. A reliable and rapid detection of faulty sensor data and failure events at WTWs processes is of prime importance for its efficient and effective operation. Therefore, the vast majority of WTWs operated in the UK make use of event detection systems that automatically generate alarms after the detection of abnormal behaviour on observed signals to ensure an early detection of WTW’s process failures. Event detection systems usually deployed at WTWs apply thresholds to the monitored signals for the recognition of WTW’s faulty processes. The research work described in this thesis investigates new methods for near real-time event detection at WTWs by the implementation of statistical process control and machine learning techniques applied for an automated near real-time recognition of failure events at WTWs processes. The resulting novel Hybrid CUSUM Event Recognition System (HC-ERS) makes use of new online sensor data validation and pre-processing techniques and utilises two distinct detection methodologies: first for fault detection on individual signals and second for the recognition of faulty processes and events at WTWs. The fault detection methodology automatically detects abnormal behaviour of observed water quality parameters in near real-time using the data of the corresponding sensors that is online validated and pre-processed. The methodology utilises CUSUM control charts to predict the presence of faults by tracking the variation of each signal individually to identify abnormal shifts in its mean. The basic CUSUM methodology was refined by investigating optimised interdependent parameters for each signal individually. The combined predictions of CUSUM fault detection on individual signals serves the basis for application of the second event detection methodology. The second event detection methodology automatically identifies faults at WTW’s processes respectively failure events at WTWs in near real-time, utilising the faults detected by CUSUM fault detection on individual signals beforehand. The method applies Random Forest classifiers to predict the presence of an event at WTW’s processes. All methods have been developed to be generic and generalising well across different drinking water treatment processes at WTWs. HC-ERS has proved to be effective in the detection of failure events at WTWs demonstrated by the application on real data of water quality signals with historical events from a UK’s WTWs. The methodology achieved a peak F1 value of 0.84 and generates 0.3 false alarms per week. These results demonstrate the ability of method to automatically and reliably detect failure events at WTW’s processes in near real-time and also show promise for practical application of the HC-ERS in industry. The combination of both methodologies presents a unique contribution to the field of near real-time event detection at WTW

Robust control charts via winsorized and trimmed estimators

In process control, cumulative sum (CUSUM), exponentially weighted moving average (EWMA), and synthetic control charts are developed to detect small and moderate shifts. Small shifts which are hard to detect can be costly to the process control if left undetected for a long period. These control charts are not reliable under non-normality as the design structure of the charts is based on the sample mean. Sample mean is sensitive to outliers, a common cause of non-normality. In circumventing the problem, this study applied robust location estimators in the design structure of the control charts, instead of the sample mean. For such purpose, four robust estimators namely 20%-trimmed mean, median, modified one-step M-estimator (MOM), and winsorized MOM (WMOM) were chosen. The proposed charts were tested on several conditions which include sample sizes, shift sizes, and different types of non-normal distributions represented by the g-and-h distribution. Random variates for each distribution were obtained using SAS RANNOR before transforming them to the desired type of distribution. Robustness and detection ability of the charts were gauged through average run length (ARL) via simulation study. Validation of the charts’ performance which was done through real data study, specifically on potential diabetic patients at Universiti Utara Malaysia shows that robust EWMA chart and robust CUSUM chart outperform the standard charts. The findings concur with the results of simulation study. Even though robust synthetic chart is not among the best choice as it cannot detect small shifts as quickly as CUSUM or EWMA, its performance is much better than the standard chart under non-normality. This study reveals that all the proposed robust charts fare better than the standard charts under non-normality, and comparable with the latter under normality. The most robust among the investigated charts are EWMA control charts based on MOM and WMOM. These robust charts can fast detect small shifts regardless of distributional shapes and work well under small sample sizes. These characteristics suit the industrial needs in process monitoring