Anomaly Detection in Time Series: Theoretical and Practical Improvements for Disease Outbreak Detection

The automatic collection and increasing availability of health data provides a new opportunity for techniques to monitor this information. By monitoring pre-diagnostic data sources, such as over-the-counter cough medicine sales or emergency room chief complaints of cough, there exists the potential to detect disease outbreaks earlier than traditional laboratory disease confirmation results. This research is particularly important for a modern, highly-connected society, where the onset of disease outbreak can be swift and deadly, whether caused by a naturally occurring global pandemic such as swine flu or a targeted act of bioterrorism. In this dissertation, we first describe the problem and current state of research in disease outbreak detection, then provide four main additions to the field. First, we formalize a framework for analyzing health series data and detecting anomalies: using forecasting methods to predict the next day's value, subtracting the forecast to create residuals, and finally using detection algorithms on the residuals. The formalized framework indicates the link between the forecast accuracy of the forecast method and the performance of the detector, and can be used to quantify and analyze the performance of a variety of heuristic methods. Second, we describe improvements for the forecasting of health data series. The application of weather as a predictor, cross-series covariates, and ensemble forecasting each provide improvements to forecasting health data. Third, we describe improvements for detection. This includes the use of multivariate statistics for anomaly detection and additional day-of-week preprocessing to aid detection. Most significantly, we also provide a new method, based on the CuScore, for optimizing detection when the impact of the disease outbreak is known. This method can provide an optimal detector for rapid detection, or for probability of detection within a certain timeframe. Finally, we describe a method for improved comparison of detection methods. We provide tools to evaluate how well a simulated data set captures the characteristics of the authentic series and time-lag heatmaps, a new way of visualizing daily detection rates or displaying the comparison between two methods in a more informative way

ARL Evaluation of a DEWMA Control Chart for Autocorrelated Data: A Case Study on Prices of Major Industrial Commodities

The double exponentially weighted moving average (DEWMA) control chart, an extension of the EWMA control chart, is a useful statistical process control tool for detecting small shift sizes in the mean of processes with either independent or autocorrelated observations. In this study, we derived explicit formulas to compute the average run length (ARL) for a moving average of order q (MA(q)) process with exponential white noise running on a DEWMA control chart and verified their accuracy by comparison with the numerical integral equation (NIE) method. The results for both were in good agreement with the actual ARL. To investigate the efficiency of the proposed procedure on the DEWMA control chart, a performance comparison between it and the standard and modified EWMA control charts was also conducted to determine which provided the smallest out-of-control ARL value for several scenarios involving MA(q) processes. It was found that the DEWMA control chart provided the lowest out-of-control ARL for all cases of varying the exponential smoothing parameter and shift size values. To illustrate the efficacy of the proposed methodology, the presented approach was applied to datasets of the prices of several major industrial commodities in Thailand. The findings show that the DEWMA procedure performed well in almost all of the scenarios tested. Doi: 10.28991/ESJ-2023-07-05-020 Full Text: PD

Poisson Distributed Individuals Control Charts with Optimal Limits

The conventional method used in attribute control charts is the Shewhart three sigma limits. The implicit assumption of the Normal distribution in this approach is not appropriate for skewed distributions such as Poisson, Geometric and Negative Binomial. Normal approximations perform poorly in the tail area of the these distributions. In this research, a type of attribute control chart is introduced to monitor the processes that provide count data. The economic objective of this chart is to minimize the cost of its errors which is determined by the designer. This objective is a linear function of type I and II errors. The proposed control chart can be applied to Poisson, Geometric and Negative Binomial as the underlying distribution of count data. Control limits in this chart is calculated optimally since it is based on the probability distribution of the data and can detect a directional shift in the process rate. Some numerical results for the optimal design of the proposed control chart are provided. The expected cost of the control chart is compared to that of a one sided c chart. The effects of changing the available parameters on the cost, errors and the optimal limits of the proposed control chart are shown graphically