Model-Based Outlier Detection System with Statistical Preprocessing

Reliability, lack of error, and security are important improvements to quality of service. Outlier detection is a process of detecting the erroneous parts or abnormal objects in defined populations, and can contribute to secured and error-free services. Outlier detection approaches can be categorized into four types: statistic-based, unsupervised, supervised, and semi-supervised. A model-based outlier detection system with statistical preprocessing is proposed, taking advantage of the statistical approach to preprocess training data and using unsupervised learning to construct the model. The robustness of the proposed system is evaluated using the performance evaluation metrics sum of squared error (SSE) and time to build model (TBM). The proposed system performs better for detecting outliers regardless of the application domain

SMART OUTLIER DETECTION OF WIRELESS SENSOR NETWORK

Data sets collected from wireless sensor networks (WSN) are usually considered unreliable and subject to errors due to limited sensor capabilities and hard environmental resulting in a subset of the sensors data called outlier data. This paper proposes a technique to detect outlier data base on spatial-temporal similarity among data collected by geographically distributed sensors. The proposed technique is able to identify an abnormal subset of data collected by sensor node as outlier data. Moreover the proposed technique is able to classify this abnormal observation, an error data set or event affected set. Simulation result shows that high detection rate is achieved compared to conventional outlier detection techniques while preserving low positive false alarm rate

OUTLIER DETECTION TECHNIQUE USING CT-OCSVM AND FUZZY RULE-BASED SYSTEM IN WIRELESS SENSOR NETWORKS

The development of Wireless Sensor Networks (WSNs) has been attained in the past few years due to its important using in wide range of application. The readings of data derived from WSN nodes are not always accurate and may contain abnormal data. This paper proposed an anomaly detection and classification algorithm in WSNs. At first, an integration of Contourlet Transform (CT) algorithm and One Class Support Vector Machine (OCSVM) algorithm (CT-OCSVM) is utilized to detect outliers then Fuzzy Inference System (FIS) is used to identify the source of these outliers. The underlying aim of this paper focuses on treating the collected streams of data as raw datum of an image, which is then passed through some filters using CT to get compressed size of directional subbands coefficients. The coefficients of CT are examined by OCSVM algorithm to detect anomalies. Finally the source of anomalies is identified based on using FIS and by exploiting the spatial temporal correlation existing between the sensed data. The integrated algorithm is tested using different types of filters. Real datasets collected from a small WSN constructed in a local lab are used for testing the integrated algorithms. The simulation results have shown a high rate of accurate classification with high detection rate and low false alarm rate

Time Series Outlier Detection Based on Sliding Window Prediction

In order to detect outliers in hydrological time series data for improving data quality and decision-making quality related to design, operation, and management of water resources, this research develops a time series outlier detection method for hydrologic data that can be used to identify data that deviate from historical patterns. The method first built a forecasting model on the history data and then used it to predict future values. Anomalies are assumed to take place if the observed values fall outside a given prediction confidence interval (PCI), which can be calculated by the predicted value and confidence coefficient. The use of PCI as threshold is mainly on the fact that it considers the uncertainty in the data series parameters in the forecasting model to address the suitable threshold selection problem. The method performs fast, incremental evaluation of data as it becomes available, scales to large quantities of data, and requires no preclassification of anomalies. Experiments with different hydrologic real-world time series showed that the proposed methods are fast and correctly identify abnormal data and can be used for hydrologic time series analysis

Detection of Outliers in a Time Series of Available Parking Spaces

With the provision of any source of real-time information, the timeliness and accuracy of the data provided are paramount to the effectiveness and success of the system and its acceptance by the users. In order to improve the accuracy and reliability of parking guidance systems (PGSs), the technique of outlier mining has been introduced for detecting and analysing outliers in available parking space (APS) datasets. To distinguish outlier features from the APS’s overall periodic tendency, and to simultaneously identify the two types of outliers which naturally exist in APS datasets with intrinsically distinct statistical features, a two-phase detection method is proposed whereby an improved density-based detection algorithm named “local entropy based weighted outlier detection” (EWOD) is also incorporated. Real-world data from parking facilities in the City of Newcastle upon Tyne was used to test the hypothesis. Thereafter, experimental tests were carried out for a comparative study in which the outlier detection performances of the two-phase detection method, statistic-based method, and traditional density-based method were compared and contrasted. The results showed that the proposed method can identify two different kinds of outliers simultaneously and can give a high identifying accuracy of 100% and 92.7% for the first and second types of outliers, respectively