Detecting Network Soft-failures with the Network Link Outlier Factor (NLOF)

In this paper, we describe and experimentally evaluate the performance of our Network Link Outlier Factor (NLOF) for detecting soft-failures in communication networks. The NLOF is computed using the throughput values derived from NetFlow records. The flow throughput values are clustered in two stages, outlier values are determined within each cluster, and the flow outliers are used to compute the outlier factor or score for each network link. When sampling NetFlow records across the full span of a network, NLOF enables the detection of soft-failures across the span of the network; large NLOF scores correlate well with links experiencing failure

Layered ensemble model for short-term traffic flow forecasting with outlier detection

YesReal time traffic flow forecasting is a necessary requirement for traffic management in order to be able to evaluate the effects of different available strategies or policies. This paper focuses on short-term traffic flow forecasting by taking into consideration both spatial (road links) and temporal (lag or past traffic flow values) information. We propose a Layered Ensemble Model (LEM) which combines Artificial Neural Networks and Graded Possibilistic Clustering obtaining an accurate forecast of the traffic flow rates with outlier detection. Experimentation has been carried out on two different data sets. The former was obtained from real UK motorway and the later was obtained from simulated traffic flow on a street network in Genoa (Italy). The proposed LEM model for short-term traffic forecasting provides promising results and given the ability for outlier detection, accuracy, robustness of the proposed approach, it can be fruitful integrated in traffic flow management systems

Adapted K-Nearest Neighbors for Detecting Anomalies on Spatio–Temporal Traffic Flow

Outlier detection is an extensive research area, which has been intensively studied in several domains such as biological sciences, medical diagnosis, surveillance, and traffic anomaly detection. This paper explores advances in the outlier detection area by finding anomalies in spatio-temporal urban traffic flow. It proposes a new approach by considering the distribution of the flows in a given time interval. The flow distribution probability (FDP) databases are first constructed from the traffic flows by considering both spatial and temporal information. The outlier detection mechanism is then applied to the coming flow distribution probabilities, the inliers are stored to enrich the FDP databases, while the outliers are excluded from the FDP databases. Moreover, a k-nearest neighbor for distance-based outlier detection is investigated and adopted for FDP outlier detection. To validate the proposed framework, real data from Odense traffic flow case are evaluated at ten locations. The results reveal that the proposed framework is able to detect the real distribution of flow outliers. Another experiment has been carried out on Beijing data, the results show that our approach outperforms the baseline algorithms for high-urban traffic flow

Data Improving in Time Series Using ARX and ANN Models

Anomalous data can negatively impact energy forecasting by causing model parameters to be incorrectly estimated. This paper presents two approaches for the detection and imputation of anomalies in time series data. Autoregressive with exogenous inputs (ARX) and artificial neural network (ANN) models are used to extract the characteristics of time series. Anomalies are detected by performing hypothesis testing on the extrema of the residuals, and the anomalous data points are imputed using the ARX and ANN models. Because the anomalies affect the model coefficients, the data cleaning process is performed iteratively. The models are re-learned on “cleaner” data after an anomaly is imputed. The anomalous data are reimputed to each iteration using the updated ARX and ANN models. The ARX and ANN data cleaning models are evaluated on natural gas time series data. This paper demonstrates that the proposed approaches are able to identify and impute anomalous data points. Forecasting models learned on the unclean data and the cleaned data are tested on an uncleaned out-of-sample dataset. The forecasting model learned on the cleaned data outperforms the model learned on the unclean data with 1.67% improvement in the mean absolute percentage errors and a 32.8% improvement in the root mean squared error. Existing challenges include correctly identifying specific types of anomalies such as negative flows