Detection of network anomalies and novel attacks in the internet via statistical network traffic separation and normality prediction

With the advent and the explosive growth of the global Internet and the electronic commerce environment, adaptive/automatic network and service anomaly detection is fast gaining critical research and practical importance. If the next generation of network technology is to operate beyond the levels of current networks, it will require a set of well-designed tools for its management that will provide the capability of dynamically and reliably identifying network anomalies. Early detection of network anomalies and performance degradations is a key to rapid fault recovery and robust networking, and has been receiving increasing attention lately. In this dissertation we present a network anomaly detection methodology, which relies on the analysis of network traffic and the characterization of the dynamic statistical properties of traffic normality, in order to accurately and timely detect network anomalies. Anomaly detection is based on the concept that perturbations of normal behavior suggest the presence of anomalies, faults, attacks etc. This methodology can be uniformly applied in order to detect network attacks, especially in cases where novel attacks are present and the nature of the intrusion is unknown. Specifically, in order to provide an accurate identification of the normal network traffic behavior, we first develop an anomaly-tolerant non-stationary traffic prediction technique, which is capable of removing both pulse and continuous anomalies. Furthermore we introduce and design dynamic thresholds, and based on them we define adaptive anomaly violation conditions, as a combined function of both the magnitude and duration of the traffic deviations. Numerical results are presented that demonstrate the operational effectiveness and efficiency of the proposed approach, under different anomaly traffic scenarios and attacks, such as mail-bombing and UDP flooding attacks. In order to improve the prediction accuracy of the statistical network traffic normality, especially in cases where high burstiness is present, we propose, study and analyze a new network traffic prediction methodology, based on the frequency domain traffic analysis and filtering, with the objective_of enhancing the network anomaly detection capabilities. Our approach is based on the observation that the various network traffic components, are better identified, represented and isolated in the frequency domain. As a result, the traffic can be effectively separated into a baseline component, that includes most of the low frequency traffic and presents low burstiness, and the short-term traffic that includes the most dynamic part. The baseline traffic is a mean non-stationary periodic time series, and the Extended Resource-Allocating Network (BRAN) methodology is used for its accurate prediction. The short-term traffic is shown to be a time-dependent series, and the Autoregressive Moving Average (ARMA) model is proposed to be used for the accurate prediction of this component. Furthermore, it is demonstrated that the proposed enhanced traffic prediction strategy can be combined with the use of dynamic thresholds and adaptive anomaly violation conditions, in order to improve the network anomaly detection effectiveness. The performance evaluation of the proposed overall strategy, in terms of the achievable network traffic prediction accuracy and anomaly detection capability, and the corresponding numerical results demonstrate and quantify the significant improvements that can be achieved

A Survey of Anticipatory Mobile Networking: Context-Based Classification, Prediction Methodologies, and Optimization Techniques

A growing trend for information technology is to not just react to changes, but anticipate them as much as possible. This paradigm made modern solutions, such as recommendation systems, a ubiquitous presence in today's digital transactions. Anticipatory networking extends the idea to communication technologies by studying patterns and periodicity in human behavior and network dynamics to optimize network performance. This survey collects and analyzes recent papers leveraging context information to forecast the evolution of network conditions and, in turn, to improve network performance. In particular, we identify the main prediction and optimization tools adopted in this body of work and link them with objectives and constraints of the typical applications and scenarios. Finally, we consider open challenges and research directions to make anticipatory networking part of next generation networks

The Impact of Special Days in Call Arrivals Forecasting:A Neural Network Approach to Modelling Special Days

A key challenge for call centres remains the forecasting of high frequency call arrivals collected in hourly or shorter time buckets. In addition to the complex intraday, intraweek and intrayear seasonal cycles, call arrival data typically contain a large number of anomalous days, driven by the occurrence of holidays, special events, promotional activities and system failures. This study evaluates the use of a variety of univariate time series forecasting methods for forecasting intraday call arrivals in the presence of such outliers. Apart from established, statistical methods, we consider artificial neural networks (ANNs). Based on the modelling flexibility of the latter, we introduce and evaluate different methods to encode the outlying periods. Using intraday arrival series from a call centre operated by one of Europe’s leading entertainment companies, we provide new insights on the impact of outliers on the performance of established forecasting methods. Results show that ANNs forecast call centre data accurately, and are capable of modelling complex outliers using relatively simple outlier modelling approaches. We argue that the relative complexity of ANNs over standard statistical models is offset by the simplicity of coding multiple and unknown effects during outlying periods.NOTICE: this is the author’s version of a work that was accepted for publication in European Journal of Operational Research. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in European Journal of Operational Research, [264, 3, (2016)] DOI: 10.1016/j.ejor.2016.07.015© 2016, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0

Forecasting model for extreme rainfall using artificial neural network

Successive days of rainfall are known to cause flood. The forecasting of daily rainfall helps to estimate the occurrences of rainfall and number of wet days, while with a maximum of five consecutive days of rainfall, the magnitude of rainfall within a specified period can predict what may signify rainfall extremes. In this study, data mining and back propagation neural network (BPNN) have been established in developing the extreme rainfall forecasting models. Four forecasting models were developed to forecast the maximum five consecutive days of rainfall amount (PX5D) of the next month. The models only use the extreme rainfall indices outlined by STARDEX as predictors in forecasting. The first developed model uses six extreme rainfall indices in forecasting, the second model uses the values of the PX5D index of a three-month delay, the third model uses the previous six-month PX5D values, while the fourth model was developed to forecast the PX5D using the values of the same index of a twelve-month delay. It was found that when using the six extreme rainfall core indices, the forecasting error was the lowest. A regression model has been developed using the six extreme rainfall indices to compare the performance measurements with the BPNN model that uses the same indice

Nonlinear Dynamic Chaos Theory Framework for Passenger Demand Forecasting in Smart City

Recently chaos theory has emerged as a powerful tool to address forecasting problems of nonlinear time series, since it is able to meet the dynamical and geometrical structures of very complex systems, reaching higher accuracy on the prediction values than the classical approaches. This paper aims at applying the chaos theory principles to different problems, in order to pursue high levels of accuracy on the predicted results. After the verification of the chaotic behavior of the datasets taken into analysis through the largest Lyapunov exponent research, the detection of the suitable embedding dimension and time delay has been carried out, in order to reconstruct the phase space of the underlying dynamical systems. Three different predictive methods have been proposed for different datasets. Finally, the performance comparison with the moving average model, a deep neural network based strategy, and a chaos theory based algorithm recently proposed in literature has been provided