Enhanced classification of network traffic data captured by intrusion prevention systems

A common practice in modern computer networks is the deployment of Intrusion Prevention Systems (IPSs) for the purpose of identifying security threats. Such systems provide alerts on suspicious activities based on a predefined set of rules. These alerts almost always contain high percentages of false positives and false negatives, which may impede the efficacy of their use. Therefore, with the presence of high numbers of false positives and false negatives, the analysis of network traffic data can be ineffective for decision makers which normally require concise, and preferably, visual forms to base their decisions upon. Machine learning techniques can help extract useful information from large datasets. Combined with visualisation, classification could provide a solution to false alerts and text-based outputs of IPSs. This research developed two new classification techniques that outperformed the traditional classification methods in accurate classification of computer network traffic captured by an IPS framework. They are also highly effective. The main purpose of these techniques was the effective identification of malicious network traffic and this was demonstrated via extensive experimental evaluation (where many experiments were conducted and results are reported in this thesis). In addition, an enhancement of the principal component analysis (PCA) was presented as part of this study. This enhancement proved to outperform the classical PCA on classification of IPS data. Details of the evaluation and experiments are provided in this thesis. One of the classification methods described in this thesis achieved accuracy values of 98.51% and 99.76% on two computer network traffic dataset settings, whereas the Class-balanced Similarity Based Instance Transfer Learning (CB-SBIT) algorithm achieves accuracy values of 93.56% and 96.25% respectively on the same dataset settings. This means the proposed method outperforms the state-of-the-art algorithm. As for the PCA enhancement mentioned above, using its resulting principal components as inputs to classifiers leads to improved accuracy when compared to the classical PCA

PCA filtering and probabilistic SOM for network intrusion detection

The growth of the Internet and, consequently, the number of interconnected computers, has exposed significant amounts of information to intruders and attackers. Firewalls aim to detect violations according to a predefined rule-set and usually block potentially dangerous incoming traffic. However, with the evolution of attack techniques, it is more difficult to distinguish anomalies from normal traffic. Different detection approaches have been proposed, including the use of machine learning techniques based on neural models such as Self-Organizing Maps (SOMs). In this paper, we present a classification approach that hybridizes statistical techniques and SOM for network anomaly detection. Thus, while Principal Component Analysis (PCA) and Fisher Discriminant Ratio (FDR) have been considered for feature selection and noise removal, Probabilistic Self-Organizing Maps (PSOM) aim to model the feature space and enable distinguishing between normal and anomalous connections

A P2P Botnet detection scheme based on decision tree and adaptive multilayer neural networks

In recent years, Botnets have been adopted as a popular method to carry and spread many malicious codes on the Internet. These malicious codes pave the way to execute many fraudulent activities including spam mail, distributed denial-of-service attacks and click fraud. While many Botnets are set up using centralized communication architecture, the peer-to-peer (P2P) Botnets can adopt a decentralized architecture using an overlay network for exchanging command and control data making their detection even more difficult. This work presents a method of P2P Bot detection based on an adaptive multilayer feed-forward neural network in cooperation with decision trees. A classification and regression tree is applied as a feature selection technique to select relevant features. With these features, a multilayer feed-forward neural network training model is created using a resilient back-propagation learning algorithm. A comparison of feature set selection based on the decision tree, principal component analysis and the ReliefF algorithm indicated that the neural network model with features selection based on decision tree has a better identification accuracy along with lower rates of false positives. The usefulness of the proposed approach is demonstrated by conducting experiments on real network traffic datasets. In these experiments, an average detection rate of 99.08 % with false positive rate of 0.75 % was observed

Structural Analysis of Network Traffic Matrix via Relaxed Principal Component Pursuit

The network traffic matrix is widely used in network operation and management. It is therefore of crucial importance to analyze the components and the structure of the network traffic matrix, for which several mathematical approaches such as Principal Component Analysis (PCA) were proposed. In this paper, we first argue that PCA performs poorly for analyzing traffic matrix that is polluted by large volume anomalies, and then propose a new decomposition model for the network traffic matrix. According to this model, we carry out the structural analysis by decomposing the network traffic matrix into three sub-matrices, namely, the deterministic traffic, the anomaly traffic and the noise traffic matrix, which is similar to the Robust Principal Component Analysis (RPCA) problem previously studied in [13]. Based on the Relaxed Principal Component Pursuit (Relaxed PCP) method and the Accelerated Proximal Gradient (APG) algorithm, we present an iterative approach for decomposing a traffic matrix, and demonstrate its efficiency and flexibility by experimental results. Finally, we further discuss several features of the deterministic and noise traffic. Our study develops a novel method for the problem of structural analysis of the traffic matrix, which is robust against pollution of large volume anomalies.Comment: Accepted to Elsevier Computer Network

A Study of Feature Reduction Techniques and Classification for Network Anomaly Detection

Due to the launch of new applications the behavior of internet traffic is changing. Hackers are always looking for sophisticated tools to launch attacks and damage the services. Researchers have been working on intrusion detection techniques involving machine learning algorithms for supervised and unsupervised detection of these attacks. However, with newly found attacks these techniques need to be refined. Handling data with large number of attributes adds to the problem. Therefore, dimensionality based feature reduction of the data is required. In this work three reduction techniques, namely, Principal Component Analysis (PCA), Artificial Neural Network (ANN), and Nonlinear Principal Component Analysis (NLPCA) have been studied and analyzed. Secondly, performance of four classifiers, namely, Decision Tree (DT), Support Vector Machine (SVM), K Nearest Neighbor (KNN) and Naïve Bayes (NB) has been studied for the actual and reduced datasets. In addition, novel performance measurement metrics, Classification Difference Measure (CDM), Specificity Difference Measure (SPDM), Sensitivity Difference Measure (SNDM), and F1 Difference Measure (F1DM) have been defined and used to compare the outcomes on actual and reduced datasets. Comparisons have been done using new Coburg Intrusion Detection Data Set (CIDDS-2017) dataset as well widely referred NSL-KDD dataset. Successful results were achieved for Decision Tree with 99.0 percent and 99.8 percent accuracy on CIDDS and NSLKDD datasets respectively

Role based behavior analysis

Tese de mestrado, Segurança Informática, Universidade de Lisboa, Faculdade de Ciências, 2009Nos nossos dias, o sucesso de uma empresa depende da sua agilidade e capacidade de se adaptar a condições que se alteram rapidamente. Dois requisitos para esse sucesso são trabalhadores proactivos e uma infra-estrutura ágil de Tecnologias de Informacão/Sistemas de Informação (TI/SI) que os consiga suportar. No entanto, isto nem sempre sucede. Os requisitos dos utilizadores ao nível da rede podem nao ser completamente conhecidos, o que causa atrasos nas mudanças de local e reorganizações. Além disso, se não houver um conhecimento preciso dos requisitos, a infraestrutura de TI/SI poderá ser utilizada de forma ineficiente, com excessos em algumas áreas e deficiências noutras. Finalmente, incentivar a proactividade não implica acesso completo e sem restrições, uma vez que pode deixar os sistemas vulneráveis a ameaças externas e internas. O objectivo do trabalho descrito nesta tese é desenvolver um sistema que consiga caracterizar o comportamento dos utilizadores do ponto de vista da rede. Propomos uma arquitectura de sistema modular para extrair informação de fluxos de rede etiquetados. O processo é iniciado com a criação de perfis de utilizador a partir da sua informação de fluxos de rede. Depois, perfis com características semelhantes são agrupados automaticamente, originando perfis de grupo. Finalmente, os perfis individuais são comprados com os perfis de grupo, e os que diferem significativamente são marcados como anomalias para análise detalhada posterior. Considerando esta arquitectura, propomos um modelo para descrever o comportamento de rede dos utilizadores e dos grupos. Propomos ainda métodos de visualização que permitem inspeccionar rapidamente toda a informação contida no modelo. O sistema e modelo foram avaliados utilizando um conjunto de dados reais obtidos de um operador de telecomunicações. Os resultados confirmam que os grupos projectam com precisão comportamento semelhante. Além disso, as anomalias foram as esperadas, considerando a população subjacente. Com a informação que este sistema consegue extrair dos dados em bruto, as necessidades de rede dos utilizadores podem sem supridas mais eficazmente, os utilizadores suspeitos são assinalados para posterior análise, conferindo uma vantagem competitiva a qualquer empresa que use este sistema.In our days, the success of a corporation hinges on its agility and ability to adapt to fast changing conditions. Proactive workers and an agile IT/IS infrastructure that can support them is a requirement for this success. Unfortunately, this is not always the case. The user’s network requirements may not be fully understood, which slows down relocation and reorganization. Also, if there is no grasp on the real requirements, the IT/IS infrastructure may not be efficiently used, with waste in some areas and deficiencies in others. Finally, enabling proactivity does not mean full unrestricted access, since this may leave the systems vulnerable to outsider and insider threats. The purpose of the work described on this thesis is to develop a system that can characterize user network behavior. We propose a modular system architecture to extract information from tagged network flows. The system process begins by creating user profiles from their network flows’ information. Then, similar profiles are automatically grouped into clusters, creating role profiles. Finally, the individual profiles are compared against the roles, and the ones that differ significantly are flagged as anomalies for further inspection. Considering this architecture, we propose a model to describe user and role network behavior. We also propose visualization methods to quickly inspect all the information contained in the model. The system and model were evaluated using a real dataset from a large telecommunications operator. The results confirm that the roles accurately map similar behavior. The anomaly results were also expected, considering the underlying population. With the knowledge that the system can extract from the raw data, the users network needs can be better fulfilled, the anomalous users flagged for inspection, giving an edge in agility for any company that uses it