Unsupervised Anomaly Detection with Unlabeled Data Using Clustering

Intrusions pose a serious security risk in a network environment. New intrusion types, of which detection systems are unaware, are the most difficult to detect. The amount of available network audit data instances is usually large; human labeling is tedious, time-consuming, and expensive. Traditional anomaly detection algorithms require a set of purely normal data from which they train their model. We present a clustering-based intrusion detection algorithm, unsupervised anomaly detection, which trains on unlabeled data in order to detect new intrusions. Our method is able to detect many different types of intrusions, while maintaining a low false positive rate as verified over the Knowledge Discovery and Data Mining - KDD CUP 1999 dataset

Hierarchic Clustering Algorithm Used for Anomaly Detecting

AbstractThe popularity of using Internet contains some risks of network attacks. Intrusion detection is one major research problem in network security, whose aim is to prevent unauthorized access to system resources and data. This paper choose the clustering algorithm based on the hierarchical structure, to form normal behavior profile on the audit records and adjust the profile timely as the program behavior changed. The algorithm can convert the problem to resolve the problem of massive data processing to the hot research point of anomaly detection. Moreover, in order to improve the results of testing further, we choose data processing algorithm to get high-quality data source. As the experiment shown, we get effective experimental result

Outlier Detection using Boxplot-Mean Algorithm

In this paper, we present a novel method for the detection of outlier in intrusion detection system. The proposed detection algorithm, are called hybrid algorithm. It is combination of two algorithm k-mean and boxplot. Experimental results demonstrate to be superior to existing SCF algorithm. One of the most common problems in existing SCF technique detection techniques is that such as ignoring dependency among categorical variables, handling data streams and mixed data sets. Moreover, identifying number of outliers in advance is an impractical issue in the SCF algorithm and other outlier identification techniques. This paper investigates the performances of boxplot-mean method for detecting different types of abnormal data. Keywords: Outlier detection techniques, clustering, scf, genetic and boxplotmean technique

PENGELOMPOKAN TRAFIK BERDASARKAN WAKTU DENGAN ALGORITMA CLUSTREAM UNTUK DETEKSI ANOMALI PADA ALIRAN TRAFIK

Pada perkembangan teknologi jaringan internet sekarang ini banyak membahas tentang fenomena-fenomena serangan atapun ancaman terhadap sebuah komputer atau server. Banyak sekali macam-macam tipe ancaman pada komputer dalam sebuah jaringan internet seperti DoS (Denial of Service), DDoS (Distributed Denial of Service), flash-crowd, dan sebagainya. Oleh karena itu, untuk memudahkan dalam pengambilan informasi agar sesuai dengan keinginan, perlu adanya pengelompokan dalam anomali trafik tersebut untuk mengenali tipe-tipe serangan yang baru. Dari permasalahan tersebut perlu suatu sistem deteksi anomali trafik yang mempunyai kemampuan untuk mendeteksi anomali dan mengenali setiap serangan yang datang dengan dilakukan pengelompokkan berdasarkan waktu dan grup. Waktu dan grup adalah parameter untuk meningkatkan akurasi deteksi algoritma. Pada penelitian ini dibangun sebuah metode IDS yang menggunakan algoritma clustream. Hasil dari penelitian ini, sistem yang dibangun secara real-time dapat bekerja dengan baik dalam deteksi dan membedakan antara trafik normal dan anomali trafik. Pengelompokan trafik dilakukan per-2 detik, setelah itu akan dianalisis dengan algoritma clustream. Algoritma ini terbagi menjadi online (micro-clustering) dan offline (macro-clustering). Di mana macro-clustering akan menggunakan data hasil dari micro-clustering. Kata Kunci : anomali trafik, clustering, algoritma clustream, stream traffi

Multi-Stage Intrusion Detection Approach for Network Security

Nowadays, the massive increment in applications running on a computer and excessive in network services forces to take convenient security policies into an account. Many methods of intrusion detection proposed to provide security in a computer system and network using data mining methods. These methods comprise of the outlier, unsupervised and supervised methods. As we know, each data mining method is not able to find different types of attacks. So, for removing this vulnerability, we are using Multi-Stage Intrusion Detection Method that containing outlier, unsupervised and supervised detection approaches for improving the performance and detection accuracy by reducing the false alarms for detection of known and unknown attacks. We have used NSL-KDD, KDD Corrected and GureKDD dataset in our experiment. We have compared our proposed outlier method GBBK + with GBBK method and our method gives the same result with the less time complexity. The Unsupervised classification algorithm k − point performing the unnecessary comparison of objects iteratively by reducing number of attributes every time up to the threshold that is improved and named as k – point + . Empirically, the proposed scheme compared with existing methods, and the results shows that the proposed method outperform in term of time complexity and detection accuracy

A survey of machine learning methods applied to anomaly detection on drinking-water quality data

Abstract: Traditional machine learning (ML) techniques such as support vector machine, logistic regression, and artificial neural network have been applied most frequently in water quality anomaly detection tasks. This paper presents a review of progress and advances made in detecting anomalies in water quality data using ML techniques. The review encompasses both traditional ML and deep learning (DL) approaches. Our findings indicate that: 1) Generally, DL approaches outperform traditional ML techniques in terms of feature learning accuracy and fewer false positive rates. However, is difficult to make a fair comparison between studies because of different datasets, models and parameters employed. 2) We notice that despite advances made and the advantages of the extreme learning machine (ELM), application of ELM is sparsely exploited in this domain. This study also proposes a hybrid DL-ELM framework as a possible solution that could be investigated further and used to detect anomalies in water quality data

Applicability of clustering to cyber intrusion detection

Maintaining cyber security is a complex task, utilizing many levels of network information along with an array of technology. Current practices for combating cyber attacks typically use Intrusion Detection Systems (IDSs) to passively detect and block multi-stage attacks. Because of the speed and force at which a new type of cyber attack can occur, automated detection and response is becoming an apparent necessity. Anomaly-based detection systems, such as statistical-based or clustering algorithms, attempt to address this by analyzing the relative differences in network and host activity. Signature-based IDS systems are typically more accurate for known attacks, but require time and resources for an analyst to update the signature database. This work hypothesizes that the latency from zero-day attack to signature creation can be shortened via anomaly-based algorithms. In particular, the summarizing ability of clustering is leveraged and examined in its applicability of signature creation. This work first investigates a modified density-based clustering algorithm as an IDS, with its strengths and weaknesses identified. Being able to separate malicious from normal activity, the modified algorithm is then applied in a supervised way to signature creation. Lessons learned from the supervised signature creation are then leveraged for the development of unsupervised real-time signature classification. Automating signature creation and classification via clustering turns out satisfactory but with limitations. Density supports for new signatures via clustering can be diluted and lead to misclassification

GraphBAD: A General Technique for Anomaly Detection in Security Information and Event Management

The reliance on expert knowledge—required for analysing security logs and performing security audits—has created an unhealthy balance, where many computer users are not able to correctly audit their security configurations and react to potential security threats. The decreasing cost of IT and the increasing use of technology in domestic life are exacerbating this problem, where small companies and home IT users are not able to afford the price of experts for auditing their system configuration. In this paper, we present GraphBAD, a graph-based analysis tool that is able to analyse security configurations in order to identify anomalies that could lead to potential security risks. GraphBAD, which does not require any prior domain knowledge, generates graph-based models from security configuration data and, by analysing such models, is able to propose mitigation plans that can help computer users in increasing the security of their systems. A large experimental analysis, conducted on both publicly available (the well-known KDD dataset) and synthetically generated testing sets (file system permissions), demonstrates the ability of GraphBAD in correctly identifying security configuration anomalies and suggesting appropriate mitigation plans

Unsupervised Intrusion Detection with Cross-Domain Artificial Intelligence Methods

Cybercrime is a major concern for corporations, business owners, governments and citizens, and it continues to grow in spite of increasing investments in security and fraud prevention. The main challenges in this research field are: being able to detect unknown attacks, and reducing the false positive ratio. The aim of this research work was to target both problems by leveraging four artificial intelligence techniques. The first technique is a novel unsupervised learning method based on skip-gram modeling. It was designed, developed and tested against a public dataset with popular intrusion patterns. A high accuracy and a low false positive rate were achieved without prior knowledge of attack patterns. The second technique is a novel unsupervised learning method based on topic modeling. It was applied to three related domains (network attacks, payments fraud, IoT malware traffic). A high accuracy was achieved in the three scenarios, even though the malicious activity significantly differs from one domain to the other. The third technique is a novel unsupervised learning method based on deep autoencoders, with feature selection performed by a supervised method, random forest. Obtained results showed that this technique can outperform other similar techniques. The fourth technique is based on an MLP neural network, and is applied to alert reduction in fraud prevention. This method automates manual reviews previously done by human experts, without significantly impacting accuracy