Anomaly detection of web-based attacks

Tese de mestrado em Segurança Informática, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2010Para prevenir ataques com sucesso, é crucial que exista um sistema de detecção que seja preciso e completo. Os sistemas de detecção de intrusão (IDS) baseados em assinaturas são uma das abordagens mais conhecidas para o efeito, mas não são adequados para detectar ataques web ou ataques previamente desconhecidos. O objectivo deste projecto passa pelo estudo e desenho de um sistema de detecção de intrusão baseado em anomalias capaz de detectar esses tipos de ataques. Os IDS baseados em anomalias constroem um modelo de comportamento normal através de dados de treino, e em seguida utilizam-no para detectar novos ataques. Na maioria dos casos, este modelo é representativo de mais exemplos de comportamento normal do que os presentes nos dados de treino, característica esta a que chamamos generalização e que é fundamental para aumentar a precisão na detecção de anomalias. A precisão da detecção e, portanto, a utilidade destes sistemas, é consideravelmente influenciada pela fase de construção do modelo (muitas vezes chamada fase de treino), que depende da existência de um conjunto de dados sem ataques que se assemelhe ao comportamento normal da aplicação protegida. A construção de modelos correctos é particularmente importante, caso contrário, durante a fase de detecção, provavelmente serão geradas grandes quantidades de falsos positivos e falsos negativos pelo IDS. Esta dissertação detalha a nossa pesquisa acerca da utilização de métodos baseados em anomalias para detectar ataques contra servidores e aplicações web. As nossas contribuições incidem sobre três vertentes distintas: i) procedimentos avançados de treino que permitem aos sistemas de detecção baseados em anomalias um bom funcionamento, mesmo em presença de aplicações complexas e dinâmicas, ii) um sistema de detecção de intrusão que compreende diversas técnicas de detecção de anomalias capazes de reconhecer e identificar ataques contra servidores e aplicações web e iii) uma avaliação do sistema e das técnicas mais adequadas para a detecção de ataques, utilizando um elevado conjunto de dados reais de tráfego pertencentes a uma aplicação web de grandes dimensões alojada em servidores de produção num ISP Português.To successfully prevent attacks it is vital to have a complete and accurate detection system. Signature-based intrusion detection systems (IDS) are one of the most popular approaches, but they are not adequate for detection of web-based or novel attacks. The purpose of this project is to study and design an anomaly-based intrusion detection system capable of detecting those kinds of attacks. Anomaly-based IDS can create a model of normal behavior from a set of training data, and then use it to detect novel attacks. In most cases, this model represents more instances than those in the training data set, a characteristic that we designate as generalization and which is necessary for accurate anomaly detection. The accuracy of such systems, which determines their effectiveness, is considerably influenced by the model building phase (often called training), which depends on having data that is free from attacks resembling the normal operation of the protected application. Having good models is particularly important, or else significant amounts of false positives and false negatives will likely be generated by the IDS during the detection phase. This dissertation details our research on the use of anomaly-based methods to detect attacks against web servers and applications. Our contributions focus on three different strands: i) advanced training procedures that enable anomaly-based learning systems to perform well even in presence of complex and dynamic web applications; ii) a system comprising several anomaly detection techniques capable of recognizing and identifying attacks against web servers and applications and iii) an evaluation of the system and of the most suitable techniques for anomaly detection of web attacks, using a large data set of real-word traffic belonging to a web application of great dimensions hosted in production servers of a Portuguese ISP

Mahalanobis Distance Map Approach for Anomaly Detection

Web servers and web-based applications are commonly used as attack targets. The main issues are how to prevent unauthorised access and to protect web servers from the attack. Intrusion Detection Systems (IDSs) are widely used security tools to detect cyber-attacks and malicious activities in computer systems and networks. In this paper, we focus on the detection of various web-based attacks using Geometrical Structure Anomaly Detection (GSAD) model and we also propose a novel algorithm for the selection of most discriminating features to improve the computational complexity of payload-based GSAD model. Linear Discriminant method (LDA) is used for the feature reduction and classification of the incoming network traffic. GSAD model is based on a pattern recognition technique used in image processing. It analyses the correlations between various payload features and uses Mahalanobis Distance Map (MDM) to calculate the difference between normal and abnormal network traffic. We focus on the detection of generic attacks, shell code attacks, polymorphic attacks and polymorphic blending attacks. We evaluate accuracy of GSAD model experimentally on the real-world attacks dataset created at Georgia Institute of Technology. We conducted preliminary experiments on the DARPA 99 dataset to evaluate the accuracy of feature reduction

Anomaly-based Techniques for Web Attacks Detection

The widespread use of the Internet comes accompanied with severe threats for web applications security. Intrusion Detection Systems (IDS) have been considered to deal with the diversity and complexity of web attacks. In this context, this work proposes an algorithm for web attack detection, exploring ananomaly-based technique: the wavelet transform. The proposed algorithm analyzes anomalies within variations on characters frequencies in web requests. Experimental results show high rates of detection without false positive occurrences.Keywords: Web Attacks, Anomaly Detection, Wavelet Transform, Web Applications

Payload-based anomaly detection in HTTP traffic

University of Technology, Sydney. Faculty of Engineering and Information Technology.Internet provides quality and convenience to human life but at the same time it provides a platform for network hackers and criminals. Intrusion Detection Systems (IDSs) have been proven to be powerful methods for detecting anomalies in the network. Traditional IDSs based on signatures are unable to detect new (zero days) attacks. Anomaly-based systems are alternative to signature based systems. However, present anomaly detection systems suffer from three major setbacks: (a) Large number of false alarms, (b) Very high volume of network traffic due to high data rates (Gbps), and (c) Inefficiency in operation. In this thesis, we address above issues and develop efficient intrusion detection frameworks and models which can be used in detecting a wide variety of attacks including web-based attacks. Our proposed methods are designed to have very few false alarms. We also address Intrusion Detection as a Pattern Recognition problem and discuss all aspects that are important in realizing an anomaly-based IDS. We present three payload-based anomaly detectors, including Geometrical Structure Anomaly Detection (GSAD), Two-Tier Intrusion Detection system using Linear Discriminant Analysis (LDA), and Real-time Payload-based Intrusion Detection System (RePIDS), for intrusion detection. These detectors perform deep-packet analysis and examine payload content using n-gram text categorization and Mahalanobis Distance Map (MDM) techniques. An MDM extracts hidden correlations between the features within each payload and among packet payloads. GSAD generates model of normal network payload as geometrical structure using MDMs in a fully automatic and unsupervised manner. We have implemented the GSAD model in HTTP environment for web-based applications. For efficient operation of IDSs, the detection speed is a key point. Current IDSs examine a large number of data features to detect intrusions and misuse patterns. Hence, for quickly and accurately identifying anomalies of Internet traffic, feature reduction becomes mandatory. We have proposed two models to address this issue, namely two-tier intrusion detection model and RePIDS. Two-tier intrusion detection model uses Linear Discriminant Analysis approach for feature reduction and optimal feature selection. It uses MDM technique to create a model of normal network payload using an extracted feature set. RePIDS uses a 3-tier Iterative Feature Selection Engine (IFSEng) to reduce dimensionality of the raw dataset using Principal Component Analysis (PCA) technique. IFSEng extracts the most significant features from the original feature set and uses mathematical and graphical methods for optimal feature subset selection. Like two-tier intrusion detection model, RePIDS then uses MDM technique to generate a model of normal network payload using extracted features. We test the proposed IDSs on two publicly available datasets of attacks and normal traffic. Experimental results confirm the effectiveness and validation of our proposed solutions in terms of detection rate, false alarm rate and computational complexity

An adaptive anomaly request detection framework based on dynamic web application profiles

Web application firewall is a highly effective application in protecting the application layer and database layer of websites from attack access. This paper proposes a new web application firewall deploying method based on Dynamic Web application profiling (DWAP) analysis technique. This is a method to deploy a firewall based on analyzing website access data. DWAP is improved to integrate deeply into the structure of the website to increase the compatibility of the anomaly detection system into each website, thereby improving the ability to detect abnormal requests. To improve the compatibility of the web application firewall with protected objects, the proposed system consists of two parts with the main tasks are: i) Detect abnormal access in web application (WA) access; ii) Semi-automatic update the attack data to the abnormal access detection system during WA access. This new method is applicable in real-time detection systems where updating of new attack data is essential since web attacks are increasingly complex and sophisticated