Analyzing HTTP requests for web intrusion detection

Many web application security problems related to intrusion have resulted from the rapid development of web applications. To reduce the risk of web application problems, web application developers need to take measures to write secure applications to prevent known attacks. When such measures fail, it is important to detect such attacks and find the source of the attacks to reduce the estimated risks. Intrusion detection is one of the powerful techniques designed to identify and prevent harm to the system. Most defensive techniques in Web Intrusion Systems are not able to deal with the complexity of cyber-attacks in web applications. However, machine learning approaches could help to detect known and unknown web application attacks. In this paper, we present machine learning techniques to classify the HTTP requests in the well-known dataset CSIC 2010 HTTP (Giménez et al., 2012) as normal or abnormal traffic, and we compare our experimental results with the results reported by Pham et al. in 2016 and Nguyen et al. in 2011. These experiments produce results for overlapping sets of machine-learning techniques and different sets of features, allowing us to compare how good the various feature sets are for the various machine-learning techniques, at least on this dataset. Keywords: intrusion detection system; anomaly detection; web application attacks; machine learning

Are Public Intrusion Datasets Fit for Purpose: Characterising the State of the Art in Intrusion Event Datasets

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.In recent years cybersecurity attacks have caused major disruption and information loss for online organisations, with high profile incidents in the news. One of the key challenges in advancing the state of the art in intrusion detection is the lack of representative datasets. These datasets typically contain millions of time-ordered events (e.g. network packet traces, flow summaries, log entries); subsequently analysed to identify abnormal behavior and specific attacks [1]. Generating realistic datasets has historically required expensive networked assets, specialised traffic generators, and considerable design preparation. Even with advances in virtualisation it remains challenging to create and maintain a representative environment. Major improvements are needed in the design, quality and availability of datasets, to assist researchers in developing advanced detection techniques. With the emergence of new technology paradigms, such as intelligent transport and autonomous vehicles, it is also likely that new classes of threat will emerge [2]. Given the rate of change in threat behavior [3] datasets become quickly obsolete, and some of the most widely cited datasets date back over two decades. Older datasets have limited value: often heavily filtered and anonymised, with unrealistic event distributions, and opaque design methodology. The relative scarcity of (Intrusion Detection System) IDS datasets is compounded by the lack of a central registry, and inconsistent information on provenance. Researchers may also find it hard to locate datasets or understand their relative merits. In addition, many datasets rely on simulation, originating from academic or government institutions. The publication process itself often creates conflicts, with the need to de-identify sensitive information in order to meet regulations such as General Data Protection Act (GDPR) [4]. Another final issue for researchers is the lack of standardised metrics with which to compare dataset quality. In this paper we attempt to classify the most widely used public intrusion datasets, providing references to archives and associated literature. We illustrate their relative utility and scope, highlighting the threat composition, formats, special features, and associated limitations. We identify best practice in dataset design, and describe potential pitfalls of designing anomaly detection techniques based on data that may be either inappropriate, or compromised due to unrealistic threat coverage. Such contributions as made in this paper is expected to facilitate continuous research and development for effectively combating the constantly evolving cyber threat landscape