Towards real-time intrusion detection for NetFlow and IPFIX

DDoS attacks bring serious economic and technical damage to networks and enterprises. Timely detection and mitigation are therefore of great importance. However, when flow monitoring systems are used for intrusion detection, as it is often the case in campus, enterprise and backbone networks, timely data analysis is constrained by the architecture of NetFlow and IPFIX. In their current architecture, the analysis is performed after certain timeouts, which generally delays the intrusion detection for several minutes. This paper presents a functional extension for both NetFlow and IPFIX flow exporters, to allow for timely intrusion detection and mitigation of large flooding attacks. The contribution of this paper is threefold. First, we integrate a lightweight intrusion detection module into a flow exporter, which moves detection closer to the traffic observation point. Second, our approach mitigates attacks in near real-time by instructing firewalls to filter malicious traffic. Third, we filter flow data of malicious traffic to prevent flow collectors from overload. We validate our approach by means of a prototype that has been deployed on a backbone link of the Czech national research and education network CESNET

Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX

Flow monitoring has become a prevalent method for monitoring traffic in high-speed networks. By focusing on the analysis of flows, rather than individual packets, it is often said to be more scalable than traditional packet-based traffic analysis. Flow monitoring embraces the complete chain of packet observation, flow export using protocols such as NetFlow and IPFIX, data collection, and data analysis. In contrast to what is often assumed, all stages of flow monitoring are closely intertwined. Each of these stages therefore has to be thoroughly understood, before being able to perform sound flow measurements. Otherwise, flow data artifacts and data loss can be the consequence, potentially without being observed. This paper is the first of its kind to provide an integrated tutorial on all stages of a flow monitoring setup. As shown throughout this paper, flow monitoring has evolved from the early 1990s into a powerful tool, and additional functionality will certainly be added in the future. We show, for example, how the previously opposing approaches of deep packet inspection and flow monitoring have been united into novel monitoring approaches

A Two-stage Flow-based Intrusion Detection Model ForNext-generation Networks

The next-generation network provides state-of-the-art access-independent services over converged mobile and fixed networks. Security in the converged network environment is a major challenge. Traditional packet and protocol-based intrusion detection techniques cannot be used in next-generation networks due to slow throughput, low accuracy and their inability to inspect encrypted payload. An alternative solution for protection of next-generation networks is to use network flow records for detection of malicious activity in the network traffic. The network flow records are independent of access networks and user applications. In this paper, we propose a two-stage flow-based intrusion detection system for next-generation networks. The first stage uses an enhanced unsupervised one-class support vector machine which separates malicious flows from normal network traffic. The second stage uses a self-organizing map which automatically groups malicious flows into different alert clusters. We validated the proposed approach on two flow-based datasets and obtained promising results

Real-time DDoS attack detection for Cisco IOS using NetFlow

Flow-based DDoS attack detection is typically performed by analysis applications that are installed on or close to a flow collector. Although this approach allows for easy deployment, it makes detection far from real-time and susceptible to DDoS attacks for the following reasons. First, the fact that the flow export process is timeout-based and that flow collectors typically provide data to analysis applications in chunks, can result in detection delays in the order of several minutes. Second, by the nature of flow export, attack traffic may be amplified by the flow export process if the original packets are small enough and are part of small flows. We have shown in a previous work how to perform DDoS attack detection on a flow exporter instead of a flow collector, i.e., close to the data source and in a real-time fashion, which however required access to a fully-extendible flow monitoring infrastructure. In this work, we investigate whether it is possible to operate the same detection system on a widely deployed networking platform: Cisco IOS. Since our ultimate goal is to identify besides the presence of an attack also attackers and targets, we rely on NetFlow. In this context, we present our DDoS attack detection prototype that has shown to generate a constant load on the underlying platform — even under attacks — underlining that DDoS attack detection can be performed on a Cisco Catalyst 6500 in production networks, if enough spare capacity is available

Detection of HTTPS brute-force attacks in high-speed computer networks

Tato práce představuje přehled metod pro detekci síťových hrozeb se zaměřením na útoky hrubou silou proti webovým aplikacím, jako jsou WordPress a Joomla. Byl vytvořen nový dataset, který se skládá z provozu zachyceného na páteřní síti a útoků generovaných pomocí open-source nástrojů. Práce přináší novou metodu pro detekci útoku hrubou silou, která je založena na charakteristikách jednotlivých paketů a používá moderní metody strojového učení. Metoda funguje s šifrovanou HTTPS komunikací, a to bez nutnosti dešifrování jednotlivých paketů. Stále více webových aplikací používá HTTPS pro zabezpečení komunikace, a proto je nezbytné aktualizovat detekční metody, aby byla zachována základní viditelnost do síťového provozu.This thesis presents a review of flow-based network threat detection, with the focus on brute-force attacks against popular web applications, such as WordPress and Joomla. A new dataset was created that consists of benign backbone network traffic and brute-force attacks generated with open-source attack tools. The thesis proposes a method for brute-force attack detection that is based on packet-level characteristics and uses modern machine-learning models. Also, it works with encrypted HTTPS traffic, even without decrypting the payload. More and more network traffic is being encrypted, and it is crucial to update our intrusion detection methods to maintain at least some level of network visibility

A Survey on Big Data for Network Traffic Monitoring and Analysis

Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions

Unveiling flat traffic on the internet: An SSH attack case study

Many types of brute-force attacks are known to exhibit a characteristic ‘flat’ behavior at the network-level, meaning that connections belonging to an attack feature a similar number of packets and bytes, and duration. Flat traffic usually results from repeating similar application-layer actions, such as login attempts in a brute-force attack. For typical attacks, hundreds of attempts span over multiple connections, with each connection containing the same, small number of attempts. The characteristic flat behavior is used by many Intrusion Detection Systems (IDSes), both for identifying the presence of attacks and — once detected — for observing deviations, pointing out potential compromises, for example. However, flatness of network traffic may become indistinct when TCP retransmissions and control information come into play. These TCP phenomena affect not only intrusion detection, but also other forms of network traffic analysis. The contribution of this work is twofold. First, we analyze the impact of retransmissions and control information on network traffic based on traffic measurements. To do so, we have developed a flow exporter extension that was deployed in both a campus and a backbone network. Second, we show that intrusion detection results improve dramatically by up to 16 percentage points once IDSes are able to ‘flatten’ network traffic again, which we have validated by means of analyzing log files of almost 60 hosts over a period of one month

Analyzing the influence of the sampling rate in the detection of malicious traffic on flow data

[EN] Cyberattacks are a growing concern for companies and public administrations. The literature shows that analyzing network-layer traffic can detect intrusion attempts. However, such detection usually implies studying every datagram in a computer network. Therefore, routers routing a significant volume of network traffic do not perform an in-depth analysis of every packet. Instead, they analyze traffic patterns based on network flows. However, even gathering and analyzing flow data has a high-computational cost, and therefore routers usually apply a sampling rate to generate flow data. Adjusting the sampling rate is a tricky problem. If the sampling rate is low, much information is lost and some cyberattacks may be neglected, but if the sampling rate is high, routers cannot deal with it. This paper tries to characterize the influence of this parameter in different detection methods based on machine learning. To do so, we trained and tested malicious-traffic detection models using synthetic flow data gathered with several sampling rates. Then, we double-check the above models with flow data from the public BoT-IoT dataset and with actual flow data collected on RedCAYLE, the Castilla y León regional academic network.S