Towards Coordinated, Network-Wide Traffic Monitoring for Early Detection of DDoS Flooding Attacks

DDoS flooding attacks are one of the biggest concerns for security professionals and they are typically explicit attempts to disrupt legitimate users' access to services. Developing a comprehensive defense mechanism against such attacks requires a comprehensive understanding of the problem and the techniques that have been used thus far in preventing, detecting, and responding to various such attacks. In this thesis, we dig into the problem of DDoS flooding attacks from four directions: (1) We study the origin of these attacks, their variations, and various existing defense mechanisms against them. Our literature review gives insight into a list of key required features for the next generation of DDoS flooding defense mechanisms. The most important requirement on this list is to see more distributed DDoS flooding defense mechanisms in near future, (2) In such systems, the success in detecting DDoS flooding attacks earlier and in a distributed fashion is highly dependent on the quality and quantity of the traffic flows that are covered by the employed traffic monitoring mechanisms. This motivates us to study and understand the challenges of existing traffic monitoring mechanisms, (3) We propose a novel distributed, coordinated, network-wide traffic monitoring (DiCoTraM) approach that addresses the key challenges of current traffic monitoring mechanisms. DiCoTraM enhances flow coverage to enable effective, early detection of DDoS flooding attacks. We compare and evaluate the performance of DiCoTraM with various other traffic monitoring mechanisms in terms of their total flow coverage and DDoS flooding attack flow coverage, and (4) We evaluate the effectiveness of DiCoTraM with cSamp, an existing traffic monitoring mechanism that outperforms most of other traffic monitoring mechanisms, with regards to supporting early detection of DDoS flooding attacks (i.e., at the intermediate network) by employing two existing DDoS flooding detection mechanisms over them. We then compare the effectiveness of DiCoTraM with that of cSamp by comparing the detection rates and false positive rates achieved when the selected detection mechanisms are employed over DiCoTraM and cSamp. The results show that DiCoTraM outperforms other traffic monitoring mechanisms in terms of DDoS flooding attack flow coverage

A Data Streaming Algorithm for Estimating Subpopulation Flow Size Distribution ∗ ABSTRACT

Statistical information about the flow sizes in the traffic passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and improve network performance through traffic engineering. Previous work on estimating the flow size distribution for the complete population of flows has produced techniques that either make inferences from sampled network traffic, or use data streaming approaches. In this work, we identify and solve a more challenging problem of estimating the size distribution and other statistical information about arbitrary subpopulations of flows. Inferring subpopulation flow statistics is more challenging than the complete population counterpart, since subpopulations of interest are often specified a posteriori (i.e., after the data collection is done), making it impossible for the data collection module to “plan in advance”. Our solution consists of a novel mechanism that combines data streaming with traditional packet sampling to provide highly accurate estimates of subpopulation flow statistics. The algorithm employs two data collection modules operating in parallel — a NetFlow-like packet sampler and a streaming data structure made up of an array of counters. Combining the data collected by these two modules, our estimation algorithm uses a statistical estimation procedure that correlates and decodes the outputs (observations) from both data collection modules to obtain flow statistics for any arbitrary subpopulation. Evaluations of this algorithm on real-world Internet traffic traces demonstrate its high measurement accuracy

A Data Streaming Algorithm for Estimating Subpopulation Flow Size Distribution ∗ ABSTRACT

Statistical information about the flow sizes in the traffic passing through a network link helps a network operator to characterize network resource usage, infer traffic demands, detect traffic anomalies, and improve network performance through traffic engineering. Previous work on estimating the flow size distribution for the complete population of flows has produced techniques that either make inferences from sampled network traffic, or use data streaming approaches. In this work, we identify and solve a more challenging problem of estimating the size distribution and other statistical information about arbitrary subpopulations of flows. Inferring subpopulation flow statistics is more challenging than the complete population counterpart, since subpopulations of interest are often specified a posteriori (i.e., after the data collection is done), making it impossible for the data collection module to “plan in advance”. Our solution consists of a novel mechanism that combines data streaming with traditional packet sampling to provide highly accurate estimates of subpopulation flow statistics. The algorithm employs two data collection modules operating in parallel — a NetFlow-like packet sampler and a streaming data structure made up of an array of counters. Combining the data collected by these two modules, our estimation algorithm uses a statistical estimation procedure that correlates and decodes the outputs (observations) from both data collection modules to obtain flow statistics for any arbitrary subpopulation. Evaluations of this algorithm on real-world Internet traffic traces demonstrate its high measurement accuracy

Robust and Scalable Sampling Algorithms for Network Measurement

Recent growth of the Internet in both scale and complexity has imposed a number of difficult challenges on existing measurement techniques and approaches, which are essential for both network management and many ongoing research projects. For any measurement algorithm, achieving both accuracy and scalability is very challenging given hard resource constraints (e.g., bandwidth, delay, physical memory, and CPU speed). My dissertation research tackles this problem by first proposing a novel mechanism called residual sampling, which intentionally introduces a predetermined amount of bias into the measurement process. We show that such biased sampling can be extremely scalable; moreover, we develop residual estimation algorithms that can unbiasedly recover the original information from the sampled data. Utilizing these results, we further develop two versions of the residual sampling mechanism: a continuous version for characterizing the user lifetime distribution in large-scale peer-to-peer networks and a discrete version for monitoring flow statistics (including per-flow counts and the flow size distribution) in high-speed Internet routers. For the former application in P2P networks, this work presents two methods: ResIDual-based Estimator (RIDE), which takes single-point snapshots of the system and assumes systems with stationary arrivals, and Uniform RIDE (U-RIDE), which takes multiple snapshots and adapts to systems with arbitrary (including non-stationary) arrival processes. For the latter application in traffic monitoring, we introduce Discrete RIDE (D-RIDE), which allows one to sample each flow with a geometric random variable. Our numerous simulations and experiments with P2P networks and real Internet traces confirm that these algorithms are able to make accurate estimation about the monitored metrics and simultaneously meet the requirements of hard resource constraints. These results show that residual sampling indeed provides an ideal solution to balancing between accuracy and scalability