An Unsupervised Approach to DDoS Attack Detection and Mitigation in Near-Real Time

We present an approach for Distributed Denial of Service (DDoS) attack detection and mitigation in near-real time. The adaptive unsupervised machine learning methodology is based on volumetric thresholding, Functional Principal Component Analysis, and K-means clustering (with tuning parameters for flexibility), which dissects the dataset into categories of outlier source IP addresses. A probabilistic risk assessment technique is used to assign “threat levels” to potential malicious actors. We use our approach to analyze a synthetic DDoS attack with ground truth, as well as the Network Time Protocol (NTP) amplification attack that occurred during January of 2014 at a large mountain-range university. We demonstrate the speed and capabilities of our technique through replay of the NTP attack. We show that we can detect and attenuate the DDoS within two minutes with significantly reduced volume throughout the six waves of the attack

A functional approach to scanner detection

Detecting scanning in Internet traffic is a well-studied topic with no single, definitive approach. Among the proposed methods are two which are widely accepted, but with known limitations: one based on a static fanout ratio, and another on principal component analysis (PCA). We introduce a two-step procedure based on Functional PCA and k-means clustering which we argue provides significantly better robustness and data-driven applicability. We validate and compare using synthetic datasets with ground truth about anomalies on FTP and HTTP port traffic flows; our method identifies all scanners. We also compare approaches using NTP flow data prior to a reflective DDoS attack in 2014, providing a real-world example to illustrate the deficiencies of existing approaches and how they are addressed by our functional framework procedure. Lastly, we discuss insights into the traffic that cannot be obtained by the previous methods