Most memory efficient distributed super points detection on core networks

The super point, a host which communicates with lots of others, is a kind of special hosts gotten great focus. Mining super point at the edge of a network is the foundation of many network research fields. In this paper, we proposed the most memory efficient super points detection scheme. This scheme contains a super points reconstruction algorithm called short estimator and a super points filter algorithm called long estimator. Short estimator gives a super points candidate list using thousands of bytes memory and long estimator improves the accuracy of detection result using millions of bytes memory. Combining short estimator and long estimator, our scheme acquires the highest accuracy using the smallest memory than other algorithms. There is no data conflict and floating operation in our scheme. This ensures that our scheme is suitable for parallel running and we deploy our scheme on a common GPU to accelerate processing speed. We also describe how to extend our algorithm to sliding time. Experiments on several real-world core network traffics show that our algorithm acquires the highest accuracy with only consuming littler than one-fifth memory of other algorithms

Distributed super point cardinality estimation under sliding time window for high speed network

Super point is a special kind of host whose cardinality, the number of contacting hosts in a certain period, is bigger than a threshold. Super point cardinality estimation plays important roles in network field. This paper proposes a super point cardinality estimation algorithm under sliding time window. To maintain the state of previous hosts with few updating operations, a novel counter, asynchronous time stamp (AT), is proposed. For a sliding time window containing k time slices, AT only needs to be updated every k time slices at the cost of 1 more bit than a previous state-of-art counter which requires $log_2(k+1)$ bits but updates every time slice. Fewer updating operations mean that more AT could be contained to acquire higher accuracy in real-time. This paper also devises a novel reversible hash function scheme to restore super point from a pool of AT. Experiments on several real-world network traffic illustrate that the algorithm proposed in this paper could detect super points and estimate their cardinalities under sliding time window in real time.Comment: 13 page