A Quasi-Likelihood Approach for Accurate Traffic Matrix Estimation in a High Speed Network

Knowing the traffic matrix, i.e., packet/byte counts between pairs of nodes in a network, is important for network management. The main challenges for accurate traffic matrix estimation in a high speed network are the computation and memory limitations. In this paper, we propose a novel algorithm for traffic matrix estimation that can yield accurate estimates whereas uses small memory and per packet update overhead. Our algorithm constructs a compact probabilistic traffic digest at each network node, and derives a Quasi Maximum Likelihood Estimate (Quasi-MLE) of the traffic matrix by correlating the traffic digests received at a central location. Our new approach is highly efficient, requiring no prior knowledge of the exact packet size distributions. We derive accurate approximation of the relative error distribution of our estimate. For an origin-destination (OD) pair (o, d), we show that by using an array of size M for each traffic digest at o and d, the relative estimation standard error is O(M − 12 (σo + σd) 1 2), where σo, σd are the noise-to-signal ratios, defined as the ratios of non-OD packet/byte counts to OD packet/byte counts at the origin and destination. This is superior to the state-of-the-art algorithms, especially for large σo and σd, where the estimation is more challenging. We further demonstrate the effectiveness of our approach using both model and real Internet trace-driven simulations

Enhancing distributed traffic monitoring via traffic digest splitting.

Lam, Chi Ho.Thesis (M.Phil.)--Chinese University of Hong Kong, 2009.Includes bibliographical references (leaves 113-117).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.viChapter 1 --- Introduction --- p.1Chapter 1.1 --- Motivation --- p.1Chapter 1.2 --- Organization --- p.4Chapter 2 --- Related Works and Background --- p.7Chapter 2.1 --- Related Works --- p.7Chapter 2.2 --- Background --- p.9Chapter 2.2.1 --- Datalite --- p.9Chapter 2.2.2 --- Proportional Union Method --- p.14Chapter 2.2.3 --- Quasi-Likelihood Approach --- p.18Chapter 3 --- Estimation Error of Existing TD-based TMA schemes --- p.24Chapter 3.1 --- Error Accumulation and Amplification of Existing Schemes --- p.25Chapter 3.1.1 --- Pu --- p.25Chapter 3.1.2 --- Qmle --- p.26Chapter 3.1.3 --- Datalite --- p.26Chapter 3.2 --- Estimation Error of 3-sets intersection cases --- p.28Chapter 3.2.1 --- Pu --- p.28Chapter 3.2.2 --- Datalite --- p.30Chapter 4 --- Error Reduction Via Traffic Digest Splitting --- p.36Chapter 4.1 --- Motivation --- p.36Chapter 4.2 --- Objective Functions for Optimal TD-splitting --- p.39Chapter 4.3 --- Problem Formulation of Threshold-based Splitting --- p.41Chapter 4.3.1 --- Minimizing Maximum Estimation Error --- p.42Chapter 4.3.2 --- Minimizing R.M.S. Estimation Error --- p.46Chapter 4.4 --- Analysis of Estimation Error Reduction Via Single-Level TD-splitting --- p.48Chapter 4.4.1 --- Noise-to-signal Ratio Reduction --- p.49Chapter 4.4.2 --- Estimation Error Reduction --- p.52Chapter 4.5 --- Recursive Splitting --- p.56Chapter 4.5.1 --- Minimizing Maximum Estimation Error --- p.57Chapter 4.5.2 --- Minimizing R.M.S. Estimation Error --- p.59Chapter 5 --- Realization of TD-splitting for Network Traffic Measurement --- p.61Chapter 5.1 --- Tracking Sub-TD Membership --- p.64Chapter 5.1.1 --- Controlling the Noise due to Non-Existent Flows on a Target Link --- p.64Chapter 5.1.2 --- Sub-TD Membership Tracking for Single-level TD-splitting --- p.65Chapter 5.1.3 --- Sub-TD Membership Tracking under Recursive Splitting --- p.66Chapter 5.2 --- Overall Operations to support TD-splitting for Network-wide Traffic Measurements --- p.67Chapter 5.2.1 --- Computation Time for TD-splitting --- p.69Chapter 6 --- Performance Evaluation --- p.72Chapter 6.1 --- Applying TD-splitting on Generic Network Topology --- p.72Chapter 6.1.1 --- Simulation Settings --- p.73Chapter 6.1.2 --- Validity of the Proposed Surrogate Objective Functions --- p.75Chapter 6.1.3 --- Performance of Single-level TD-splitting --- p.77Chapter 6.1.4 --- Performance of Recursive TD-splitting --- p.88Chapter 6.1.5 --- Heterogeneous NSR Loading --- p.95Chapter 6.2 --- Internet Trace Evaluation --- p.99Chapter 6.2.1 --- Simulation Results --- p.100Chapter 7 --- Conclusion --- p.105Chapter A --- Extension of QMLE for Cardinality Estimation of 3-sets Intersection --- p.107Bibliography --- p.11