Traffic analysis is important to the operation of IP networks. The
input to the analysis is raw data such as packet header traces or NetFlow
records and the output is often the size aggregates such as the traffic
generated by various applications or by individual customers. Storing the raw
data allows the flexibility of running arbitrary new analyses in the future,
but the sheer amount of raw data is often a challenge. Sampling based
techniques such as smart sampling aim at reducing the amount of raw data while
preserving the ability of future analyses to accurately estimate the traffic of
any large aggregate. There are three important measures of the traffic of an
aggregate: the number of bytes, the number of packets and the number of flows.
Current data reduction solutions allow estimating only one of these measures.
In this paper we propose the idea of unified summaries that allow the analyses
to get unbiased estimates for all three measures. Our unified summary that
takes as input flow records is based on smart sampling and the one that reads
in packet header traces is based on sample and hold. The most important
contributions of this paper are the development of novel unbiased statistical
estimators for the number of flows, the development of methods for combining
summaries measuring bytes and packets using less memory than separate
summaries, and experimental evaluation of the proposed solutions based on
traces of traffic.Pre-2018 CSE ID: CS2004-079

Estan, Cristian

eScholarship - University of California

UC San DiegoTechnical ReportsTitleUnified Summaries for Internet trafficPermalinkhttps://escholarship.org/uc/item/42x194xhAuthorEstan, CristianPublication Date2004-06-15 Peer reviewedeScholarship.org Powered by the California Digital LibraryUniversity of CaliforniaUnified Summaries for Internet trafficCristian EstanMay 12, 2004AbstractTraffic analysis is important to the operation of IPnetworks. The input to the analysis is raw data suchas packet header traces or NetFlow records and theoutput is often the size aggregates such as the traf-fic generated by various applications or by individualcustomers. Storing the raw data allows the flexibilityof running arbitrary new analyses in the future, butthe sheer amount of raw data is often a challenge.Sampling based techniques such as smart samplingaim at reducing the amount of raw data while pre-serving the ability of future analyses to accuratelyestimate the traffic of any large aggregate.There are three important measures of the trafficof an aggregate: the number of bytes, the number ofpackets and the number of flows. Current data re-duction solutions allow estimating only one of thesemeasures. In this paper we propose the idea of uni-fied summaries that allow the analyses to get unbi-ased estimates for all three measures. Our unifiedsummary that takes as input flow records is basedon smart sampling and the one that reads in packetheader traces is based on sample and hold. The mostimportant contributions of this paper are the devel-opment of novel unbiased statistical estimators forthe number of flows, the development of methods forcombining summaries measuring bytes and packetsusing less memory than separate summaries, and ex-perimental evaluation of the proposed solutions basedon traces of traffic.1 IntroductionInternet traffic consists of individual conversationscalled flows breaken into individual messages calledpackets. Often analyses aggregate the traffic intogroups based on the fields defining the flows (e.g.web traffic, traffic coming from UCSD), and groupswith large traffic are of most interest. The traffic ofa group (also called aggregate) can be measured inpackets or flows, and the two measures reveal diffe-rent types of important information. Keeping trackof all packets is impractical, and the traffic must besummarized. It is important to develop methods forcompactly summarizing the traffic that allow low er-ror approximate analyses. There are separate so-lutions for generating these summaries for the casewhen traffic is measured in packets and for the casewhen it is measured in flows. In this writeup I pro-pose a solution that supports accurate unbiased anal-yses for both cases.2 Summarization methodsIn this paper I discuss three abstract summariza-tion methods: ordinary sampling [7, 3], sample andhold [6, 4] and smart sampling [2]. Each of thesecomputes in a single pass over the packet headers atraffic summary consisting of flow records. Each ofthese methods has as a tuning knob that allows theuser to trade off the size of the summary (the numberof records) and the accuracy of the analyses one canperform based on them. For simplicity I will assumefor now that the flow records only count the numberof packets. In Section 6 I will discuss counting bytes.The abstract summarization methods discussed here1compute a summary for a sequence of packets withina certain time interval and ignore information of thetiming or ordering of the packets within the interval(the binned model [3]).2.1 Ordinary samplingAs the stream of packets is arriving, packets are sam-pled independently at random with probability p. Forthe sampled packets the entry corresponing to theflow identifier in the packet is looked up in a hash ta-ble and the packet counter associated with that en-try incremented. If there is no entry for the flowthe packet belongs to, one is created and its packetcounter initialized to 1. The flow identifier and thepacket counter form the flow record.2.2 Sample and holdAs the stream of packets is arriving, packets are sam-pled independently at random with probability p. Forthe sampled packets, unless the flow the packet be-longs to already has an entry in the flow table, an en-try is created and the packet counter initialized to 0.For all packets that have an entry, the entry’s packetcounter is incremented by 1. I effect this means thatfor each flow, its packets are sampled independentlyat random with probability p and after one packet issammpled all packets belonging to the flow (includingthe sampled one) are counted.2.3 Smart samplingAs the stream of packets is arriving, we build a hashtable with a record for each active flow. At the endof the interval we have the exact number of pack-ets belonging to each flow and the flow records aresampled independently at random with probabilitiesdepending on their packet count. Flow records withone packet are sampled with probability p. Flowrecords with s packets are sampled with probabilitymin(sp, 1). The packet counts in the sampled recordsare left unchanged.Algorithm Memory ProcessingOrdinary sampling low lowSample and hold low highSmart sampling high highTable 1: Comparison of the resource consumption ofthe three algorithms2.4 Estimating the traffic of aggre-gatesThe user of the summary is interested in estimatingthe size of various aggregates containing one or moreflows. The size can be measured in packets, flows orbytes. When one estimates the size of an aggregate,one first finds all flows in the summary that are partof the aggregate and sums up the individual estimatesfor each of these flows. By the linearity of expecta-tion, the expected value for the estimate of the sizeof the aggregate is the sum of the expected values forthe flow in the aggregate. By the independence ofthe sampling decisions for the flows, the variance inthe estimate for the aggregate is the sum of the vari-ances for individual flows. For the rest of this paperI will focus on the expectation and variance for theindividual flows.3 Brief comparison of summa-rization methodsWhile the three summarization methods are quite dif-ferent, they are part of the same family of algorithmsbased on sampling and hash tables with per flow en-tries. For all three summarization algorithms, if weset p = 1, we obtain the list of all active flows withtheir exact packet counts. For all three algorithms, ifthe traffic mix contains only single packet flows, theoutput is a random sample of the traffic in the formof flow records with packet count of 1. But the lessextreme configurations are more useful and for these,there are differences among the algorithms.The processing and memory costs of these algo-rithms vary as shown in Table 1. Ordinary sampling21 10 100 1000Number of packets in flow s (log scale)00.20.40.60.81Probability that flow appears in summaryOrdinary sampling & sample and holdSmart samplingFigure 1: The probability for a flow getting an entryincreases with the flow size. If all algorithms use thesame sampling rate p, the probability for the creationof the entry is consistently higher for smart sampling,but it is never higher by more than 58.2% (which isachieved for flows of size s = 1/p).uses little memory since it creates entries only for thesampled packets and performs little processing sinceit does table lookups only for the sampled packets.Sample and hold consumes just as little memory asordinary sampling since it also creates entries onlyfor the sampled packets, but it incurs high process-ing costs since it performs a lookup for each packetand updates the counter of the corresponding flow iffound. Smart sampling uses high amounts of mem-ory as it creates an entry for each flow. In the fol-lowing sections we will see that the algorithms withhigher resource consumption also have more accurateresults.When comparing the accuracy of the summariesproduced by algorithms, it is fair to compare sum-maries of the same size, since for each algorithm,larger summaries give more accurate results. Thesize of the summaries depends on the packet samplingprobability p. The expected size of the summary isthe sum of the probabilities to have an entry for allactive flows. Figure 1 shows the probability that aflow of size s has an entry in the summary, which is1 − (1 − p)s for ordinary sampling and sample andhold and min(ps, 1) for smart sampling. The entrycreation probability is consistently higher for smartsampling, but but it is never higher by more than1/(e − 1) = 58.2% which is achieved for flows of sizes = 1/p. Thus in the following sections I will com-pare the three algorithms using the same samplingprobability. Ensuring that the expected report sizeswere the same would require knowledge of the distri-bution of flows sizes. While this introduces a smallbias towards smart sampling (because it will generateslightly larger summaries), it simplifies the compar-ison, since no knowledge of the distribution of flowsizes is needed.4 Estimating packet countsFor ordinary sampling, the packet counter c of a flowof size s has a binomial distribution with parametersp and n. The case when c = 0 corresponds to theentry for the flow not being present in the summary.The unbiased packet count estimate for a flow recordwith c packets is 1/p ·c. The variance of this estimateis s/p(1 − p).For sample and hold, the unbiased estimate for thenumber of packets is c+1/p−1. The variance of thisestimate is 1/p(1/p−1)(1−(1−p)s). See Appendix Afor the details of the analysis.For smart sampling the unbiased estimate for thenumber of packets is max(1/p, c). Its variance iss max(1/p− s, 0) [2].In Figure 2 I compare the relative error (definedas the ratio of the standard deviation of the esti-mate for the flow size and the actual flow size) ofthe three algorithms for a packet sampling probabi-lity of p = 1/100 as the flow size s increases from 1to 1,000. For algorithms, the relative error decreasesas the size of the flow increases. Smart sampling’serror drops to 0 when the flow size reaches 1/p, sincefrom there on, we get an exact flow count in the sum-mary. Sample and hold’s error is better than that forordinary sampling and the difference becomes moreand more pronounced as the flow size grows beyond1/p. For s = 1/p, ordinary sampling’s erros is largerthan sample and holds by 1/√1 − e−1 − 1 = 25.8%.Perhaps a more meaningful thing is to look at how310 100 1000Number of packets in flow s (log scale)0123Relative error for packets estimates SD[x]/x Ordinary samplingSample and holdSmart samplingFigure 2: Comparison of the relative error of thepacket estimates as a function of flow size for thethree types of summaries for p = 1/100. The relativeerror is computed as the ratio between the standarddeviation of the estimator and the actual flow size.1 10 100 1000Number of packets in flow s (log scale)00.10.20.30.4Relative error for packets SD[x]/x for 1000 packet aggregateOrdinary samplingSample and holdSmart samplingFigure 3: The relative error for an aggregate of 1000packets decreases as the size of the flows increases ifwe use sample and hold or smart sampling, while itstays constant if we use ordinary sampling.the relative error in the estimate of an aggregate ofsize 1,000 packets changes as the size of the flows thatmake it up increases from 1 to 1,000 packets. Thisis what Figure 3 shows. We can see how the errorfor ordinary sampling is not affected by the flow sizewhile it decreases as the flow sizes go up for sampleand hold and smart sampling.5 Estimating flow countsIt has been proven [1] that one cannot get unbiasedestimates for the number of flows from a random sam-ple of the packets. Therefore we cannot estimate thenumber of flows from based on the ordinary samplingsummary.There is an estimator for the number of flows thatis based on the sample and hold summary, and it isan original contribution of this paper. To estimatethe number of flows in an aggregate, one finds allthe matching flow records and counts the flows thathave a packet counter larger than 1 as 1 flow andthose that have a packet counter of exactly one packetas 1/p flows. It is obvious that this works for flowswith 1 packet. It is plausible that it works for flowswith s >> 1/p. Can we prove that it is an unbiasedestimator for all flow sizes? With a probability α thatdepends on the size of the flow, one of the packetsbefore the last one gets sampled. In this case, thecontribution to the flow count of the estimate is 1,so the estimator is unbiased. With probability 1− αnone of the packets before the last one gets sampled.Within this case, with probability p the last packetis sampled and our flow contributes 1/p to the totaland with probability 1 − p it is not sampled and itcontributes 0, thus the expectation is p · 1/p + (1 −p) · 0 = 1. The variance of this unbiased estimator is(1 − p)s−1(1/p − 1) as shown in Appendix A.For smart sampling, the estimate for the numberof flows is simple. Since the flow record has the ex-act size of the flow s, we know the exact probabilitythat the flow got selected: min(1, ps). Therefore, wecount each flow in the summary as 1/ min(1, ps) flowsfor the estimate. The variance of this estimator ismax(0, 1/(ps) − 1).Figure 4 compares the relative error of sample and41 10 100 1000Number of packets in flow s (log scale)012345678910Relative error for packets estimates SD[x]/x Sample and holdSmart samplingFigure 4: Smart sampling has lower error for flowcount estimates that sample and hold.hold and smart sampling for the flow count for a oneflow aggregate as the flow size increases. While thesample and hold estimator is unbiased, it is clearlyless accurate that that based on smart sampling forflows shorter than 1/p = 100.6 Estimating byte countsWith ordinary sampling we can add a byte counter tothe flow records. Multiplying the value of this bytecounter by 1/p gives an unbiased estimator for thenumber of bytes in the flow. If there is no limit onmaximum packet size, the variance of this estimatorcan be unbounded. If packets have size at most bmaxbytes, the variance of the estimate for the number ofbytes of a flow with s packets is at most b2maxs/p(1−p). The flows with large packets would have highervariance for their byte counts than the flows withsmall packets.For sample and hold we can also add a byte counterto the entry, but there is no unbiased estimator forthe number of bytes in a flow unless we make assump-tions about the sizes of the packets in the flow (e.g.all packets have the same size). Another solution isto run a separate instance of sample and hold thatsamples bytes with probability q. One can estimatethe number of bytes in a flow by adding 1/q − 1 tothe counter and the variance analysis from the packetcase also carries over. The only problem is that in-stead of having one summary, we will have two: onefor packet counts and one for byte counts.For smart sampling, we can also count the numberof bytes in the flow records. Since we know the exactsampling probability for the packet ps, multiplying byits inverse gives us an unbiased estimate for the num-ber of bytes. However, just like with ordinary sam-pling, the variance of this estimate depends on sizeof the packets of the flow: flows with smaller pack-ets will have lower variance than flows with largerpackets. Like for sample and hold, we can run twoinstances for the algorithm, one tht provides unbiasedlow variance estimates for packet counts and one forbytes.6.1 Sharing between byte and packetsummariesBy correlating the choice of which record we samplefor the packet and byte summaries, we can reducethe total size of the summary without affecting theaccuracy of either of them.7 Adding multiple summariesAn important question is computing the sum of mul-tiple summaries. For example we have separate sum-maries for the hours of the day and we want to com-pute a summary for the whole day.8 ConclusionsReferences[1] S. Chaudhuri, R. Motwani, and V. Narasayya.Random sampling for histogram construction:How much is enough? In Proceedings of the ACMSIGMOD, 1998.[2] Nick Duffield, Carsten Lund, and Mikkel Tho-rup. Charging from sampled network usage.In SIGCOMM Internet Measurement Workshop,November 2001.5[3] Cristian Estan, Ken Keys, David Moore, andGeorge Varghese. Building a better netflow.In Proceedings of the ACM SIGCOMM, August2004.[4] Cristian Estan and George Varghese. New di-rections in traffic measurement and accounting.In Proceedings of the ACM SIGCOMM, August2002.[5] Philippe Flajolet. On adaptive sampling. In Com-puting, volume 34, pages 391–400. 1990.[6] Phillip B. Gibbons and Yossi Matias. Newsampling-based summary statistics for improvingapproximate query answers. In Proceedings ofthe ACM SIGMOD, pages 331–342, June 1998.[7] Sampled netflow. http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/120s11/12s sanf.htm.A Analysis detailsI focus the analysis on one flow of size s packets. Sinceour estimate for the size of an aggregate (whethertraffic is measured in packets or flows) is the sum ofthe contribution of individual flows, by the linearityof expectation we can obtain the expected size of theaggregate by adding the expected contributions of theindividual flows. Since the packet sampling decisionsare independent, the variance in the estimate for theaggregate is the sum of the variances of the contribu-tions of individual flows. Let i < s be the number ofpackets missed before an entry for our flow is createdand pi = p(1− p)i be the probability to miss exactlyi packets.Let x be the number of packets our flow contributesto an aggregate. What is its expected value? We willprove by induction that E[x] = s.Base case If s=1, the packet is sampled with pro-bability p and in that case it is counted as 1+1/p−1 =1/p packets. With probability 1−p it is not sampled(and it counts as 0). Thus E[x] = p ·1/p+0 = 1 = s.Inductive step By induction hypothesis we knowthat for s′ = s − 1, E[x′] = s′ = s − 1.E[x] = p · (s + 1/p− 1) + (1 − p)E[x′]= ps + 1 − p + (1 − p)(s − 1)= ps + 1 − p + s − ps − 1 + p = sLet y be the number of flows our flow contributesto an aggregate (it can be 0, 1, or 1/p) . What isits expected value? We can compute it by lookingseparately at the case where one packet before thelast is sampled and at the case where no packet beforetha last one is sampled (and this last case hgas twosubcases based on whether the last packet is sampledor no). Let α =∑s−2i=0pi = 1 − (1 − p)s−1 be theprobability of catching one of the packets before thelast one.E[y] = α · 1 + (1 − α)(p · 1/p + (1 − p)0) = 1What are the variances of x and y?E[x2] =1p(1p− 1)(1 − (1 − p)s) + s2Base case If s=1, E[x2] = p(1/p)2+0 = 1/p. Also1/p(1/p− 1)(1 − (1 − p)1) + 12 = 1/p− 1 + 1 = 1/p.Inductive step By induction hypothesis we knowthat for s′ = s − 1, E[x′2] = 1/p(1/p − 1)(1 − (1 −p)s′) + s′2 = 1/p(1/p− 1)(1 − (1 − p)s−1) + (s − 1)2.6E[x2] = p(1p− 1 + s)2+ (1 − p)E[x′2]=1p+ 2(s − 1) + p(s − 12) + (1 − p)(s − 1)2+(1 − p)1p(1p− 1)(1 − (1 − p)s−1)=1p− 1 + 1 + 2(s − 1) + (s − 12)+(1 − p)1p(1p− 1) −1p(1p− 1)(1 − p)s=1p(1p− 1)(p + 1 − p) + s2−1p(1p− 1)(1 − p)s=1p(1p− 1)(1 − (1 − p)s) + s2V AR[x] = E[x2] − E[x]2 =1p(1p− 1)(1 − (1 − p)s)E[y2] = α · 1 + (1 − α)(p (1/p)2+ (1 − p) · 0)= α + (1 − α)1/pV AR[y] = E[y2] − E[y]2 = α + (1 − α)1/p − 1= (1 − α)(1p− 1)= (1 − p)s−1(1p− 1)7

English

Unified Summaries for Internet traffic

https://escholarship.org/uc/item/42x194xh

Unified Summaries for Internet traffic

Abstract

Similar works

Full text

Available Versions

eScholarship - University of California