Injection of IP packet loss is a versatile method for emulating real-world
network conditions in performance studies. In order to reproduce realistic
packet-loss patterns, stochastic fault-models are used. In this report we
desribe our implementation of a Linux kernel module using a Continuous-Time
Gilbert Model for packet-loss injection

Dräger, Matthias

Reinecke, Philipp

Wolter, Katinka

Institutional Repository of the Freie Universität Berlin

NetemCG – IP Packet-Loss Injection using aContinuous-Time Gilbert ModelPhilipp Reinecke, Matthias Dra¨ger, and Katinka WolterTR-B-11-05September 2011NetemCG – IP Packet-Loss Injection using aContinuous-Time Gilbert ModelPhilipp Reinecke, Matthias Dra¨ger, and Katinka WolterFreie Universita¨t BerlinInstitut fu¨r InformatikTakustraße 914195 Berlin, Germany{philipp.reinecke, matthias.draeger, katinka.wolter}@fu-berlin.deAbstract. Injection of IP packet loss is a versatile method for emu-lating real-world network conditions in performance studies. In order toreproduce realistic packet-loss patterns, stochastic fault-models are used.In this report we desribe our implementation of a Linux kernel moduleusing a Continuous-Time Gilbert Model for packet-loss injection.1 IntroductionTestbed-driven performance and dependability evaluation of distributed systemsrequires the emulation of a realistic networking environment. Common distur-bances such as packet loss may have an adverse effect on both dependabilityand performance of the network. As illustrated in e.g. [1, 2], packet loss mayaffect dependability of higher networking and system layers even with the reli-able TCP protocol, which guarantees reliable data transmission. Consequently,methods for reproducing disturbances are required.Injection of IP packet loss is an indispensible tool when evaluating fault-tolerant network protocols. Fault injection at the IP level also provides a con-venient way for assessing performance and reliability of distributed systems inwhich the network stack is considered just an off-the-shelf component. As the in-jected faults force the network stack to apply the same fault-handling proceduresthat are executed under real-world operating conditions (such as e.g. packet re-transmissions in TCP), the same operational patterns of the network stack thatare present in a real network can be expected to emerge in the testbed.IP packet loss patterns have been the focus of intense study in the pastdecades (e.g. [3–5]). While in the simplest case packet loss may be describedby a Bernoulli model, packet loss is often comprised of bursts of elevated lossprobability, which can be modelled more closely with Gilbert-Elliot (GE) mod-els [3–5]. A Gilbert-Elliot model describes the loss process as a Markov Chainwith different loss probabilities for each state.There exist a number of fault injection modules for the Linux kernel. TheBernoulli loss model is implemented by the default Linux kernel module Netem.11 Note that Netem supports the injection of other disturbances as well as a simplecorrelated loss model, but these are out of scope for this report.Machine A Machine CModuleFIMachine BFig. 1. General approach to packet-level fault-injection.In [6] the default module was modified such that it allows replaying packet-losstraces. The Netem-CLG (Correlated Loss Generator) module [7] extends thedefault Netem implementation by a discrete-time extended Gilbert-Elliot model.In this report we describe our implementation of a continuous-time Gilbert-Elliot model for packet-loss generation in the Netem module. We start with thetheoretical background in Section 2. We then describe our modifications to thedefault module in Section 3 before illustrating installation and use of the modulein Section 5.2 BackgroundThe most straightforward way of injecting packet loss according to a loss processdescribed by a loss model is depicted in Figure 1: One computer is set up toact as a router between two or more hosts or networks. On this machine, theTCP/IP stack is augmented by a fault-injection (FI) module. For each arrivingoutside packet the FI module determines whether to forward the packet to itsdestination or to discard it, according to the loss model. With a Bernoulli lossmodel, for each packet one trial is performed and the packet is dropped accordingto the outcome of this trial. With a Markovian (or Gilbert) model, the droppingprobability for the Bernoulli trial is selected according to the current state ofthe model when the packet arrives. Markovian models have the advantage thatthey enable accurate modelling of elevated loss models or loss bursts.Markovian loss models, often called Gilbert models are typically defined asDiscrete-Time Markov Chains (DTMCs) (e.g. [5]). The left part of Figure 2shows an example of a two-state discrete-time Markovian loss model with lossprobabilities l1 and l2 and state transition probabilities p12 and p21. In a DTMCloss model, state changes occur at discrete time intervals. Certainly, this view isappropriate for modelling the loss process as described by a packet trace: Eachpacket constitutes a time instant at which a state-change may occur. Implemen-tation of fault-injection using a discrete-time model is straightforward as well:With each arriving packet the next state is selected randomly from the possiblesuccessor states (including the same state) of the current state.In the NetemCG module we take an alternative view, in that we assumea Continous-Time Markov Chain (CTMC), instead of a DTMC. Here, stateλ12p211 − p21p121 − p12(s1, l1) (s2, l2) (s1, l1) (s2, l2)λ21Fig. 2. DTMC and CTMC models of the packet-loss process.sojourn times are described by exponentially distributed random variables. Theright-hand side of Figure 2 illustrates a two-state continuous-time Markovianloss model, again with loss rates l1, l2. Transition rates are λ12 and λ21. Packet-loss injection according to a continuous-time Gilbert model is implemented byplaying the CTMC, as follows: Upon entering a state, the state sojourn time isdrawn from an exponential distribution with rate equal to the sum of outgoingrates. When this time has passed, the next state is chosen based on the embeddedMarkov chain.The main difference between the DTMC and CTMC models is that the for-mer only perform state-changes if and when a packet arrives, whereas state-changes in the latter occur independently of the arrival process. That is, theformer actually relies on a DTMC embedded in the arrival process, whereas thelatter is independent of the arrival process. While both models are equally capa-ble of describing a packet-loss trace, they have different effects on packet streamswhose arrival rate changes in response to packet loss, as illustrated in [8].3 ImplementationWe implemented support for the continuous-time Gilbert model as a modifi-cation of the Netem module. The model is specified in a text file and sent tothe Netem module via the tc command from the iproute2 package [9]. Somehands-on examples for using the module are shown in Section 5. In the follow-ing subsections we describe a few of the implementation challenges, and ourapproaches to solving them.3.1 Parameter specificationA loss model in NetemCG consists of at least one node, and an arbitrary (in-cluding zero) number transitions. It is possible to create a loss model with anarbitrary number of nodes and transitions. Both the default Netem module andthe NetemCLG module only allow for small models with fixed size and thusrequire only a few parameters for specifying the fault process. These parame-ters can easily be specified on the command-line of the tc tool. However, thissimplicity comes at the cost of allowing only a limited number of states. For theNetemCG module, where we wanted to be able to specify an arbitrary numberModelDescription = { ‘\#’ { Comment } ‘\n’| { ‘‘ ’’ ‘\n’ }| Node ‘\n’};Comment = { Any character };Node = NodeNumber LossProbability {[ TransitionList ]} ‘\n’};NodeNumber = Any natural number;LossProbability = Any real number in [0,1];TransitionList = { Transition };Transition = (NodeNumber ‘,’ TransitionRate, ‘,’ Scale);TransitionRate = Any natural number;Scale = Any integer number;Fig. 3. NetemCG Input File Syntaxof states, parameter specification on the command line would have been imprac-tical. Therefore, we augmented the tc tool such that it loads model descriptionsfrom a text file and transmits them to the kernel module. This was made diffi-cult by the fact that the routines used for communication with the module arelimited with respect to the size of the data to be transmitted. Note that a sim-ilar problem was encountered by the authors of the trace-driven approach [6],where the trace had to be loaded into the kernel module. In [6], an additionalproblem was the size of the trace, which could be larger than the amount ofmemory available to the module. The solution applied by [6] was to load thetrace sequentially, sending a new chunk on-demand. In NetemCG, we need thecomplete model inside the module at any time, and cannot load it on-demand.We implemented loading the model into the kernel as follows: First, theCTMC model described in the input file is parsed and the Embedded MarkovChain (EMC) as well as the mean sojourn times for each state are computed. Wethen transmit the number of nodes, the list of nodes (specified by their sojourntimes) and finally the EMC to the kernel, using the Netlink interface.The syntax of the model file is straightforward: Each line can contain eitherwhitespace, a comment, or the description of a node. Comments are specifiedby starting the line with a #. Nodes are specified by the number of the node,followed by the loss probability (in [0, 1]) and the list of transitions in brackets.Each transition is a tuple of the number of the target state, the transition rate,and the exponent of the scale parameter (more on that in the next section).Figure 3.1 gives the EBNF description of the input file syntax.3.2 State TransitionsThe loss process is generated in the kernel module as follows: First, each time apacket arrives a Bernoulli experiment is performed using the current loss prob-ability. Second, packet loss probabilities change according to the CTMC lossmodel, i.e. every time the CTMC enters a new state, that state’s loss proba-bility becomes the current loss probability. The CTMC is played based on theEMC and the state sojourn rates. Upon entering a state, the module draws anexponentially-distributed random number with parameter equal to that state’ssojourn rate, and sets a kernel timer using this value. When the timer expires,the module chooses the next state based on the outgoing probabilities in theEMC, and enters the next state. Initially, the module is in the first state givenin the input file.3.3 Kernel LimitationsSince the module runs within the Linux kernel, a few limitations do apply. First,the kernel does not support floating-point numbers. This requires mapping ofrandom numbers in the range [0, 1] to the range of long integers ([0, 232 − 1]).Second, in order to be able to specify loss period lengths in different ranges, weintroduced a scale parameter s. The scale parameter is a natural number in theinterval [0, 9] and is used as the negative exponent to base 10 when scaling therate. Third, in order to generate random variates with exponential distributionefficiently, we applied the inversion method. Unfortunately, the kernel does notsupport the logarithm operation. We remedied this by implementing the loga-rithm using a table of pre-computed logarithm values, applying binary search toidentify the required value.Note that some practical limits are imposed by both the kernel and therange of values for the variables: First, the resolution of the kernel timer limitsthe minimum sojourn time for a state. With the default resolution of 250Hz, theminimum sojourn time is 4ms. The maximum sojourn time is bounded by theMAX INT value, which is typically 232 − 1. Second, the size of data structures inthe tc command limits the maximum mean sojourn time that can be specifiedto 109 ms. Third, the sum of all outgoing rates for a node must be less than 232to avoid overflows.4 Compilation and Installation of NetemCGWe provide the source code as patches to the Linux kernel source tree and theiproute2 package. The patches are available at http://cst.mi.fu-berlin.de/staff/reinecke.html.To patch the Netem module a Linux kernel version 2.6.35 and newer is re-quired. The patch can be applied as follows:cd <Linux kernel source>patch -p1 < netemcg-kernel.patchThe new kernel version can then be configured, compiled, and installed in theusual way. Note that the Netem module must be selected and that kernel modulesmust be compiled as well.Likewise, the iproute2 package needs to be patched bycd <iproute2 source directory>patch -p1 < netemcg-iproute2.patchp1 p2λ1λ2Fig. 4. A simple generic model.Compilation then consists of running make. The new version may be installedusing make install, although tc can also be run directly from the source di-rectory.5 ExamplesWe start with a discussion of the model file syntax. Consider the model givenin Figure 4. The model consists of two states with loss probability p1 and p2 andtransition rates λ1 and λ2. In order to specify this model for use by NetemCG,we assign numbers to the states and then give the transitions for each node. Inthis example, the model can be described by the following lines:1 p1 [(2, λ1 ,0)]2 p2 [(1, λ2 ,0)]Note the trailing zero in each transition specification. This is the value ofthe scale parameter si for the transition rate. The scale parameter allows thespecification of transition rates less than 1, since the effective transition rate iscomputed asλ′i= λi · 10−si .To illustrate this further, let us now assume that λ1 = 0.05, λ2 = 0.95, p1 = 0,and p2 = 1.0. The resulting model file example1.model is1 0.0 [(2,5,2)]2 1.0 [(1,95,2)]If the model is saved in the file example1, it can be loaded into NetemCGwith the following commandtc qdisc add dev eth0 root netem model example1.modelFinally, let us consider a larger loss model, as shown in Figure 5. This modelcontains multiple states and multiple transitions. Note that state 2 has twooutgoing transitions. We can specify the model as1 0.0 [(2,2,0)]2 1.0 [(1,5,1)(3,5,1)]3 0.5 [(1,3,0)]0%100%50%λ1 = 2λ2 = 0.5λ3 = 0.5λ4 = 3Fig. 5. Example 2: Loss model with multiple nodes und multiple transitions.6 ConclusionIn this report we gave a description of the NetemCG module for injecting packetloss using a continuous-time Gilbert-model with arbitrary structure and size.References1. Reinecke, P., van Moorsel, A.P.A., Wolter, K.: The Fast and the Fair: A Fault-Injection-Driven Comparison of Restart Oracles for Reliable Web Services. In:QEST ’06: Proceedings of the 3rd International Conference on the QuantitativeEvaluation of Systems, Washington, DC, USA, IEEE Computer Society (2006) 375–3842. Reinecke, P., Wolter, K.: Phase-Type Approximations for Message TransmissionTimes in Web Services Reliable Messaging. In Kounev, S., Gorton, I., Sachs, K.,eds.: Performance Evaluation – Metrics, Models and Benchmarks. Volume 5119 ofLecture Notes in Computer Science., Springer (June 2008) 191–2073. Zhang, Y., Paxson, V., Shenker, S.: The Stationarity of Internet Path Properties:Routing, Loss, and Throughput. ACIRI Technical Report (2000)4. Zhang, Y., Duffield, N., Paxson, V., Shenker, S.: On the Constancy of InternetPath Properties. In: IMW ’01: Proceedings of the 1st ACM SIGCOMM Workshopon Internet Measurement, New York, NY, USA, ACM (2001) 197–2115. Hohlfeld, O., Geib, R., Haß linger, G.: Packet Loss in Real-Time Services: MarkovianModels Generating QoE Impairments. In: Proc. of the 16th International Workshopon Quality of Service (IWQoS). (June 2008) 239–2486. Keller, A., Baumann, R., Fiedler, U.: TCN Trace Control for Netem.http://tcn.hypert.net/ (2009) (last seen 6 June 2011).7. Salsano, S., Ludovici, F., Ordine, A.: Definition of a general and intuitive loss modelfor packet networks and its implementation in the Netem module in the Linux kernel.Technical report, University of Rome “Tor Vergata”8. Reinecke, P., Wolter, K.: On Stochastic Fault-Injection for IP-Packet Loss Emu-lation. In: Proceedings of the 8th European Performance Engineering Workshop –EPEW 2011. (2011) (to appear).9. Hubert, B., Graf, T., Maxwell, G., van Mook, R., van Oosterhoutand P. Schroeder,M., Spaans, J., Larroy, P.: Linux Advanced Routing and Traffic Control.http://lartc.org Last visited August 18th, 2011.

NetemCG – IP packet-loss injection using a continuous-time Gilbert model

https://refubium.fu-berlin.de/bitstream/fub188/19073/1/paper.pdf

NetemCG – IP packet-loss injection using a continuous-time Gilbert model

Abstract

Similar works

Full text

Available Versions

Institutional Repository of the Freie Universität Berlin