James Roberts

Leonardo Linguaglossa

Luigi Iannone

Massimo Gallo

Paolo Giaccone

Tianzhu Zhang

English

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)

04 August 2020POLITECNICO DI TORINORepository ISTITUZIONALEA benchmarking methodology for evaluating software switch performance for NFV / Zhang, Tianzhu; Linguaglossa,Leonardo; Roberts, James; Iannone, Luigi; Gallo, Massimo; Giaccone, Paolo. - ELETTRONICO. - (2019), pp. 251-253.((Intervento presentato al convegno IEEE Conference on Network Softwarization (NetSoft) tenutosi a Parigi nel June 28,2019.OriginalA benchmarking methodology for evaluating software switch performance for NFVieeePublisher:PublishedDOI:10.1109/NETSOFT.2019.8806695Terms of use:openAccessPublisher copyrightcopyright 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all otheruses, in any current or future media, including reprinting/republishing this material for advertising or promotionalpurposes, creating .(Article begins on next page)This article is made available under terms and conditions as specified in the  corresponding bibliographic description inthe repositoryAvailability:This version is available at: 11583/2730203 since: 2019-09-11T10:29:28ZIEEEA benchmarking methodology for evaluatingsoftware switch performance for NFVTianzhu Zhang∗, Leonardo Linguaglossa∗, James Roberts∗, Luigi Iannone∗, Massimo Gallo†, Paolo Giaccone‡∗Network and Computer Science Department (INFRES), Telecom ParisTech, France†Nokia Bell Labs, France‡Department of Electronics and Telecommunications (DET), Politecnico di Torino, ItalyAbstract—Interest in software networking has grown signifi-cantly since the introduction of Network Function Virtualization(NFV). Software switches are used in NFV to steer traffic betweendifferent virtualized network functions and physical NetworkInterface Cards (NICs). It is becoming more and more importantto objectively evaluate and compare the performance of themultiple alternative implementations that have recently beenproposed. A comprehensive performance analysis is still missingfor two main reasons: (i) the amount of time required to configureand compare all such tools is enormous; (ii) it is very difficultto define a proper methodology to compare different solutionsin a fair manner. In this paper we propose a methodologybased on four simple yet representative test scenarios used toevaluate the performance of software switches. We apply thismethodology to measure throughput and latency metrics for 6state-of-the-art software switches namely, OVS-DPDK, snabb,BESS, FastClick, VPP and netmap VALE. Our work constitutesa first step to building a better understanding of design tradeoffsand identifying performance bottlenecks.I. CONTEXTNetwork Function Virtualization (NFV) is expected to havea significant impact on the networking infrastructure, consid-erably reducing CapEx and OpEx by replacing proprietary,expensive and inflexible hardware middleboxes with virtual-ized network functions (VNFs) implemented on commodityservers. Software switches are widely adopted by NFV plat-forms as the dataplane to steer traffic and deliver services.For example, E2 [1] and ParaBox [2] use BESS [3] astheir dataplane while ClickOS [4] and HyperNF [5] adoptnetmap’s VALE switch [6]. Other implementations such asOVS-DPDK [7], snabb [8], and FD.io VPP [9] also aim to de-liver high performance NFV solutions. It is thus of paramountimportance for network operators and service providers tounderstand the performance baselines and to identify thebottlenecks of software switches.In agreement with the recent observations by Fang etal. [10], we believe that evaluating the performance of softwareswitches is in practice a very difficult task. As noted in [10],most works consider the evaluation of a single design at atime [11, 12, 13], or compare just of a small subset of designsin a narrow test scenario [14, 15, 16].We have additionally observed that the metrics chosen tocompare designs are not always sufficient to fully determinewhich one is “best”. For instance, we reproduced a verysimple scenario with a software switch deployed as a L2-5 0 5 10 15 20 25 30 0  10  20  30  40  50T hr ou gh pu t ( Mp ps )Latency (µs)BESSOVS-DPDKVPPFastClicksnabbnetmap 0 10 20 30 40 50 0  10  20  30  40  50S td  d ev ia ti onLatency (µs)BESSOVS-DPDKVPPFastClicksnabbnetmapFig. 1: Scatter plot of latency/throughput and latency/std.deviation. The throughput results shown in the left plot areobtained with bidirectional tests between two 10Gbps NICsusing 64B packets.forwarder between two NICs, and measured overall through-put (bidirectional), latency (round-trip time in µs), and thestandard deviation of the latter. We tried to understand ifthese metrics are correlated, or otherwise stated: is the designwith the highest throughput also offering the smallest latency?Fig. 1 reports our findings. In the left plot, we report measuredthroughput against the latency. It may be seen that there isindeed a negative correlation, implying the switch with highestthroughput is also the one providing the minimal latency.We then plotted the measured latency against the standarddeviation in the measurements: this time no pattern couldbe detected, meaning that there is no correlation between thepacket processing rate and performance stability. For instance,while OVS-DPDK can offer a high processing throughput andlow latency, it does so with a very high variance in the latencymeasurements, thus showing that performance can be ratherunstable.We conclude that it is necessary to provide: (i) a faircomparison for all the state-of-the-art software switches, sinceexisting works consider only one or a small subset of them;(ii) simple yet representative test scenarios that can help usto develop a better understanding of the performance baseline978-1-5386-9376-6/19/$31.00 c©2019 IEEEserverDUTNIC1(a) physical-to-physicalserverDUTNIC1 vNF(b) physical-to-virtualserverDUTvNF(c) virtual-to-virtualserverDUTNIC1vNF(d) loopbackFig. 2: Test scenarios proposed in our paper.and identify the bottleneck. The latter is especially importantgiven the fact that network functions are deployed in virtualenvironments such as virtual machines (VM) and containers.II. METHODOLOGYIn our work, we consider 6 state-of-the-art soft-ware switch implementations, namely OVS-DPDK [7],BESS [3], snabb [8], netmap VALE [6], FastClick [17] andFD.io VPP [9]. All the selected designs are open-sourceprojects with high performance on our testbed. In particular,OVS-DPDK is an OpenFlow (OF) switch performing match-action instructions on a per-packet basis. It can also forwardpackets to a remote OF controller for further processing.BESS, FastClick, Snabb and FD.io VPP are modular tools.They are capable of composing complex services and in-terconnecting VNFs with the network. Netmap’s VALE isa pure layer-2 switch. Other open-source software switchimplementations are excluded either because of very lowperformance (eg., Lagopus [18])1 or high degree of similarity(eg., ClickNF [19])2.We assume VNFs can run either inside virtual machinesor containers and we consider throughput and latency (interms of round-trip time) as the main performance indicators.Each switch implementation, the device under test (DUT),is evaluated with 4 different test scenarios: physical NIC tophysical NIC (p2p), physical NIC to VNF (p2v), VNF to VNF(v2v) and physical NIC - VNF - physical NIC or loopback (lo),as illustrated in Fig. 2 The data flows of the 4 scenarios areexplained as follows:(p2p) physical-to-physical: In this scenario, packets arrive viaa physical NIC, are fetched by the DUT and then forwardedout via the other NIC. Measurements are performed on bothsource and sink of the packet flow, outside the DUT. Thisis the most basic test scenario with no VNFs running in thevirtual environment and provides the baseline results for allsoftware switches.(p2v) physical-to-virtual: As for the p2p scenario, packetsarrive via a physical NIC and fetched by the software switchunder test. They are then forwarded into a simple VNF, wheremeasurements are performed. This scenario demonstrates thetypical hop of an NFV data flow and reflects how efficientlysoftware switches can relay packets from NIC to VNFs.1According to our benchmarking result, Lagopus cannot achieve more than2 Mpps even for the most simple scenario.2ClickNF, just like FastClick, is an extension of the Click Modular Router.Our benchmarking result shows ClickNF achieves similar throughput andlatency with respect to FastClick.(v2v) virtual-to-virtual: In the v2v scenario, packets are gen-erated by a source VNF and injected into the DUT. The DUTthen forwards the packets to a sink VNF for measurement.This scenario demonstrates a DUT’s efficiency in terms oftraffic forwarding between VNFs.(lo) loopback: in the loopback scenario, packets arrive via aphysical NIC and are fetched by the DUT, which forwardsthem to a VNF. The VNF is a lightweight cross-connectfunction that forwards the received packets from one virtualinterface to the DUT via another virtual interface. The DUTthen proceeds to forward the packets to the other NIC. Thisis a typical NFV scenario with a single VNF on the servicechain and provides baseline reference for scenarios of multi-VNF chains.We believe that these four test scenarios provide a base-line performance characterisation for the software switches.The p2p scenario provides the baseline result showing howefficiently software switches can process packets betweenphysical NICs. The p2v and v2v scenarios, representing twotypical dissected hops of NFV data flows, allow us to analyseand identify potential performance bottlenecks. The loopbackscenario provides results for a service chain with a single VNFand can be used as a basis to predict the performance of multi-VNF chains.III. EVALUATION PLATFORMExperiments are conducted on a commodity server runningLinux 4.8.0-41-generic distribution. The server is equippedwith 2 Intel Xeon E5-2690 v3 @ 2.60GHz CPUs (each with24 physical cores), using 32k/256k/30720K L1-3 caches, and2 Intel 82599ES dual-port 10-Gbps NICs. All the VNFs arehosted by QEMU/KVM virtual machines. We use Moon-Gen [20] as traffic generator, FlowWatcher-DPDK [21] asVNF for p2v and v2v scenarios to measure the throughput.DPDK l2fwd [22] is the VNF used to forward packets inthe loopback scenario. For the packet exchange between VMsand software switches, we adopt the netmap passthrough [23]exclusively for the netmap VALE switch, and the vhost-userprotocol [24] for the other considered software switches. Bothenable zero-copy packet delivery through shared memory. Forall the experiments, we use identical settings for the virtualmachines and software switches to ensure a fair comparison.We pin threads of the software switches to isolated CPU coresand disable Turbo Boost to reduce performance variance. 0 2 4 6 8 10 12 14 16BESS FastClick OVS-DPDK snabb VPP netmap14.88 MppsP ac ke t pr oc es si ng  r at e [ Mp ps ]p2p p2v v2v loopbackFig. 3: preliminary throughput test results for all scenarios.Note that these results are obtained with unidirectional tests,thus limited up to 14.88 Mpps for 64B packets.IV. EXPERIMENTAL RESULTSIn this section, we provide some preliminary results for thethroughput test. Fig. 3 illustrates the unidirectional throughputfor the considered software switches in the scenarios describedin Fig.2. As we observe, no single software switch prevails inthe considered scenarios. For example, BESS outperforms theothers in the p2v case thanks to its efficient implementationof packet delivery to virtual machines. Snabb prevails in thev2v scenario because its vhost-user implementation is moreefficient for packet delivery between virtual machines.ACKNOWLEDGMENTSThis work has been carried out at LINCS (http://www.lincs.fr) and benefited from support of NewNet@Paris, Cisco’sChair “NETWORKS FOR THE FUTURE” at Telecom ParisTech(https://newnet.telecom-paristech.fr).REFERENCES[1] S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Ratnasamy, L. Rizzo,and S. Shenker, “E2: a framework for NFV applications,” in Proceedingsof the 25th Symposium on Operating Systems Principles. ACM, 2015,pp. 121–136.[2] Y. Zhang, B. Anwer, V. Gopalakrishnan, B. Han, J. Reich, A. Shaikh,and Z.-L. Zhang, “Parabox: Exploiting parallelism for virtual networkfunctions in service chaining,” in Proceedings of the Symposium on SDNResearch. ACM, 2017, pp. 143–149.[3] S. Han, K. Jang, A. Panda, S. Palkar, D. Han, and S. Ratnasamy,“SoftNIC: A software NIC to augment hardware,” 2015.[4] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco,and F. Huici, “Clickos and the art of network function virtualization,”in 11th {USENIX} Symposium on Networked Systems Design andImplementation ({NSDI} 14), 2014, pp. 459–473.[5] K. Yasukata, F. Huici, V. Maffione, G. Lettieri, and M. Honda, “Hypernf:Building a high performance, high utilization and fair nfv platform,” inProceedings of the 2017 Symposium on Cloud Computing. ACM, 2017,pp. 157–169.[6] L. Rizzo and G. Lettieri, “VALE, a switched Ethernet for virtualmachines,” in International conference on Emerging networking exper-iments and technologies. ACM, 2012, pp. 61–72.[7] “Open vSwitch with DPDK,” http://docs.openvswitch.org/en/latest/intro/install/dpdk/.[8] M. Paolino, N. Nikolaev, J. Fanguede, and D. Raho, “SnabbSwitchuser space virtual switch benchmark and performance optimization forNFV,” in 2015 IEEE Conference on Network Function Virtualizationand Software Defined Network (NFV-SDN). IEEE, 2015, pp. 86–92.[9] “VPP - fd.io,” https://wiki.fd.io/view/VPP.[10] V. Fang, T. Lvai, S. Han, S. Ratnasamy, B. Raghavan, and J. Sherry,“Evaluating software switches: Hard or hopeless?” EECS Department,University of California, Berkeley, Tech. Rep. UCB/EECS-2018-136,2018.[11] L. Linguaglossa, D. Rossi, S. Pontarelli, D. Barach, D. Marjon, andP. Pfister, “High-speed data plane and network functions virtualizationby vectorizing packet processing,” Computer Networks, vol. 149, pp.187 – 199, 2019.[12] P. Emmerich, D. Raumer, F. Wohlfart, and G. Carle, “Performancecharacteristics of virtual switching,” in 2014 IEEE 3rd InternationalConference on Cloud Networking (CloudNet). IEEE, 2014, pp. 120–125.[13] B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme,J. Gross, A. Wang, J. Stringer, P. Shelar et al., “The design andimplementation of Open vSwitch,” in USENIX Symposium on NetworkedSystems Design and Implementation (NSDI), 2015, pp. 117–130.[14] Z. Niu, H. Xu, Y. Tian, L. Liu, P. Wang, and Z. Li, “BenchmarkingNFV software dataplanes,” arXiv preprint arXiv:1605.05843, 2016.[15] G. Lettieri, V. Maffione, and L. Rizzo, “A survey of fast packetI/O technologies for network function virtualization,” in InternationalConference on High Performance Computing. Springer, 2017, pp. 579–590.[16] N. Pitaev, M. Falkner, A. Leivadeas, and I. Lambadaris, “Characteriz-ing the performance of concurrent virtualized network functions withOVS-DPDK, FD.IO VPP and SR-IOV,” in ACM/SPEC InternationalConference on Performance Engineering. ACM, 2018, pp. 285–292.[17] T. Barbette, C. Soldani, and L. Mathy, “Fast userspace packet process-ing,” in 2015 ACM/IEEE Symposium on Architectures for Networkingand Communications Systems (ANCS). IEEE, 2015, pp. 5–16.[18] “Lagopus switch and router,” http://www.lagopus.org/.[19] M. Gallo and R. Laufer, “ClickNF: a modular stack for custom networkfunctions,” in USENIX Annual Technical Conference (ATC)), 2018, pp.745–757.[20] P. Emmerich, S. Gallenmu¨ller, D. Raumer, F. Wohlfart, and G. Carle,“Moongen: A scriptable high-speed packet generator,” in Proceedings ofthe 2015 Internet Measurement Conference. ACM, 2015, pp. 275–287.[21] T. Zhang, L. Linguaglossa, M. Gallo, P. Giaccone, and D. Rossi,“FlowMon-DPDK: Parsimonious per-flow software monitoring at linerate,” in TMA Conference, 2018.[22] “L2 forwarding sample application,” https://doc.dpdk.org/guides-18.08/sample app ug/l2 forward real virtual.html.[23] V. Maffione, L. Rizzo, and G. Lettieri, “Flexible virtual machinenetworking using netmap passthrough,” in 2016 IEEE InternationalSymposium on Local and Metropolitan Area Networks (LANMAN).IEEE, 2016, pp. 1–6.[24] “Features/VirtioVhostUser,” https://github.com/qemu/qemu/blob/master/docs/interop/vhost-user.txt.

A benchmarking methodology for evaluating software switch performance for NFV

https://iris.polito.it/retrieve/handle/11583/2730203/240661/1570535220.pdf

A benchmarking methodology for evaluating software switch performance for NFV

Abstract

Similar works

Full text

Available Versions

PORTO@iris (Publications Open Repository TOrino - Politecnico di Torino)