Berger, Michael Stübert

Dittmann, Lars

Fagertun, Anna Manolova

Ruepp, Sarah Renée

Rytlig, Andreas

Wessing, Henrik

This paper proposes and analyzes the efficiency of novel backpressure schemes for Terabit switch fabrics. The proposed schemes aim at buffer optimization under uniform traffic distribution with Bernoulli packet arrival process. Results show that a reduction of the needed maximum buffer capacity with up to 47% can be achieved with switch-internal backpressure mechanisms at the expense of a small control overhead

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 20, 2017Internal Backpressure for Terabit Switch FabricsFagertun, Anna Manolova; Ruepp, Sarah Renée; Rytlig, Andreas; Berger, Michael Stübert; Wessing,Henrik; Dittmann, LarsPublished in:I E E E Communications LettersLink to article, DOI:10.1109/LCOMM.2011.112311.111791Publication date:2012Link back to DTU OrbitCitation (APA):Fagertun, A. M., Ruepp, S. R., Rytlig, A., Berger, M. S., Wessing, H., & Dittmann, L. (2012). InternalBackpressure for Terabit Switch Fabrics. I E E E Communications Letters, 16(2), 265-267. DOI:10.1109/LCOMM.2011.112311.1117911Internal Backpressure for Terabit Switch FabricsAnna V. Manolova, Member, IEEE, Sarah Ruepp, Member, IEEE Andreas Rytlig,Michael Berger, Member, IEEE, Henrik Wessing, Member, IEEE, and Lars Dittmann, Member, IEEEAbstract—This paper proposes and analyzes the efficiency ofnovel backpressure schemes for Terabit switch fabrics. The pro-posed schemes aim at buffer optimization under uniform trafficdistribution with Bernoulli packet arrival process. Results showthat a reduction of the needed maximum buffer capacity withup to 47% can be achieved with switch-internal backpressuremechanisms at the expense of a small control overhead.Index Terms—Terabit switches, 100 Gigabit Ethernet, back-pressure, flow control, switch architecture.I. INTRODUCTIONTHE Internet is increasingly populated by bandwidth-demanding applications. Following the evolution of 10Gigabit Ethernet (GE), 100 Gigabit Ethernet is currentlyemerging as a promising candidate to fulfill the request forincreased line speed.Switching 100 GE requires that the selected switch ar-chitecture is scalable enough to accommodate high capacitytransmission [1], as the 100 GE line speed quickly aggregatesto the Terabit scale within the switch fabric. Single-stageswitch fabrics, e.g. crossbar switches, do not scale up to aTerabit system. Hence, a multi-stage switch fabric, where thetraffic can be distributed on different chips, is a promisingapproach. In particular, the Clos’ architecture relies on threestages of switching modules, where each module connects toall the modules in the adjacent stages via a unique path [2],as illustrated in Figure 1. Each of these modules may thenbe constructed as an individual crossbar switch. However,the sheer amount of data that must be switched in a Terabitsystem faces a number of obstacles. Since the different trafficflows arriving at the input of a switch are independent ofeach other, it may happen that two cells are destined for thesame destination simultaneously. This causes contention andrequires the use of queues and buffers and to avoid packet loss.However, the available memory in a Terabit switching systemis scarce. In order to avoid buffer overflow and associatedpacket loss, backpressure (BP) mechanisms can be applied [3].In general two types of flow control can be applied: inter-nally (link-level) between the individual stages (modules) ofa switch; and end-to-end between the input and output trafficmanagers [4]. Another classification can be done based onthe operation. Backpressure (BP) is the simplest flow control,which is based on sending a queue status indicator froma downstream queue to an upstream queue: 0 (operationalstatus) and 1 (overloaded condition). Under such a scheme,This work has been partially supported by the Danish Advanced TechnologyFoundation (Højteknologifonden) through the research project ”The Road to100 Gigabit Ethernet”.A. V. Manolova, S. Ruepp, A. Rytlig, M. Berger, H. Wessing and L.Dittmann are with DTU-Fotonik, Technical University of Denmark, Kgs. Lyn-gby, Denmark; e-mail: fanva,srru,s052800,msbe,hewe,laditg@fotonik.dtu.dkthe upstream queue stops sending traffic to the particulardownstream queue [3]–[5]. Credit based flow control schemesare another alternative, where downstream queues signal backto upstream queues special ”credits” which are used by theupstream queues in order to decide if and how much traffic canbe forwarded downstream [6], [7]. This type of flow control ismore fine-grained since it operates on per virtual connection(per flow) basis. Despite its flexibility and efficiency theapproach requires higher complexity. Book-keeping and creditcontrol for switches with many inputs/outputs become a hurdleespecially at high speeds.In this paper, we propose and evaluate the performance ofa novel BP scheme for Terabit switches. The most significantcontribution of our work is that unlike the standard queue-to-queue operation of the BP schemes, our scheme is queue-to-module, where the controlled module does not employ queue-ing. This gives us the ability to apply finer analysis and control.We consider a specific Clos network architecture, which influ-ences the design of the mechanisms. Furthermore, we focuson link-level BP which facilitates independent implementationof the switch fabric and the traffic manages and makes inter-operability easy. Last, we use discrete-event simulation for theevaluation of the BP schemes, not mathematical formulationsas in the majority of the existing work.The remainder of the paper is organized as follows: Sec-tion II explains the switch architecture. Section III detailsthe proposed BP mechanisms. In Section IV the simulationstudy is presented and in Section V the results are discussed.Section VI concludes the paper.II. SWITCH ARCHITECTUREWe focus on a Clos-based packet switch architecture dueto its potential to provide adequate throughput at highbit rates [8]. Clos switches have three stages whichcan be either buffered or bufferless. Depending in whichstage the memory is allocated, different architecturesare possible, e.g., Space-Memory-Space, Memory-Space-Memory [9], Memory-Memory-Memory, Space-Memory-Memory (SMM) [8] (adopted in this work). For the first archi-tecture to be practical advanced schedulers are required [10],whereas the second architecture requires complex dispatchingschemes [9]. The Memory-Memory-Memory is considered thebest performing Clos architecture, achieving 100% throughput,but it has been suggested recently that the memory in the firststage is not necessary [8]. This led to the consideration ofthe SMM architecture where the purpose of the buffers in thecentral modules is to resolve contention amongst cells fromdifferent input modules and the scheduling in the bufferlessfirst stage is reduced to a simple load balancing scheme.2Several Input Module (IM) connection matrix configurationsfor the SMM architecture are possible [11]. The Desynchro-nized Static Round Robin (DSRR) [8] and the simple Randomscheme are dynamic control schemes, where the connectionmatrix changes every time-slot. Such schemes though increasethe complexity of the system and introduce an undesired out-of-sequence cell delivery, which leads to increased end-to-enddelay [11]. In this work we adopt a static IM connectionmatrix configuration, which results in a cheaper switch withreduced requirements for re-sequencing buffers (since no out-of-sequence cell delivery is observed), which lowers the end-to-end delay [11]. Under this scheme, the inputs of an IM arestatically (permanently) connected to specific outputs. We useVirtual Output Queueing (VOQ) in all Central Modules (CMs)and Output Modules (OMs) (Figure 1), where each modulehas one Input queue per incoming link, which is logicallyseparated in virtual output queues (one queue per output linkfrom the module). Furthermore, simple round-robin schedulersare used for traffic forwarding in CMs and OMs.III. BACKPRESSURE SCHEMESOur proposed BP signals are operated between the CMs andthe IMs (see Figure 1). An advantage of keeping the BPoperation within the switch is that it is independent of thetraffic manager. Unlike standard BP schemes, which operateon a queue-to-queue association, the BP we propose operateson a queue-to-module association, where a downstream queuecontrols the connectivity matrix of an upstream IM. In thisway we achieve buffer overflow control in the CMs via trafficredistribution (i.e. load balancing) amongst the other CMs andnot through buffering as such. A BP signal is sent when thequeue length in a CM exceeds a certain threshold value. Thefollowing BP schemes are proposed: Coarse control:– When a CM detects total buffer overflow (accumu-lated over all Input queues), it sends a 1-bit signalto all IMs which change their connection matrix,following a round-robin principle, i.e. a connectioni ! j is changed to connection i ! j + 1 mod N(N - number of outputs from the IM) (BP coarse); Fine control: When a CM detects a buffer overflow of anInput queue, it sends a 1-bit signal to the correspondingIM, which changes its connection matrix:– by following a round-robin principle (BP fine);– by redirecting the traffic from the line card withthe minimum number of cells, received within thelast 100 time-slots, towards the CM sending the BPsignal (BP fine load deflect);– each time slot following a round-robin principle fora limited duration of time (BP fine load balance).The control method for the BP scheme is time-based. Whena BP signal is sent, a back-off timer is activated during whichno other CM can send a BP signal to a given IM. This isrequired in order for the system to stabilize and the effects ofthe change in the connection matrix to become effective. Forthe BP fine load balance, the back-off timer also controlsthe duration for which the IM changes its connection matrix.Line cards Line cards1nNn x m k x k m x n1nNInput QueueVirtual Output Queues (VOQ)IM CM OMBackpressureIM Connection Matrix configurationsFig. 1. Switch fabric based on 3-stage Clos network. The arrow denotesbackpressure signals. Example VOQ configuration in one CM indicated.IV. SIMULATION STUDYWe evaluate the efficiency of the proposed BP schemes usingOpnet Modeler [12]. We utilize a 9x9 switch with three IMs,three CMs and three OMs. Links run at 100 Gbps speed. Thegenerated packets have uniformly distributed lengths between512 and 12176 bits and are segmented into 256 bit cells forforwarding through the switch. We employ Bernoulli trafficgenerators, i.e., the generated traffic is smoother than Poisson,which corresponds to the application scenario for a core 100GEthernet switch. The traffic is uniformly distributed betweenall source/sink pairs. 100 ms of operation is simulated, re-sulting in amount of generated packets between 5:5  106 and13  106 with a step of 1:5  106 for the indicated loads.The efficiency of the schemes is evaluated in terms of themaximum needed buffer space (in number of cells) withoutcell loss. No upper bound for the Input queues is set, instead -the maximum required buffer space for any of the Input queuesin all CMs is observed. BP is initiated when the buffer space ina CM exceeds a given threshold. The threshold value is set tobe 60% lower than the maximum needed buffer space if no BPis applied, averaged over five independent simulation runs. Theused back-off timer for all schemes is equal to 500 time-slots.Five independent simulation runs have been performed withdifferent random seeds. The presented results are the averageof all runs. 95% confidence intervals have been calculated.The obtained maximum needed buffer size for the differentBP schemes is compared to a case where no BP is used andthe IMs have static configuration. The needed control overhead(number of one-bit signals for BP control) is obtained as well.V. RESULTSFigure 2 illustrates the maximum amount of needed bufferspace (in number of cells) for all investigated schemes. As canbe expected, the Static case without BP has the worst perfor-mance especially at high loads. All BP schemes perform betterthan the Static case, which clearly illustrates their efficiency.In order to exactly evaluate the improvement each BP schemecan offer, Figure 3 presents the average improvement factorin % for each BP scheme compared to the Static case, basedon the obtained average values from Figure 2. We can seethat BP fine load balance significantly outperforms the restof the schemes. This is due to the specifics of the scheme’soperation, where the load from all input line cards, connected3102103 0.4  0.5  0.6  0.7  0.8  0.9Maximum buffer usage [cells]Input load per sourceStaticBP coarseBP fineBP fine load deflectBP fine load balanceFig. 2. Maximum buffer usage among all Input queues of all CMs. 10 15 20 25 30 35 40 45 50 0.3  0.4  0.5  0.6  0.7  0.8  0.9  1Improvement factor (% compared to static)Input load per sourceBP coarseBP fineBP fine load deflectBP fine load balanceFig. 3. Improvement of the CMs’ buffer usage compared to Static scheme.to a single IM is balanced among all CMs for a certain durationof time. The efficiency of the scheme will thus depend onthe duration of the backoff period and the dynamics of theincoming traffic flow. It is interesting to observe that BP fineoutperforms BP fine load deflect (though only marginally),which is due to the fact that modules operate and send BPsignals independently. This lack of coordination may lead tocases where two of the IMs might target their most loadedinterfaces to only one CM, effectively overloading it.Table I shows the cost of using the BP schemes in terms ofcontrol overhead. As expected the coarse-grained BP schemehas the lowest overhead since it activates BP when the totalbuffer size of the CM has exceeded the threshold value(i.e. the reaction is per averaged buffer size for all inputqueues), whereas the fine-grained schemes activate the BP ona per input queue overflow basis. All fine-grained schemesrequire approximately the same amount of control overhead(100   150 kbps). The observed overhead does not followa particular tendency because of the chosen threshold val-ues. Since the threshold value is different for each testedload condition and is correlated to the performance of thestatic case (i.e. it is not an independent value), this leads todifferent conditions for activating the BP schemes for eachload point. Nevertheless, considerably higher overhead for thefine-grained schemes is expected, which is supported by theresults. It is important to notice that more control informationTABLE ICONTROL OVERHEAD (IN KILO BITS) PER BP FOR ALL TESTED LOADS.Input load BP coarse BP fine BP fine BP fineper node load deflect load balance0:4 30:5 0:2 121:6 0:3 124:4 0:3 110:2 0:60:5 27:3 0:3 122:2 0:7 126:7 0:5 108:3 0:30:6 35:1 0:4 140:1 0:4 147:4 0:8 121 0:70:7 29:7 0:6 121:7 1:1 129:9 0:6 107 0:90:8 23:1 0:5 97:8 0:5 107:8 1 90:3 0:60:9 48  0:3 134:1 2:2 158:6 1:4 134:6 1:5does not guarantee better performance. For example, at 0:9input load the BP coarse scheme has a higher improvementfactor than the BP fine load deflect (about 10%) and stillgenerates about 3 times lower overhead. On the other hand, atthe same load BP coarse has lower improvement factor thanBP fine load balance (about 10%), but has 2:8 times loweroverhead. Since the statistical aspects of the results are nottaken into account (only the average values), none of the BPschemes can be conclusively declared to be the best, especiallyat high loads.The standard trade-off between efficiency and cost is evi-dent. The overall evaluation of the efficiency of the schemesincludes not only the achieved performance but also theircomplexity and the application scenario. The BP schemesneed to be properly tuned to the given traffic conditions forachieving the best possible performance.VI. CONCLUSIONIn this paper, we propose and evaluate the efficiency of fournovel backpressure flow control schemes for SMM Clos-based Terabit switches. Unlike existing schemes our proposalsoperate on a queue-to-module principle. The main focus of theschemes is buffer-space optimization which is paramount forimproved system integration, scalability and cost-efficiency.Simulation results show that at the cost of about 110kbps con-trol overhead we can achieve between 35% to 47% reductionin the needed maximum buffer space.REFERENCES[1] C. Hermsmeyer et al., “Towards 100G packet processing: Challengesand technologies,” Bell Labs Technical Journal, vol. 14, no. 2, 2009.[2] C. Clos, “A study of non-blocking switching networks,” Bell SystemsTechnical Journal, pp. 406–424, 1953.[3] T. Kanazawa et al., “Input and Output Queueing Packet Switch withBackpressure Mode for Loss Sensitive Packets in Threshold Scheme,”in IEEE PACRIM, 1997, pp. 527–530.[4] H. J. Chao, “Flow Control in a Multi-Plane Multi-Stage Buffered PacketSwitch,” in High Performance Switching and Routing, HPSR, 2007.[5] F. Chiussi et al. , “Backpressure in shared-memory-based ATM switchesunder multiplexed bursty sources,” in IEEE INFOCOM, 1996.[6] H. T. Kung and R. Morris, “Credit-based flow control for atm networks,”IEEE Network, vol. 9, pp. 40–48, March/April 1995.[7] R. Schoenen and A. Dahlhoff, “Closed Loop Credit-Based Flow controlwith Internal Backpressure in Input and Output Queued Switches,” inHPSR, 2000, pp. 195–203.[8] X. Li, Z. Zhou, and M. Hamdi, “Space-Memory-Memory Architecturefor CLOS-network Packet Switches,” in IEEE ICC, 2005.[9] R. Rojas-Cessa and H. Chao, “Maximum Weight Matching DispatchingScheme in Buffered Clos-Network Packet Switches,” in IEEE ICC, 2004.[10] J. Kleban and S. Piotrowski, “Performance Evaluation of SelectedPacket Dispatching Schemes for the CBC Switches,” in CommunicationsLetters, IEEE, 2009.[11] S. Ruepp et al., “Performance Evaluation of 100 Gigabit EthernetSwitches under Bursty Traffic,” in ONDM, 2011.[12] OPNET Technologies, Inc., http://www.opnet.com.

Internal Backpressure for Terabit Switch Fabrics

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology

Crossref