A k-ary n-cube interconnect architecture is proposed, as an off-chip communications architecture for line cards, to increase the throughput of the currently used memory system. The k-ary n-cube architecture allows multiple packet processing elements on a line card to access multiple memory modules. The main advantage of the proposed architecture is that it can sustain current line rates and higher while distributing the load among multiple memories. Moreover, the proposed interconnect can scale to adopt more memories and/or processors and as a result increasing the line card processing power. Our results portray that k-ary n-cube sustained higher incoming traffic load while keeping latency lower than its shared-bus competitor. © 2005 IEEE

Engel, Jacob

Kocak, T

English

Koçak, T

Explore Bristol Research

K-ary n-cube based off-chip communications architecture for high-speed packet processors

A k-ary n-cube interconnect architecture is proposed, as an off-chip communications architecture for line cards, to increase the throughput of the currently used memory system. The k-ary n-cube architecture allows multiple packet processing elements on a line card to access multiple memory modules. The main advantage of the proposed architecture is that it can sustain current line rates and higher while distributing the load among multiple memories. Moreover, the proposed interconnect can scale to adopt more memories and/or processors and as a result increasing the line card processing power. Our results portray that k-ary n-cube sustained higher incoming traffic load while keeping latency lower than its shared-bus competito

                          Engel, J., & Kocak, T. (2006). K-ary n-cube based off-chip communicationsarchitecture for high-speed packet processors. 1903 - 1906.10.1109/MWSCAS.2005.1594497Link to published version (if available):10.1109/MWSCAS.2005.1594497Link to publication record in Explore Bristol ResearchPDF-documentUniversity of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only the publishedversion using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/about/ebr-terms.htmlTake down policyExplore Bristol Research is a digital archive and the intention is that deposited content should not beremoved. However, if you believe that this version of the work breaches copyright law please contactopen-access@bristol.ac.uk and include the following information in your message:• Your contact details• Bibliographic details for the item, including a URL• An outline of the nature of the complaintOn receipt of your message the Open Access Team will immediately investigate your claim, make aninitial judgement of the validity of the claim and, where appropriate, withdraw the item in questionfrom public view.K-ary n-cube Based Off-Chip CommunicationsArchitecture for High-Speed Packet ProcessorsJacob Engel and Taskin KocakDepartment of Electrical and Computer EngineeringUniversity of Central FloridaOrlando, FL 32816e-mail: {jengel, tkocak}@cs.ucf.eduAbstract— A k-ary n-cube interconnect architecture is pro-posed, as an off-chip communications architecture for line cards,to increase the throughput of the currently used memory system.The k-ary n-cube architecture allows multiple packet processingelements on a line card to access multiple memory modules.The main advantage of the proposed architecture is that it cansustain current line rates and higher while distributing the loadamong multiple memories. Moreover, the proposed interconnectcan scale to adopt more memories and/or processors and asa result increasing the line card processing power. Our resultsportray that k-ary n-cube sustained higher incoming traffic loadwhile keeping latency lower than its shared-bus competitor.I. INTRODUCTIONIn this paper, we present an off-chip k-ary n-cube in-terconnect that provides an effective solution, under certainconstraints, to the increasing demand for memory bandwidthon line cards. Memory bandwidth is affected by two majorfactors: higher line rates and increase in deep packet inspec-tion operations. Network line rates are constantly increasing,currently reaching 40 Gb/sec. Line cards are required toperform multiple functions to support new services whichincreases the traffic overhead to incoming line rates by 40%-100%. Moreover, there exist an unstoppable expansion inlookup tables which requires more memory space and moreparameters to examine within each packet.In most line card architectures there exist a direct inter-connect, such as busses or switches, that connects differentprocessing elements to memory modules. The heart of theline card is the network processor which performs differentoperations in order to analyze the flow of incoming packets.The nature of packet processing requires frequent read/writeoperations to memories which are distributed around the NPU.As line-rate data and NPU processing power increase,memory access time becomes the main system bottleneck(it requires a minimum of 8 ns processing time at 40 Gbpsline-rate) during data store/retrieve operations. The growingdemand for memory bandwidth contrasts the notion of di-rect interconnect methodologies and replaced it with indirect,packet-based networks such as mesh, torus or k-ary n-cubes.A shared bus cannot scale well as the number of modules(processing elements or memories) connected to it increases.In addition, it requires an arbitration mechanism that becomesdistributed (rather than centralized) as the number of modulesconnected to the bus grows.Solutions to the memory bandwidth bottleneck are limitedby area on the line card and NPU I/O pins. Pin constraintsbound the bus size that can be interfaced with the NPU.Hence, only a packet-based network-on-board can provide therequired performance improvement between the NPU and off-chip memory modules.II. K-ARY n-CUBE INTERCONNECT ARCHITECTUREA k-ary n-cube network consists of N = kn nodes, where nrepresents the dimension of the network and k represents thenumber of nodes in each dimension of the structure. Each nodeis connected to all of its nearest neighbors via bi-directionalchannels. The address/location of a node can be representedas a vector consists of two bit-vector fields. One bit vectorrepresents the location of the node within its plane. This vectorcan repeat its value within other planes. The second vectorrepresents the node dimension location. For example, node mcan have an address vector with mi being the node’s locationin its dimension.The k-ary n-cube architecture shown in Fig. 1 is a packet-based multiple path interconnect that allows network packetsto be shared by different processor and memory modules onthe network line card. Memories are distributed around pro-cessing elements, such as traffic manager, QoS co-processor,classification processor, to allow data sharing among modulesand direct processor memory storage. If a link goes down, notonly should the fault be limited to the link, the additional linksfrom the intermediate nodes should ensure the connectivitycontinues.A. Routing mechanismThe routing algorithm routes a packet from a source de-vice s = {s1, s2, ..., sm} to a destination module d ={d1, d2, ..., dn}, by choosing a direction to travel in each ofthe three dimensions to balance the channel load. A memorypacket sent by a processing element (PE) will always attemptto take the shortest path as long as packets are admissible(accepted by ideal nodes). If a node is oversubscribed (i.e.,all ports are occupied), packets in transit will take a differentroute using the traffic controller (TC) in each corner (node).The architecture path diversity offers alternative paths betweensource and destination modules.19030-7803-9197-7/05/$20.00 © 2005 IEEE.Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on July 9, 2009 at 05:55 from IEEE Xplore.  Restrictions apply.Fig. 1. k-ary n-cube architecture on the line cardIn wormhole routing, the header flit is sent first. While theheader propagates through the interconnect, it sets the nodeswitches in a certain position corresponding to the traffic loadon the node’s channels. The rest of the packets comprisingthe message are transmitted in a pipeline manner followingthe message header. The main advantage in using wormholerouting in this k-ary n-cube structure is that it diminishes thelatency as the size of the message increases while increasingits throughput. The major part of the latency is hidden in thetransfer of the first packet. The rest of the packets follow itand introduce only wire transfer delay. As the message sizeincreases, the ratio of consecutive latencies decreases.Before a node forwards a packet to one of its adjacentnodes, it polls the status of each node. The traffic controllerat each node has a “sensor status” flag which determines ifthe node is currently processing a packet (i.e., busy), or if thenode is idle and ready to accept a new packet (i.e., not busy).Depending on its direction preferences (some nodes may gethigher preference than others if they are located closer to thepacket’s destination), TC will chose an admissible node toforward the packet. If at least one adjacent node is availableto forward a packet, it will require only one clock cycle to doso.III. PERFORMANCE EVALUATIONWe use standard performance metrics such as latency toevaluate the k-ary n-cube interconnect performance. Latency isdefined as the time it takes for a complete message to reach itsdestination. The load or offered-load is the number of packetsinjected into the interconnect network per second and dependson the processing elements which are generating them.The k-ary n-cube architecture and performance measures arebased on the following assumptions:• Messages are uniformly distributed among modules.• Nodes are generating traffic independently of each other.• It takes only one propagation delay cycle for a flit tomove from one node to another.• Each physical channel consists of five virtual channels.A. Latency of packet-switched multiprocessor shared-busBus Performance, in terms of latency and throughput, of ashared bus is affected by the following factors:• Number of processors or memories connected to it (as thenumber of modules connected to the shared-bus increases,performance degrades)• The shared-bus length (average is between 5-12 inches).• Shared-bus width (increasing width results in highercost).Most shared bus systems do not have more than 30 processors/ memories. Communication protocol is trivial and is basedon simple connection oriented mechanism. In addition, sharedbus systems require arbitration mechanism to send/receivedata from multiple modules. Bus arbitration adds delay to theoverall system latency. Some systems use multiple buses toreduce the effect of the factors mentioned above. Althoughthe bandwidth of the multiple bus architecture is higher thanthat using a single shared bus, the system is more costlyand requires complex bus arbitration logic. As a result ofthe shared bus disadvantages many multi processor-memorysystems include multiple shared buses and/or other type ofinterconnects.The system illustrated in Fig. 2 describes a multiple-processor (multiple-memory) packet-based shared bus. Aprocessor-memory communication is only allowed when thebus is not in use by other devices.Fig. 2. Shared-bus multiple processor with arbitrationA processor wishes to communicate with a memory receivespermission to send data from the arbiter. The data is transferredusing flits. The bus width determines the flit size. Whenmultiple processors wish to use the bus it can cause contention.Therefore, the bus is being monitored by the arbiter and onlythe arbiter grants processors access to the bus. Bus of this typefabricated on a PCB line card has an average length between5 inches to 12 inches. The bus length is an important factorin determining the packet propagation delay through the bus(not including queuing, arbitration or transfer time).Lshared−bus =Mw∗ Tw ∗ l +Wq (1)Equ. 1 represents the latency of a shared bus. The first com-ponent embodies the delay associated with message propaga-tion from source to destination through the bus. Mw representsthe number of flits to send, Tw (=254 psec) is the time it1904Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on July 9, 2009 at 05:55 from IEEE Xplore.  Restrictions apply.takes for a packet to propagate in 1 inch [7]. l representsthe bus length. The second latency component, Wq, is due toqueuing within each module and signifies the average waitingtime between message transmissions of the same module.Each module can transmit a message in its turn. Messagesare queued in each source until released.Both arbitration latency and average wait in queue com-ponents provide the queuing latency. For example, in FCFSscheme the queuing latency is equal to (N-1)*r [6]. N sym-bolizes the number of modules (processors and/or memories)connected to the bus. The value of r varies in each systemdepending on the queuing model which represents the bus typeand communicating modules connected to it. The (N-1) termrepresents the number of turns (bus cycles) a device must waitbefore it can receive bus access again. A packet-based multipleprocessors shared bus can be modelled as M/D/1 queue. M/D/1queue model has a Poisson input (random bus access requestsby processors) to a single-server queue with constant servicetimes (this is the bus throughput), unlimited capacity and FIFOqueue discipline. Arrival rate, λ, is the rate in which processorrequires memory access and depends on the incoming line rate.The arrival rate we use represent an aggregate arrival rate, mg ,which includes the amount of traffic (in flits) generated by anode per cycle per second. Note that,PE∑i=1λi < µ (2)The sum of all arrival rates cannot exceed the service rate ofthe system. Moreover, λ is used in our model as an aggregatevalue of all issued communications by PEs to the shared-bus.Service rate, µ, is the rate in which the bus can service apacket once a processor granted access to the bus. This rate isdetermined by the bus throughput (bits per second). µ includesthe arbitration cycles which each module is required to waitbefore receiving access to the bus. That is,µ =1(N − 1) ∗ r (3)where,r =Mw∗ Tw ∗ l (4)The average waiting time of a message in its generatingnode queue is denoted by the following equationWq =ρ2 ∗ µ ∗ (1− ρ) (5)B. K-ary n-cube latency equationThe latency model for k-ary n-cubes was initially modeledby M. O. Khaoua in [1] and was developed also in [3][4][5].Under uniform traffic pattern, a message passes on average k¯hops across the networkk¯ =(k − 1)2(6)The average distance of this message isd¯ = n ∗ k¯ (7)The mean message latency consists of two parts, the delaydue to message transmission and the time a message spendsif blocking occurs. For an average of d¯ hops from source todestination, latency can be expressed asLatencyk−ary,n−cube = M + d¯+d¯∑i=1Bi +Wej (8)The first term, M, is the message length in flits. Bi isthe average blocking time seen by a message at any ithhop, where (1 ≤ i ≤ d¯). Wej represents the mean waitingtime between message transmission at the ejection node. Inthis equation, it is assumed that channel to channel transfertime is 1 unit cycle. V virtual channels are used per physicalchannel in the model introduced by [2][9]. Although virtualchannels are divided into two types adaptive and deterministic,there is no distinction between them when computing virtualchannels occupancy probabilities [8]. A message is blockedat any ith hop channel when all virtual channels are busy. IfWb denote the average waiting time due to blocking and Pbrepresents the probability of blocking then the mean blockingtime expression isBi = Pbi ∗Wb (9)The blocking probability, Pb, is determined by calculatingthe probability that all virtual channels are busy. Virtualchannels analysis and analytical model is discussed in [1].Since multiple virtual channels share the bandwidth of thesame physical channel, a multiplexer is required to selectwhich virtual channel will use the physical channel. Thismultiplexer functions in a TDM manner and adds the followingcomponent to the latency equationV¯ =V∑i=0i2 ∗ PiV∑i=0i ∗ Pi(10)The mean waiting time at any node within the message’s pathis given asWb =mc ∗ S2 ∗ (1 + (S−M)2S2 )2 ∗ (1−mc ∗ S) (11)where, the latency of the k-ary n-cube network, S, is measuredin cycles and mc is the traffic rate on a given channel and isexpressed asmc =mg ∗ d¯n(12)The mean waiting time of a message at any ejection channelis given as1905Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on July 9, 2009 at 05:55 from IEEE Xplore.  Restrictions apply.Wej =mg ∗M22 ∗ (1−mg ∗M) (13)The mean waiting time in a source node is determined bymodeling the injection channel at the source node as an M/G/1queue givesWs =mgV ∗ S2 ∗ (1 + (S−M)2S2 )2 ∗ (1− mgV ∗ S)(14)The term mgV denotes the mean arrival rate.The overall message latency is composed of the sum of themean network latency, S, and the mean waiting time at thesource node, Ws, multiplied by the multiplexing factor V¯ toaccount for the virtual channels multiplexing that takes placeover the physical channel. Thus,Lmsg = (S +Ws) ∗ V¯ (15)C. Performance results k-ary n-cube interconnect vs. shared-busPerformance of k-ary n-cube is compared against a shared-bus. Since line cards contain multiple network processingelements and can reach 64 co-processors (including memorymodules), we chose two combinations of k-ary n-cubes whichcomprised of 64 nodes and evaluate their performance. Bothshared-bus and the k-ary n-cube have 32 bits channel width.Fig. 3 compares the latency of 8-ary 2-cube network with4-ary 3-cube network while traffic load increases. Moreover,performance for both types of k-ary n-cubes were analyticallycomputed using 32-flits and 64-flits messages.0 0.0005 0.001 0.0015 0.002 0.0025 0.003050010001500Traffic (messages/cycle)  Latency (cycles)8−ary 2−cube 64 flits4−ary 3−cube 64 flits8−ary 2−cube 32 flits4−ary 3−cube 32 flitsFig. 3. Latency comparison between 4-ary 3-cube and 8-ary 2-cubeFig. 3 depicts that 4-ary 3-cube network was superior to 8-ary 2-cube network with respect to load vs. latency for both 32and 64 flits message. Once the better k-ary n-cube was chosenwe compared its performance against a shared-bus (Fig. 4).Fig. 4 portrays latency comparison results for shared busvs. 4-ary 3-cube network. In both interconnects the channel10−7 10−6 10−5 10−4 10−3 10−2 10−110−910−810−710−610−510−4Traffic (messages/cycle)Latency (seconds)4−ary 3−cubeshared−busFig. 4. Latency comparison between 4-ary 3-cube and shared-buswidth is 32-bits. 4-ary 3-cube network was able to sustainmuch higher traffic rate while keeping lower latency than itscompetitor the shared-bus. Moreover, the 4-ary 3-cube networkmaintained its exceptional latency for both low as well as hightraffic loads.IV. CONCLUSIONWe presented a k-ary n-cube interconnect as an off-chipsolution for line cards. Results show that the k-ary n-cubeinterconnect had superior performance over commonly usedshare-bus and it can adapt to higher line rates while maintain-ing low latency. The k-ary n-cube interconnect can be easilyscalable to accommodate more processors/memories in orderto increase its processing power or if line rates exceed thecurrent rates and require a change in network size.REFERENCES[1] M. O. Khaoua, “A Performance Model for Duato’s Fully AdaptiveRouting Algorithm in k-Ary n-Cubes”, IEEE Transactions on Com-puters, vol. 48, no. 12, pp. 1297-1304, 1999.[2] W. J. Dally, “Virtual-Channel Flow Control”, IEEE Transactions onParallel and Distributed Systems, vol. 3, no. 2, pp. 194-199, 1992.[3] J. Kim and C. R. Das, “Hypercube Communication Delay withWormhole Routing”, IEEE Transactions on Computers, vol. 43, no.7, pp. 806-813, 1994.[4] A. Agarwal, “Limits on Interconnection Network Performance”, IEEETransactions on Parallel and Distributed Systems, vol. 2, no. 4, pp.398-412, 1991.[5] W. A. Najjar, A. Lagman, S. Sur and P. K. Srimani, “AnalyticalModels of Adaptive Routing Strategies”, Department of ComputerScience, Colorado State University, August 10, 1994.[6] W. L. Bain and S. R. Ahuja, “Performance Analysis of High-SpeedDigital Buses for Multiprocessing Systems”, Proceedings of the 8thannual symposium on Computer Architecture, pp. 107-133, 1981.[7] Y. Zhang, “Microstrip-multilayer delay line on printed-circuit board”,Technical Report, University of Nebraska, Lincoln, April, 2003.[8] H. S. Azad and M. O. Khaoua, “A Simple Mathematical Model ofAdaptive Routing in Wormhole k-ary n-cubes”, Proceedings of the2002 ACM symposium on Applied computing, pp. 835-839, 2002.[9] Y. M. Boura and C. R. Das, “Modeling Virtual Channel Flow inHypercubes”, Proceedings of the First IEEE Symposium on HighPerformance Computer Architecture , pp. 166-175, 1995.1906Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on July 9, 2009 at 05:55 from IEEE Xplore.  Restrictions apply.

A Performance Model for Duato’s Fully Adaptive Routing Algorithm in k-Ary n-Cubes”,

A Simple Mathematical Model of Adaptive Routing in Wormhole k-ary n-cubes”,

Analytical Models of Adaptive Routing Strategies”,

Hypercube Communication Delay with Wormhole Routing”,

Limits on Interconnection Network Performance”,

Microstrip-multilayer delay line on printed-circuit board”,

Modeling Virtual Channel Flow in Hypercubes”,

Performance Analysis of High-Speed Digital Buses for Multiprocessing Systems”,

Virtual-Channel Flow Control”,

Engel, J

Kocak, Taskin

University of Central Florida (UCF): STARS (Showcase of Text, Archives, Research & Scholarship)

K-Ary N-Cube Based Off-Chip Communications Architecture For High-Speed Packet Processors

https://research-information.bris.ac.uk/files/3016353/engel_IEEE_MWSCAS_2005.pdf

K-ary n-cube based off-chip communications architecture for high-speed packet processors

Abstract

Similar works

Full text

Available Versions

Explore Bristol Research

Explore Bristol Research

Explore Bristol Research

University of Central Florida (UCF): STARS (Showcase of Text, Archives, Research & Scholarship)