High Speed Interconnect (HSI) systems are essential in order to transport the large amount of data generated by the detectors, to storage devices in high energy physics experiments.
The integration of a new packet fragmentation
mechanism on the S-Link to Gigabit Ethernet Adapter will permit not only the physical interconnection between involved interfaces, but also the conversion of data formats and transmission protocols.
This paper presents and discusses simulation results of the S-Link to Gigabit Ethernet Adapter, with and without the new packet fragmentation mechanism, based on the data format adopted in the LHCb experiment

Jacobsson, Richard

Jost, Beat

Neufeld, Niko

Neves, Joaquim José Santos Esteves

Universidade do Minho: RepositoriUM

New Packet Fragmentation forS-Link to Gigabit Ethernet AdapterJoaquim E. Neves1,2, Niko Neufeld1, Richard  Jacobsson1 and Beat Jost11EP-Division, CERN, CH-1211 Geneva 23, Switzerland2Escola de Engenharia da Universidade do Minho, P-4800-058 Guimarães, PortugalABSTRACTHigh Speed Interconnect (HSI) systems are essential inorder to transport the large amount of data generated bythe detectors, to storage devices in high energy physicsexperiments.The integration of a new packet fragmentationmechanism on the S-Link to Gigabit Ethernet Adapterwill permit not only the physical interconnection betweeninvolved interfaces, but also the conversion of dataformats and transmission protocols.This paper presents and discusses simulation results ofthe S-Link to Gigabit Ethernet Adapter, with and withoutthe new packet fragmentation mechanism, based on thedata format adopted in the LHCb experiment.K e y w o r d s :  Gigabit Ethernet, LHCb, PacketFragmentation, S-Link.1. INTRODUCTIONIn the Large Hadron Collider experiments, such as theLHCb [1,2] which will take place at CERN, thetransmission between detectors, connected to the Front-end Motherboards (FEMB), and storage devices,connected to the Read-out Motherboards (ROMB), issupported by Data Acquisition and Control Systems.Usually the throughput capacity of these transmissionsystems is asymmetric, since a large amount ofinformation has to be transferred from FEMBs toROMBs, while only a small volume of data is needed inthe opposite direction for the control and management ofsystem.The LHCb experiment integrates the Gigabit Ethernet(GbE) [3] as the link technology, to connect the output ofthe FEMB electronics boards to the input of the ROMBSub-Farm Controllers (SFCs), and the S-Link [4], as astandard interface between the FEMB and the DataAcquisition (DAQ) system.Since the Readout Network already supports the GbEinterface, an S-Link card for the FEMB, based on GbE,has been designed and will be implemented within a shorttime. This module, designated S-Link to Gigabit EthernetAdapter (SGbEA), converts physical interfaces, dataformats and transmission protocols between S-Link andGigabit Ethernet interfaces.Next, in Section 2, it is discussed the technicalspecification of the S-Link to Gigabit Ethernet Adapter,while Section 3 presents simulation results of itsthroughput capacity for two different modes of operationand packet lengths. Section 4 summarizes the mainconclusions of this paper.2. TECHNICAL SPECIFICATIONAs the name specifies, the Gigabit Ethernet is able tosupport the transmission of variable length data packetsover a standardized copper or optical medium with debitslower than 1 Gbit/s on the physical interface.On the other hand, the S-LINK is defined as a simpleFIFO-like user interface (at both ends of the transmissionlink), which remains independent of the technology usedto implement the physical link and provides the transferof event data and control words, error detection, optionalflow control and test facilities [5].The S-LINK specification describes the interface betweenthe FEMB and the Link Source Card (LSC) and theinterface between the Link Destination Card (LDC) andthe ROMB, in either simplex or duplex version.In the simplex version, since there is no communicationpath from the LDC to the LSC, the transmission isunidirectional, while, in the duplex version, the returnchannel between the LDC and the LSC allows thetransmission of flow control commands from the ROMBto the FEMB. In both versions a single, high-density, 64-pin connector is used to connect the FEMB to the LSCand the LDC to the ROMB, allowing, per clock cycle, thetransport of a 32 bit word at the frequency of 40 MHz,which corresponds to a throughput of 1.28 Gbit/s.S-Link to Gigabit Ethernet AdapterThere is more than one solution for the implementation ofthe Front-End Multiplexers (FEMs) and the ReadoutUnits (RUs). The S-Link to Gigabit Ethernet Adapter is apossible solution to interconnect these systems, since itimplements the Ethernet functionalities of the Physicaland Media Access Control (MAC) Layers, and the S-linkspecifications. The Architecture of the LHCb DataAcquisition, based on SGbEA, is depicted in Figure 1.Read-outMotherboardGigabitEthernetFront-endMotherboardS-Link toGigabit EthernetAdapterFigure 1 – Architecture of the LHCb Data AcquisitionSystem, based on S-Link to Gigabit Ethernet Adapter.The SGbEA has to generate the Start Of Packet (SOP)and the End Of Packet (EOP) signals for the MACdevice. It has also to stop and restart the transmissionwhen receiving watermark flags from the MAC or S-linkFIFOs. Optionally, it can process the fragmentation of theS-link Packets on Ethernet frames, as is explained below.Operational ModesSince within the LHCb communication links betweenFEMs and RUs, the data can be transported with twodifferent formats, the SGbEA supports two operationalmodes: Short Packet Mode (SPM) and Long PacketMode (LPM).In SPM, the length of the packets generated by the FEMBis variable, but always smaller than the maximum lengthof the Ethernet MAC frames. On the other hand, since theheader of the LHCb data format already includes theheader of the MAC Frames, no protocol conversion isnecessary and the S-link packet is forwarded directlyfrom the S-link to the Gigabit Ethernet.In LPM, the FEMB can generate data packets ranging inlength from 52 Bytes to 32 Kbytes.In this operational mode the length of the S-link packetcan be higher than the maximum length of the EthernetMAC frames. For this reason, the header of each S-linkpacket, which contains a field with its length, has to bememorized in the SGbEA RAM, before calculating thenumber of fragments into which it must be split in orderto be transmitted. In this mode, the SGbEA inserts, withineach fragment header, the Type/length of the frame onthe Ethernet header field, together with the number offragments and the current fragment number of the S-linkpacket.Optionally, in both operational modes, the EthernetSources and Destination Address can be generated by theFEMB (on a packet-by-packet basis), or can bepreviously inserted in the SGbEA Registers, by thecontrol system and then transferred to each packetfragment headerPhysical ImplementationThe first prototype of the SGbEA module has beenimplemented on a small daughter board over a PCIMezzanine Card (PMC), according to the IEEE CommonMezzanine Card standard [5]. Since the next version ofthe SGbEA will already be connected to the FEMB, thenew packet fragmentation can be fully tested, at themaximum S-link throughput, without the presentconstraints of the PCI Bus.3. SIMULATION RESULTSA model of the SGbEA Module, presented in Figure 1,has been developed for simulation purposes, togetherwith a S-link packet generator.Packets with different lengths and data formats weregenerated by this model at the S-link interface, in order tobe processed by the SGbEA, described on VHSICHardware Description Language (VHDL), on theVisualHDL platform, supported by a Computer-aidedEngineering (CAE) system.Since preliminary results of the SGbEA simulation, withShort and Long Packet Modes, have already beenreported [7], the performance of the SGbEA with andwithout the new packet fragmentation will subsequentlybe presented.In both cases the S-link overhead, introduced by thecontrol words used for signaling the beginning and theend of the LHCb data packets, was reduced to aminimum: one word for the start and another one for theend of packet.322561024179261441228204832760,0000,2000,4000,6000,8001,0001,2001,4001,600Packet Length (Bytes)Throughput (M Packet/s)Previous VersionShort Packet Mode (SPM)Long Packet Mode (LPM)Figure 2 – Maximum throughput achieved with the previous version of the SGbEA and with the proposed Short and LongPacket Modes, for different packet length.In another interface, at the Ethernet side, in addition tothe MAC overhead, a minimum inter frame gap of 12bytes was been guaranteed.Maximum ThroughputFigures 2 reports the results of the maximum throughputachieved by SGbEA, for different packet lengths, withSPM, LPM and without packet fragmentation.It is interesting to note that, in the previous version,without packet fragmentation, for a 64 byte packet, thereference value for packet length on several interfaces ofthe LHCb data acquisition system, the maximumthroughput reached by SGbEA is approximately 1,1 MPacket/s, a value very close to the trigger rate of thoseinterfaces.On the other hand, with the proposed packetfragmentation, the maximum exceeds 1,4 M Packet/swith SPM, and 1.2 M Packet/s with LPM, which meansthat not only SPM but also LPM can be used with thesepacket lengths in the interfaces with a trigger rate ofabout 1.1 MHz.For longer packets, the decrease of the throughput isproportional to the increase of packet length, as theoverhead introduced by the fragmentation processbecomes negligible. It should also be noted that, both inSPM and without packet fragmentation, the length of theLHCb data packets has to be lower than the payload ofthe Gigabit Ethernet MAC frame.Relation with Ethernet CapacityFigure 3 shows the percentage of transmission capacity,at the GbE interface, which could be reached with theprevious version of the SGbEA, the Short Packet Modeproposed and the Gigabit Ethernet capacity, for differentpacket lengths.As the figures show, due to the overhead of the Ethernetframes, the transfer rate achieved by packets lengths witha size to the order of tens of bytes, is clearly lower thanthe transmission capacity, both in the S-link and in theGbE interfaces.In contrast, for longer packets with over a hundred bytes,the transfer rate begins to be limited by the effective GbEcapacity. The Figure also shows that the proposed SPMabsorbs the entire GbE capacity, independently of thepacket length, while the throughput achieved by theprevious version remains a little distant of that value, inparticular for short length packets.3212851210240,000,200,400,600,801,00Packet Length (Bytes)Throughput (%)Simulation SPMGbE-MaximumPrevious VersionFigure 3 – Relationship between the throughput,achieved with the previous version of the SGbEA, theShort Packet Mode proposed and Gigabit Ethernetcapacity, for different packet lengths.Figure 4 shows the percentage of transmission capacity,at the GbE interface, which could be reached with theprevious version of the SGbEA, the Long Packet Modeproposed and Gigabit Ethernet capacity, for differentpacket lengths.32768204812288286720,000,200,400,600,801,00Packet Length (Bytes)Throughput (%)Simulation LPMGbE-MaximumPrevious VersionFigure 4 – Relationship between the throughput,achieved with the previous version of the SGbEA, theLong Packet Mode proposed and Gigabit Ethernetcapacity, for different packet lengths.As is depicted in Figure 4, the effective occupancy of theEthernet payload increases with the length of the packetwith short length packets (as was observed in the SPMsimulations), and remains constant for longer packets.Figure 4 shows that only the proposed LPM is able totransfer packets with lengths longer than the payload ofthe Gigabit Ethernet MAC frame, and the GbE occupancyis almost filled.4. CONCLUSIONSA new packet fragmentation mechanism, for the S-Linkto Gigabit Ethernet Adapter, has been developed to beused on LHC experiments at CERN.As the simulation results show, this technique suppressesthe drawbacks of the previsions of this system, which didnot process any kind of packet fragmentation of S-linkdata.The new version of the S-Link to Gigabit EthernetAdapter, with this packet fragmentation mechanism cantransmit Short and/or Long Packets over the LHCb DataAcquisition System, based on Gigabit Ethernet.5. REFERENCES[1] “LHCb Online System - Data acquisition andexperiment control”, Technical Design Report,CERN/LHCC 2001-40, December 2001. http://lhcb-comp.web.cern.ch/lhcb-comp/General/Publications/onlinetdr.pdf[2] T. Nakada, “CP Violation”, CERN AcademicTraining lecture series, Geneva, Feb 97. http://lhcb-doc.web.cern.ch/lhcb-doc/presentations/B-physics%20and%20CP%20violation/lecture/cpviol.htm [3] “IEEE Std Carrier Sense Multiple Access withCollision Detection (CSMA/CD) Access Method andPhysical Layer Specifications”, IEEE - StdAssociation, Piscataway, NJ 08855-1331, USA,2002. http://shop.ieee.org[4] O. Boyle et al., “The S-LINK InterfaceSpecification”, ECP-Division, Geneva, March 97,http://hsi.web.cern.ch/HSI/s-link/spec/spec[5] H. C. van der Bij et al., “S-LINK, a Data LinkInterface Specification for the LHC Era”, IEEETransactions on Nuclear Science, Volume: 44 Issue:3 Part: 1, Anaheim, CA, USA, June 97.[6] “IEEE P1386 Standard Mechanics for a CommonMezzanine Card Family: CMC”, Institute ofElectrical and Electronics Engineers, Piscataway, NJ08855-1331, USA.[7] J. E. Neves et al.,“S-Link to Gigabit Ethernet Adapter:- New Frame Segmentation for LHCb DataAcquisition System”, Conference on Computing inHigh Energy and Nuclear Physics - CHEP03, LaJolla, CA, USA, March 2003.

New packet fragmentation for S-link to giiga bit ethernet adapter

http://repositorium.sdum.uminho.pt/bitstream/1822/3033/1/SLGbEpf13.pdf

New packet fragmentation for S-link to giiga bit ethernet adapter

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM

Universidade do Minho: RepositoriUM