Data Acquisition and Control Systems used in high energy physics experiments,
such as those which will take place in the Large Hadron Collider (LHC) at CERN,
require the specification of data formats and transmission protocols as well as
the use of high speed links and interfaces.
  In this context, a new Frame Segmentation process will be presented and
discussed, based on data formats adopted by the LHCb experience for the
interconnection of two standardized systems: S-link and Gigabit Ethernet.
Simulation results of the transfer capacity of the proposed mechanism will be
also reported, together with guidelines for its physical implementation.Comment: 6 pages, 6 figures. Computing in High Energy and Nuclear Physics -
  CHEP03, La Jolla, CA, March 200

Jacobsson, Richard

Jost, Beat

Neufeld, Niko

Neves, Joaquim E.

English

arXiv

Data Acquisition and Control Systems used in high energy physics experiments, such as those which will take place in the Large Hadron Collider (LHC) at CERN, require the specification of data formats and transmission protocols as well as the use of high speed links and interfaces.
In this context, a new Frame Segmentation process will be presented and discussed, based on data formats adopted by the LHCb experience for the interconnection of two standardized systems: S-link and Gigabit Ethernet. Simulation results of the transfer capacity of the proposed mechanism will be also reported, together with guidelines for its physical implementation

Neves, Joaquim José Santos Esteves

Universidade do Minho: RepositoriUM

arXiv:physics/0304014 v1   4 Apr 2003Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 2003 1S-Link to Gigabit Ethernet AdapterNew Frame Segmentation for LHCb Data Acquisition SystemJoaquim E. NevesEP-Division, CERN, CH-1211 Geneva 23, Switzerland.Escola de Engenharia da Universidade do Minho, P-4800-058 Guimares, Portugal.Richard Jacobsson, Niko Neufeld, and Beat JostEP-Division, CERN, CH-1211 Geneva 23, SwitzerlandData Acquisition and Control Systems used in high energy physics experiments, such as those which will takeplace in the Large Hadron Collider (LHC) at CERN, require the specification of data formats and transmissionprotocols as well as the use of high speed links and interfaces.In this context, a new Frame Segmentation process will be presented and discussed, based on data formatsadopted by the LHCb experience for the interconnection of two standardized systems: S-link and GigabitEthernet. Simulation results of the transfer capacity of the proposed mechanism will be also reported, togetherwith guidelines for its physical implementation.1. INTRODUCTIONThe Large Hadron Collider beauty experiment forprecise measurements of CP violation [1] and raredecays (LHCb) [2] requires High Speed Intercon-nect (HSI) Systems in order to transport the largeamount of data generated by the detectors connectedto the Front-end Motherboards (FEMB) to the stor-age devices connected to the Read-out Motherboards(ROMB).In the opposite direction, only a small volume ofdata generally has to be transferred between thoseparts for the control and management of the wholesystem.The LHCb experiment adopted Gigabit Ethernet(GbE) [3] as the link technology from the output of theFEMB electronics boards to the input of the ROMBSub-Farm Controllers (SFCs), and S-Link [4], as astandard interface between the FEMB and the DataAcquisition (DAQ) system [2].The Architecture of the LHCb Data Acquisition,based both on full S-link transmission and on the S-Link to Gigabit Ethernet Adapter (SGbEA) System,is depicted in Figure 1. Since the Readout Networkalready supports the GbE, one S-Link card for theFEMB, based on GbE, was designed and built withinthe scope of the collaboration with the Atlas LHCexperiment [5].Next, in section 1, the technical specification of theLHCb Data Acquisition System is discussed, whilesection 3 describes the new frame segmentation mech-anism of the SGbEA. Simulation results for SGbEAthroughput, using two different operation modes anddata packet lengths, are presented in section 4. TheSection 5 deals with the physical implementation ofthe New Frame Segmentation and Section 6 summa-rizes the main conclusions.Figure 1: - Architecture of the LHCb Data Acquisition System, based on S-Link to Gigabit Ethernet Adapter.TUGP0012 Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 20032. SYSTEM ARCHITECTUREThe S-LINK specification defines a simple FIFO-like user interface at both ends of the transmission linkwhich remains independent of the technology used toimplement the physical link and which provides thetransfer of event data and control words, error detec-tion, optional flow control and test facilities [6]. TheS-link specification also describes the interfaces be-tween the FEMB and the Link Source Card (LSC)and between the Link Destination Card (LDC) andthe ROMB, in either simplex or duplex version.In the simplex version, since there is no commu-nication path from the LDC to the LSC, the trans-mission is unidirectional, while, in the duplex version,the return channel between the LDC and the LSC al-lows the transmission of flow control commands fromthe ROMB to the FEMB. In both versions a single,high-density, 64-pin connector is used to connect theFEMB to the LSC and the LDC to the ROMB, al-lowing, per clock cycle, the transport of a 32 bit wordat the frequency of 40 MHz, which corresponds to athroughput of 1.28 Gbit/s.There is more than one solution for the implemen-tation of the Front-End Multiplexers (FEMs) and theReadout Units (RUs). This is because, in the LHCbexperiment, the main difference between these mod-ules results from the fact that the RUs have to inter-face to the Readout Network, and hence must respectthe GbE flow control protocol, which is not true forthe links between FEMs and RUs [2].The SGbEA is a possible solution to interconnectFEMs and RUs, as is shown in Figure 1. This modulehas to implement the Ethernet functionalities of thePhysical and Media Access Control (MAC) Layers,and the S-link specifications. The protocol conversionbetween S-link and GbE can be implemented withina FPGA, together with memories (FIFO, RAM andRegisters), as can be seen in Figure 2.The SGbEA has to generate the Start Of Packet(SOP) and the End Of Packet (EOP) signals for theMAC device and to stop and restart the transmis-sion when receiving watermark flags from the MACor S-link FIFOs. Optionally, it can process the frag-mentation of the S-link Packets on Ethernet frames,as is explained in the next section.3. NEW FRAME SEGMENTATIONSince within the communication links, betweenLHCb FEMs and RUs, the data can be transportedwith two different formats, the SGbEA has to supporttwo operational modes: Short Packet Mode (SPM)and Long Packet Mode (LPM).In SPM, the length of the packets generated by theFEMB is variable, but always smaller than the max-imum length of the Ethernet MAC frames. On theother hand, since the header of the LHCb data for-mat already includes the header of the MAC Frames,no protocol conversion is necessary and the S-linkpacket is forwarded directly from the S-link FIFO tothe MAC FIFO.In LPM, the FEMB can generate data packets rang-ing in length from 52 Bytes to 32 Kbytes.In this operational mode the length of the S-linkpacket can be higher than the maximum length of theEthernet MAC frames. For this reason, the headerof each S-link packet, which contains a field with itslength, has to be memorized in the SGbEA RAM,before calculating the number of fragments into whichit must be split in order to be transmitted.Figure 2: - S-Link to Gigabit Ethernet Adapter Architecture.TUGP001Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 2003 3In this mode, the SGbEA inserts, within each frag-ment header, the Type/length of the frame on theEthernet header field, together with the number offragments and the current fragment number of the S-link packet.Optionally, in both operational modes, the EthernetSources and Destination Address can be generated bythe FEMB (on a packet-by-packet basis), or can bepreviously inserted in the SGbEA Registers, by thecontrol system and then transferred to each packetfragment header.4. SIMULATION RESULTSA VHSIC Hardware Description Language (VHDL)model of the SGbEA module was developed for sim-ulation purposes on the VisualHDL platform on aComputer-aided Engineering (CAE) system, togetherwith an S-link packet generator.Packets with different lengths and data formatswere generated by this model at the S-link interfacein either Long or Short Packet Mode, in order to beprocessed by the SGbEA and transmitted over theGigabit Ethernet.In both cases the S-link overhead, introduced by thecontrol words used for signaling the beginning and theend of the LHCb data packets, was reduced to a min-imum: one word for the start and another one for theend of packet. In another interface, at the Ethernetside, in addition to the MAC overhead, a minimuminter frame gap of 12 bytes was also guaranteed.4.1. Short Packet ModeSince in SPM there is no frame segmentation, theinclusion of a MAC frame header at the beginning ofthe packet is the only constraint imposed on the dataformat.Figures 3 and 4 report some results for the SGbEAsimulation in SPM. The maximum throughput for dif-ferent packet lengths is present in Figure 3, whileFigure 4 shows the relationship between the effectivethroughput reached with those packets, and GigabitEthernet capacity.As the figures show, due to the overhead of the Eth-ernet frames, when short packets with a size to theorder of tens of bytes are transferred, the throughputis clearly lower than the transmission capacity, bothin the S-link and in the GbE interfaces. On the otherhand, for longer packets with over a hundred bytes,the transfer rate begins to be limited by the effectiveGbE capacity.It is interesting to note that, for 64 byte packets, thereference value for packet length on several interfacesof the LHCb data acquisition system, the throughputis near 1.4 M Packet/s, allowing the use of SGBE onthese interfaces with a trigger rate of about 1.1 MHz.Figure 3: - Maximum throughput achieved by the SGbEA with Short Packet Mode, for different packet length.TUGP0014 Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 2003Figure 4: - Maximum throughput achieved by Short Packet Mode and Gigabit Ethernet capacity for different packetlength.4.2. Long Packet ModeFigures 5 and 6 present simulation results for FrameSegmentation in LPM. The maximum throughput ofthe system is shown in Figure 5, while Figure 6shows the relationship between the effective through-put achieved on the transmission of those packets andGigabit Ethernet capacity.In this mode, the LHCb data packets presented onthe S-link interface are of variable length, as in SPM,but now the packet header has 52 bytes, including theheader of the MAC frame.Figure 5: - Maximum throughput achieved by SGbEA in Long Packet Mode, for different packet lengths.TUGP001Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 2003 5Figure 6: - Maximum throughput achieved in Long Packet Mode and Gigabit Ethernet capacity for different packetlengths.As was expected, Figure 5 shows that the throughputachieved for packets of a length of less than 64 bytesis lower than that achieved in SPM, but is still higherthan 1,2 M Packet/s, which means that the LPM canalso be used with these packet lengths on the inter-faces with a trigger rate of about 1,1 MHz.For longer packets, the decrease of the throughputis proportional to the increase of packet length, asthe overhead introduced by the fragmentation processbecomes negligible. As is depicted in Figure 6, theeffective occupancy of the Ethernet payload increaseswith the length of the packet for short length packets -as was observed in the SPM simulations - and remainsconstant with longer packets.5. PHYSICAL IMPLEMENTATIONThe Frame Segmentation mechanism for LHCbData Acquisition System described previously hasbeen implemented within a FPGA on the S-Link toGigabit Ethernet Adapter board itself, developed atthe Argonne National Laboratory for Atlas LHC Ex-perience. This prototyping board was implemented ona small daughter board over a PCI 1 Mezzanine Card(PMC), according to the IEEE Common MezzanineCard standard [7].A new version of the SGbEA has been specified forimplementation in the near future. Since this newboard will already be connected to the FEMB, the1PCI, Peripheral Component Interface.new Frame Segmentation can be fully tested, at themaximum S-link throughput, without the present con-strains of the PCI Bus.The single port MAC Controller [8] could be re-placed by a dual port [9] that supports the GigabitMedia Independent Interface (GMII) and the Ten-BitInterfaces (TBI), allowing the Ethernet transmissionover copper and/or fiber media. The Physical Layermodule could also be implemented with a single dualport transceiver, either for copper or fiber media [10].While one serial port is used in the copper interface,by connecting this port to a magnetic device [11], theother is used in the fiber interface, by connecting thisport to the optical module.6. CONCLUSIONSA new Frame Segmentation mechanism, with flowcontrol capabilities, has been implemented over the S-Link to Gigabit Ethernet Adapter for the LHCb DataAcquisition System.This mechanism is not only able to halt the trans-mission, in case of overflow, but can also optionallyinsert the Ethernet Source and Destination Addressesin the MAC frames.As the simulation results show, this technique per-mits the transmission of the Short and/or Long Pack-ets onto High Speed Interconnect Systems, which willbe used in LHC experiments at CERN.TUGP0016 Computing in High Energy and Nuclear Physics - CHEP03, La Jolla, CA, March 2003References[1] T. Nakada, CP Violation, CERN AcademicTraining lecture series, Geneva, Feb 97.http://lhcb-doc.web.cern.ch/lhcb-doc/presentations/Bphysics%20and%20CP%20violation/lecture/cpviol.htm[2] LHCb Online System - Data acquisition andexperiment control, Technical Design Re-port, CERN/LHCC 2001-40, December 2001.http://lhcb-comp.web.cern.ch/lhcb-comp/General/Publications/onlinetdr.pdf[3] IEEE Std Carrier Sense Multiple Access withCollision Detection (CSMA/CD) Access Methodand Physical Layer Specifications, IEEE - StdAssociation, Piscataway, NJ 08855-1331, USA,2002. http://shop.ieee.org/store/product.asp?prodno=SS94973[4] O. Boyle et al., The S-LINK Interface Spec-ification, ECP-Division, Geneva, March 97,http://hsi.web.cern.ch/HSI/s-link/spec/spec[5] Nick Ellis and John Dawson, private communica-tions.[6] H. C. van der Bij et al., S-LINK, a Data LinkInterface Specification for the LHC Era, IEEETransactions on Nuclear Science, Volume: 44 Is-sue: 3 Part: 1, Anaheim, CA, USA, June 97.[7] IEEE P1386 Standard Mechanics for a CommonMezzanine Card Family: CMC, Institute of Elec-trical and Electronics Engineers, Piscataway, NJ08855-1331, USA[8] http://206.204.107.130/products/networking/controllers.html[9] http://www.intel.com/design/network/products/ethernet/linecard ec.htm[10] http://www.marvell.com/Internet/Products/products/1,2414,3-15-151-22,00.html[11] http://www.trans-power.com/gigganged.htmlTUGP001

S-link to giga bit ethernet adapter- new frame segmentation for LHCb data acquisition system

Data Acquisition and Control Systems used in high energy physics experiments, such as those which will take place in the Large Hadron Collider (LHC) at CERN, require the specification of data formats and transmission protocols as well as the use of high speed links and interfaces. In this context, a new Frame Segmentation process will be presented and discussed, based on data formats adopted by the LHCb experience for the interconnection of two standardized systems: S-link and Gigabit Ethernet. Simulation results of the transfer capacity of the proposed mechanism will be also reported, together with guidelines for its physical implementation.Data Acquisition and Control Systems used in high energy physics experiments, such as those which will take place in the Large Hadron Collider (LHC) at CERN, require the specification of data formats and transmission protocols as well as the use of high speed links and interfaces. In this context, a new Frame Segmentation process will be presented and discussed, based on data formats adopted by the LHCb experience for the interconnection of two standardized systems: S-link and Gigabit Ethernet. Simulation results of the transfer capacity of the proposed mechanism will be also reported, together with guidelines for its physical implementation

CERN Document Server

S-Link to Gigabit Ethernet Adapter New Frame Segmentation for LHCb Data Acquisition System

http://repositorium.sdum.uminho.pt/bitstream/1822/3057/1/0304014.pdf

S-Link to Gigabit Ethernet Adapter New Frame Segmentation for LHCb Data Acquisition System

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM

Universidade do Minho: RepositoriUM

CERN Document Server