The Data Acquisition system of the CMS experiment, at the CERN LHC collider, is designed to build 1~MB events at a sustained rate of 100 kHz and to provide sufficient computing power to filter the events by a factor of 1000. The Data to Surface (D2S) system is the first layer of the Data Acquisition interfacing the underground subdetector readout electronics to the surface Event Builder. It collects the 100~GB/s input data from a large number of front-end cards (650) , implements a first stage event building by combining multiple sources into lar ger-size data fragments, and transports them to the surface for the full event building. The Data to Surface system can operate at the maximum rate of 2 Tbps. This paper describes the layout, reconfigurability and production validation of the D2S system which is to be installed by December 2005

Arcidiacono, R

Brigljevic, V

Bruno, G

Cano, E

Cittolin, Sergio

Erhan, S

Gigi, D

Glege, F

Gulmini, M

Gutleber, J

Gómez-Reino Garrido, R

Jacobs, C

Kreuzer, P

Lo Presti, G

Magrans De Abril, I

Marinelli, N

Maron, G

Meijers, F

Meschi, E

Murray, S

Oh, A

Orsini, L

Pieri, M

Pollet, L

Rosinsky, P

Rácz, A

Schwick, C

Sphicas, P

Sumorok, J

Varela, K

CERN Document Server

Available on CMS information server CMS CR 2006/034CMS Conference Report15 July 2006R. Arcidiacono (on behalf of the CMS DAQ Group)Massachusetts Institute of Technology, Cambridge, Massachusetts, USAThe 2 Tbps ”Data to Surface” system of the CMSData Acquisition.AbstractThe Data Acquisition system of the CMS experiment, at the CERN LHC collider, is designed to build1 MB events at a sustained rate of 100 kHz and to provide sufcient computing power to lter theevents by a factor of 1000. The Data to Surface (D2S) system is the rst layer of the Data Acquisitioninterfacing the underground subdetector readout electronics to the surface Event Builder. It collectsthe 100 GB/s input data from a large number of front-end cards (650), implements a rst stage eventbuilding by combining multiple sources into larger-size data fragments, and transports them to thesurface for the full event building. The Data to Surface system can operate at the maximum rate of 2Tbps. This paper describes the layout, recongurability and production validation of the D2S systemwhich is to be installed by December 2005.Presented at IEEE-NPSS Real Time, Stockholm , June 2005The 2 Tbps ”Data to Surface” system of the CMSData Acquisition.R. Arcidiacono , V. Brigljevic, G. Bruno, E. Cano, S. Cittolin, S. Erhan, D. Gigi,F. Glege, R. Gomez-Reino Garrido, M. Gulmini, J. Gutleber, C. Jacobs, P. Kreuzer, G. Lo Presti,I. Magrans De Abril, N. Marinelli	, G. Maron, F. Meijers, E. Meschi, S. Murray, A. Oh, L. Orsini,M. Pieri, L. Pollet, A. Racz, P. Rosinsky, C. Schwick, P. Sphicas, K. Sumorok , J. Varela University of Athens, Athens, Greece	Institute of Accelerating Systems and Applications, Athens, GreeceINFN - Laboratori Nazionali di Legnaro, Legnaro, ItalyLIP, Lisbon, PortugalCERN, Geneva, SwitzerlandUniversity of California, Los Angeles, Los Angeles, California, USAUniversity of California, San Diego, San Diego, California, USA Massachusetts Institute of Technology, Cambridge, Massachusetts, USARudjer Boskovic Institute, Zagreb, CroatiaAbstract— The Data Acquisition system of the CMS experi-ment, at the CERN LHC collider, is designed to build 1 MBevents at a sustained rate of 100 kHz and to provide sufficientcomputing power to filter the events by a factor of 1000. The Datato Surface (D2S) system is the first layer of the Data Acquisitioninterfacing the underground subdetector readout electronics tothe surface Event Builder. It collects the 100 GB/s input datafrom a large number of front-end cards (650), implements a firststage event building by combining multiple sources into larger-size data fragments, and transports them to the surface for thefull event building. The Data to Surface system can operate atthe maximum rate of 2 Tbps. This paper describes the layout,reconfigurability and production validation of the D2S systemwhich is to be installed by December 2005.I. INTRODUCTIONThe CMS Trigger and Data Acquisition system is designedto collect and inspect the detector information at the LHCbunch crossing frequency of 40 MHz. It has also to select amaximum of 100 Hz events for offline analysis.At the design luminosity of ﬀﬁﬃﬂ , the LHC rateof proton collisions will be around 20 per bunch crossing,producing approximately 1 MB of data in the CMS detector.The first level trigger is designed to reduce the incoming datarate to a maximum of 100 kHz, by processing fast triggerinformation coming from the calorimeters and the muonchambers, and selecting events with interesting signatures.The Data Acquisition system (DAQ) [1] has therefore tosustain a maximum input rate of 100 kHz, for an average dataflow of 100 GB/s coming from  650 data sources, and toprovide enough computing power to reduce the rate of storedevents by a factor of 1000. A block diagram of the CMS DAQsystem is shown in Fig. 1.The data flow is as follows. The various subdetector readoutsystems store data continuously in 40 MHz pipelined buffers.Upon arrival of a synchronous L1 trigger (3  s latency), thecorresponding data are extracted from the front-end buffersand pushed into the DAQ system by the Front-End Drivers(FEDs). The event fragments are sent to the first stage ofthe event building, the FED Builder network, through a shortdistance link and a receiver interface card. This stage is incharge of transporting the fragments to the surface, and ofassembling event fragments of variable size from 650 FEDsinto 64 super-fragments of 16 kB average size. The super-fragments are then stored in large buffers in Readout Units(RU), waiting for the second stage of event building (RUBuilder), implemented with multiple 64 x 64 networks. Therewill be up to 8 RU Builders, or ”DAQ slices”, connected tothe FED Builders layer. Each FED Builder is in charge ofdistributing the super-fragments, on an event by event basis,to the existing RU Builders. Super-fragments correspondingto the same event are located in the same slice, and are readby one Builder Unit (BU) of the RU Builder network. Thecomplete event is then transferred to a single unit of the FilterFarm. The High Level Trigger (HLT) algorithms will decidewhether to store or reject the event.This architecture has an innovative double-stage event build-ing, implemented with two layers of switches, which optimizesthe traffic load to the final event builder and allows for aprogressive deployment of the full size system.II. THE DATA TO SURFACE SYSTEMThe Data to Surface (D2S) system represents the first layerof the DAQ interfacing to the subdetector front-end readouts. Itreceives and transports the data to the surface. It also performssuper-fragment building in the FED Builder networks. In orderto reduce the number of inputs to the FED Builders and tobetter exploit the available bandwidth, it allows to merge upto four data sources into a single data fragment.Fig. 1. Schematic view of the CMS Data Acquisition system.The requirements which have driven the design implemen-tation of this layer are:! the need of a common interface to the subdetector specificfront-end readout electronics, in order to reduce diversityand ease integration and maintenance operations! a sustained data throughput of each D2S element of 200MB/s over a distance of 200 m, with the capability ofabsorbing fluctuations of the event size! the capability of providing balanced traffic to the eventbuilding stages, in order to optimize the building effi-ciency, and the capability of easy reconfiguration of thedata sources routing.Fig. 2 shows the physical implementation of one element ofthe D2S system. The architecture foresees a maximum of 512of such elements. One element includes the following devices:! one S-Link64 Sender card, housed by the subdetectorFront-End Driver! a short distance link implemented as LVDS copper cable,maximum 10 m long! one Front-end Readout Link (FRL), housing up to fourS-Link64 receivers! two commercial Myrinet Network Interface Cards (NIC)[2], with two 2.0 Gbps data rate optical links (rails), usedboth in input and output of the FED Builder network! eight network switches’ ports (Myrinet technology)! two 200 m long optical fibers making up the long distancelink connecting underground to surface area, plus fourshort dual optical fibers to connect the two rails of theMyrinet NIC to the switch ports.Fig. 3 displays a picture of a D2S element, from the S-Link64 card to the optical fibers which connect the MyrinetNIC to the FED Builder switch.A. The Sender cardThe S-Link64 Sender card is a Common Mezzanine Card(CMC) developed by the DAQ group. It receives data fromthe FED via an S-Link64 port [3]. The card is able to bufferup to 1.6 kB of data before generating back-pressure to theFED. The CMC has an LVDS converter to interface with theshort distance link. The implementation of the S-Link64 porton the FEDs is part of the common functional block diagramfollowed by the subdetectors.The default data transfer rate over the LVDS cable is400 MB/s (50 MHz clock " 64 bits), double the averagedesign speed of the DAQ system. The S-Link64 card hasthe possibility of being set at two other clock frequencies, 40Fig. 2. D2S element physical implementation.Fig. 3. Picture of a D2S element, from the S-Link64 sender card (up-right)to the long distance link (optical fibers).MHz or 60 MHz, in case of less or more demanding front-endsystems respectively. The maximum sustainable output rate istherefore 480 MB/s.B. The FRL cardThe FRL card is a CompactPCI card, developed by the DAQgroup, which receives, buffers and optionally merges the eventfragments, checks the payload CRC, and pushes the data to theMyrinet NIC in fixed size packets. The input buffer memorieshave a 64 kB size.The FRL card has three different interfaces: an input in-terface which handles up to four LVDS cables; an outputinterface to the FED Builder implemented as a 64bit/66MHzPCI connector for the Myrinet NIC; a configuration andcontrol interface which is a PCI bus interface connected tothe CompactPCI backplane.The FRL card also provides monitoring features, like theability to spy on a fraction of events, via the CompactPCI bus,and to fill histograms of fragment sizes distribution. Each ofthe 512 FRL cards is expected to receive 2 kB event fragmentson average. The FRL cards push the data to the Myrinet NICsvia the internal 64bit/66MHz PCI bus. Measurements showthat a speed very close to the PCI bus limit of 528 MB/s canbe reached, depending on the FRL packet size and for eventsizes above 1 kB.C. The FED BuilderThe FED Builder is based on Myrinet technology. Myrinetis a high performance interconnect technology for clusters,composed of crossbar switches and NICs, connected by pointto point bi-directional links. It employs wormhole routing, andthe delivery of packets is guaranteed by a flow control at thelink level. The Myrinet switch supports hence back-pressure,which is propagated backwards up to the FEDs.The design choice of having 8x8 logical networks in theFED Builders has been taken. One FED Builder is madeof 8 input NICs, two layers of two 8x8 switch networks,8 output NICs (see Fig. 4, where only one network layeris shown). Each rail of one Myrinet card is connected toan independent crossbar. The FED Builder input cards areprogrammed to read fragments from the FRL and to send themto the switch, via one of the optical links, with a destinationport assigned on the basis of the fragment event number and apredefined lookup table. The FED Builder output cards, inputsof the RU Builders, concatenate fragments with the same eventnumber from all the connected input cards, building the super-fragments.The two switches layers are located one in the undergroundcavern, one in the surface DAQ room. They are implementedwith large switches, 256 external ports each, allowing for ahigh reconfigurability of the FED-Builders’ network by simplereprogramming of the switches’ routing table.The Myrinet card can transfer at 4.0 Gbps data rate overthe two optical rails. As for the Myrinet switch performance,measurements done with different traffic conditions show thatthe 95% of the efficiency is reached with constant size events,while it drops to 50% when transporting variable size events.Fig. 5 shows the FED Builder performance as measured ina test bench with an 8x8 network. Data are injected in thesystem by the FRL cards generating fragments in saturationFig. 4. 8x8 FED Builder with a two-rail network.Fig. 5. Throughput per node versus fragment size in a 8x8 FED Builder.mode. The packet size used in the FRL-NIC protocol is 4 kB.The throughput variation as a function of the average fragmentsize depends on the rms of the event size distribution. Thebest performance is obtained for constant event sizes. Forthe nominal average fragment size of 2 kB, the measuredthroughput per node is about 250 MB/s.The sustained data transfer speed through the FED buildersis 1 Tbps, given the design implementation choice of no trafficshaping through the Myrinet switches and the variable andunbalanced traffic conditions generated by the front-end datasources. This throughput meets the CMS DAQ requirements. Amaximum peak bandwidth of 2 Tbps is theoretically possible,when fully exploiting the FRL and Myrinet NIC bandwidth,and with traffic shaping in the FED Builders.III. D2S SYSTEM PRODUCTION AND VALIDATIONThe D2S system is the first part of the DAQ to be installed inthe experimental area. The delivery milestone is December 05.Full poduction of the custom made components (S-Link64and FRL cards) has been completed, and all the commercialequipment (Myrinet NICs and switches, LVDS cables, opticalfibers) has been purchased and delivered.All this material needs to be tested, classified in a databaseand validated before installation. Several testing procedureshave been developed at CERN for this purpose. Tests havealready started and will continue throughout the summer.In the following subsections, the validation and quality con-trol implemented for the custom made cards will be described.A. FRL and S-Link64 Sender validationThe FRL and S-Link64 Sender cards are qualified using twodifferent setups, both custom-built:! Stage 1 - the first setup is where visual and electricalchecks are performed and the card firmware is loaded.! Stage 2 - the second runs several software tools in order toperform a full hardware integrity and operational check.Stage 1 test bench is composed of a dedicated hardwarestation where the card under test can be plugged, inspectedand measured by ad-hoc connectors. FPGA’s firmware isprogrammed via the JTAG serial port, accessed through thePC parallel port. Operations and interface to the productiondatabase are controlled by a LabVIEW program.The Stage 2 test bench is composed of a CompactPCI crate,controlled by a PC, reference LVDS cables, and a set ofreference FRL cards (or S-Link64 cards, depending whetherS-Link64 or FRL cards respectively are to be tested), whichare used coupled with custom-built testing cards. Fig. 6 showsa diagram of this configuration.The custom-built testing card is a GIII [4] PCI card,connected to the internal 64bit/66MHz PCI bus of the FRLcard, and hosting a S-Link64 Sender card. This GIII card isable to generate pseudo-random or well-know pattern data. Itemulates the FED data source. The data are then transmitted,via the S-Link64 Sender and the LVDS cable, to the inputof the FRL. The FRL processes and pushes the data back toLVDS cable400 MB/sdata ratePCI 64b/66MHzinput2input1GIII testing cardFED emulator/Myrinet NIC emulatorCompactPCI bus to control PCS−Link64 SenderFRL cardFig. 6. Diagram of the FRL/S-Link64 test board used in the Stage2 setup.the GIII card, which emulates the Myrinet NIC-FRL protocol,and, at last, checks the integrity of the payload byte by byte.The software controlling this setup performs first JTAGchain checks (via the CompactPCI bus), memory/registerread/write checks, LVDS link autotests and eventually runsthe closed-loop data transfer initiated by the GIII cards.A linux-based Perl/Tk GUI controls the test procedure andthe database interface.B. Long duration stability testsA third setup is used to qualify part of the D2S element(from S-Link64 card to Myrinet NIC), as a whole. It is equiv-alent to a Bit Error Rate Test (BERT) system implementedwith D2S elements.The test system is composed of one PC hosting 16 FEDemulators, each with an S-Link64 Sender card, 16 LVDScables and one CompactPCI crate hosting 16 FRL cards cou-pled this time with Myrinet NICs. The LVDS cables connectFED emulators to the FRLs. Fragments data from the FEDemulators, along with the CRC, are transmitted to the FRLinputs, via the LVDS link, and pushed into the Myrinet NICs.The CRC is recomputed and compared to the original insidethe FRL. Data corruption during the transfer is detectable bychecking the CRC comparison.The test runs for 24 hours. A total of 34.5 TB of pseudo-random data, at the speed of 400 MB/s, are injected in eachlink, leading to a BER upper limit of $# %&"'( )ﬂ* at 90% C.L.per cable.IV. CONCLUSIONThe architecture of the D2S system is dictated by therequirements and constraints of the DAQ and Trigger systems.The D2S has been designed to collect and concentrate datafrom all the readout sources, transport them on the surface, andfeed balanced output streams to the event building process.This is accomplished by providing a first stage in the eventbuilding which groups 8 FRL data sources in a single outputchannel, and distributes it to different slices of the EventBuilder. This scheme allows for a scalable design of thesurface Event Builder, which can be procured and installedin phase with the requirements arising from the performanceof the accelerator and the experiment itself.The D2S components’ development phase has been com-pleted successfully. The FED builder based on Myrinet tech-nology shows a throughput per node above the required 200MB/s, for fragment sizes greater than 1 kB. The aggregatethroughput of the D2S system, in standard running conditions,is 1 Tbps.All the hardware components of the complete D2S system,custom made or commercial, have been produced or pur-chased. Validation procedures have been specifically designedfor all the components, to qualify the system.The full production of FRL and S-Link64 Sender cards iscurrently being tested at CERN, while the LVDS cables, alongwith Myrinet NICs will undergo long duration tests during thesummer. The procurement and validation of the D2S systemis on schedule and the installation is foreseen for December05.REFERENCES[1] CERN/LHCC 2002-26, CMS TDR, Data Acquisition and High LevelTrigger[2] Myricom, see http://www.myricom.com[3] A. Racz, R. McLaren, E. van der Bij, The S-Link64 bit extensionspecification: S-Link64, available at http://hsi.web.cern.ch/HSI/s-link[4] E. Cano, D. Gigi, “FEDKIT User’s Manual and Programmer’sManual”, v. 0.8d, CERN CMS internal note (2004).http://cano.home.cern.ch/cano/fedkit/fedkit-0.8d.pdf

The 2 Tbps "Data to Surface" System of the CMS Data Acquisition

The 2 Tbps "Data to Surface" System of the CMS Data Acquisition

Abstract

Similar works

Full text

Available Versions

CERN Document Server