Cano, E

Cittolin, Sergio

Csilling, Akos

Erhan, S

Gigi, D

Glege, F

Gutleber, J

Jacobs, C

Kozlovskii, Miroslaw P

Larsen, H

Magrans de Abril, Ildefons

Meijers, F

Meschi, E

Murray, S

Oh, A

Orsini, L

Pollet, Lucien

Rácz, A

Samyn, D

Scharff-Hansen, P

Schwick, C

Sphicas, Paris

Strodl, T

CERN Document Server

CMS data to surface transportation architectureE. Canoa, S. Cittolina, A. Csillinga, S. Erhanb, D. Gigia, F. Glegea, J. Gutlebera, C. Jacobsa,M. Kozlovszkya, H. Larsena, I. Magransa, F. Meijersa, E. Meschia, S. Murraya, A. Oha, L. Orsinia,L. Polleta, A. Racza, D. Samyna, P. Scharff-Hansena, P. Sphicasa,c, C. Schwicka, T. StrodlaaCERN, Div. EP, Meyrin CH-1211 Geneva 23 SwitzerlandbUniversity of California, Los Angeles, USAcUniversity of Athens, Athens, GreeceAbstractThe front-end electronics of the CMS experiment will beread out in parallel into approximately 650 modules which willbe located in the underground counting room. The data readout will then be transported over a distance of ~200 m to thesurface counting room where they will be received into deepbuffers, the "Readout Units". The latter also provide the firststep in the CMS event building process, by combining the datafrom multiple detector data sources into larger-size (~16 kB)data fragments. The second and final event-building stepmerges  64 such super-fragments into a full event. The firststage of the Event Builder, referred to as the Data to Surface(D2S) system is structured in a way to allow for a modular andscalable DAQ system whose performance can grow with theincreasing luminosity of the LHCI.  INTRODUCTIONThe CMS DAQ system has been designed to collect eventfragments from approximately 650 data sources at a first leveltrigger rate of 100 kHz.The data sources (or Front End Drivers, FED) are generat-ing for each triggered event an event fragment whose averagesize is between 300 bytes and 2.3 kB. Fragments are sentthrough a short distance link (max 15 m.) to the Front EndReadout Link card (FRL) that can merge up to two datasources. There is a maximum of 512 FRLs. From there, thedata are sent over a 200 m optical link to the surface where thefirst stage of event building (FED builder) takes place (see Fig.1). Fig. 1 D2S transportation devices At this stage, 8 fragments are merged by a 8 x 8 switch toform a super-fragment. There is a total of 64 FED builders.Super-fragments are then stored into the readout units (RU)waiting for the second stage of event building (RU builder).The RU builder is implemented with multiple 64 x 64switches: a single switch is also called "slice". The number ofslices can be 1 to 8 and depends on the number of RUsattached to each FED builders: hence, the DAQ capacity canbe adapted. Super-fragments are assembled into entire events by theRU builder and transferred to the Filter farm where higher-level trigger decisions are made by software. Accepted eventsare permanently stored for further physics analysis.With an event size of 1 MB, the system has to sustain adata throughput of 100 GB/s. This is achieved with the parallelarchitecture shown in Fig. 2.Fig. 2 CMS DAQ block diagramII.  D2S REQUIREMENTSThe D2S is in charge of collecting the data from the front-end data sources (see Tab. 1.) and to transport them to the sur-face where the multi-stage event building process takes place. Reducing the diversity in the electronic devices is neces-sary in order to facilitate the system integration (especiallyduring the initial test phase) and the maintenance operations.Therefore, the decision to use a common interface for all sub-detectors was made at a very early stage of the DAQ design.The required data throughput per data source is 200 MB/s(2 kB @ 100 kHz trigger rate) over a distance of 200 m. Thedata transportation hardware must be able to absorb stochasticfluctuations on the event size and provide enough contingencyto cope with large uncertainties on the LHC luminosity and thedetector occupancy/noise.In order to have a good working efficiency, the eventbuilder must receive a balanced traffic through its input ports.As shown in Tab. 1., some detectors have a large spread ofdata sizes at the output of their data sources. Therefore, thedata transportation hardware must be able to average the trafficDAQ staging 						over several FEDs by appropriately grouping FEDs with lowand high data volumes per event.Tab. 1. Front-end readout requirements* most of the time, the detector is empty, one muon track generates ~5 kB of dataThe full capacity of the DAQ will be needed only at fullLHC luminosity. A capacity of 25% of the nominal one isplanned to be available at startup, doubling after 6 months ofdata-taking to reach 100% after one year of operation. There-fore, the data transportation architecture must allow a progres-sive deployment of the DAQ.III.  D2S IMPLEMENTATIONTo fulfill all the above requirements, the D2S transporta-tion devices are the following (see Fig. 3):• a common DAQ interface housed by the data source (seeFig. 4)• an S-LINK64 [1] data link made out of a source card, acopper cable (max. length 15 m.) and an S-LINK64receiver/merger card• a Front-end Readout Link card (FRL) (described below)• a PCI Myrinet [4] network interface card• a 200 m duplex optical fiber• a Front-end builder switch (Myrinet technology)Fig. 3 D2S physical implementationA. The common FE/DAQ interface The interface specifies a common hardware platform and adata format which is essentially a header and a trailer withinformation relevant to the event building process [2]. .Fig. 4 FED block diagramThe payload of the event fragments is only inspected in thefilter farm, i.e. it is not considered during event building. As aresult of this common interface, there is no sub-detector spe-cific hardware after the FED in the CMS read-out chain.The DAQ port is an S-LINK64 interface. The S-LINK64protocol is based on S-LINK which has been extended tomatch CMS needs of higher bandwidth (64 bits data width,maximum input frequency 100 MHz). Both S-LINK and S-LINK64 only specify a set of connectors for sending andreceiving data, along with mechanical constraints for thesender and the receiver cards. The physical link between themis not specified beyond these constraints. In addition, bothinterfaces specify the protocols for writing into the sender cardand reading from the receiver card. Both protocols resemblestandard protocols that read from and write to a synchronousFIFO. The design and the implementation of the S-LINK64port on the FED is under the responsibility of the sub-detector.B. The short distance linkThe current implementation of the short distance link isfully compliant with S-LINK64 on the sender side. Thereceiver side (housed by the FRL) is designed to receive datafrom one or two sender cards hence performing the merge oftwo data sources. Detailed information on hardware imple-mentation and performance can be found in [3].C. The Front end Readout Link cardA significant number of FEDs are expected to provideevent fragments with less than 1 kB average size (see Tab.1.) In order to reduce the number of inputs to the FEDbuilders and to better exploit the available bandwidth in theevent builder, the FRL is able to merge data from two FEDsinto a single event fragment.The current design of the FRL is a Compact-PCI card (see Fig.5) with three interfaces:• Input interface which handles one or two S-LINK64cables.• Output interface to the long distance link, currently imple-mented as a PCI connector since a PCI-Myrinet NetworkInterface Card (NIC) has been chosen. This NIC, alongwith the 200 m optical fibre, implements the data link tothe counting room on the surface.    		 			     	   !"#$%&''()"#'*	$% !*	 !+FEDFRLFBRUIRUI-PCI (Myrinet LANai10)PCI 5 Gbs/s Link (Myrinet LANai10)LVDS-FED mergerUp to two FEDs per FRLLVDS FED-FRL cable(s)FED S-Link64 LVDS adapterTwo 2.5 Gb/s links per FRLDetector FrontEnd Driver FEDFrontend Readout LinkFED Builder: dual MxN switch  		 	 		 	 	 				     	!• Configuration and control interface which is implementedas a second PCI interface connected to the compact-PCIbackplane.The FRL is designed with a form factor compliant to Com-pact-PCI: the depth of the board is extended to host a standardPCI card on a PCI edge-connector: when assembled, the FRLand the PCI NIC are co-planar. Hence, the ensemble can beplugged in standard Compact-PCI backplanes mounted instandard Europe mechanics.Fig. 5 FRL physical implementationD. Long distance linkThe long distance link is implemented with commercialdevices. The technology chosen for the baseline design isMyrinet [4]. The PCI board features two "rails"  running eachat 2.5 Gb/s. Although a data source provides an average of200 MB/s, the total bandwidth of the long distance link ismore than twice the required throughput. This is needed forcompensating the FED builder utilisation efficiency that isabout 50% under the traffic conditions generated by the FEdata sources: variable and unbalanced event size and randomtraffic.E. The front-end builderAs each FRL hosts a dual rail NIC, the front-end builder isimplemented with a pair of 8 x 8 crossbar switch. Measure-ments done with different traffic conditions show that theusage efficiency of the switch is 95% (see Fig. 6) when theevents have constant size and drops to 55% when the eventshave a size variation equal to their average size (log-normaldistribution).Fig. 6 Switch efficiency for different event size variations By extrapolation, the available bandwidth for each FRL ishence 275 MB/s, well above the required 200 MB/s.The switch behavior for unbalanced data sources has alsobeen measured: the results are shown in Fig. 7. There is no sig-nificant efficiency loss when data sources are unbalanced: theefficiency is 45 % when four sources provide events with anaverage size of 1 kB along with four sources providing eventswith an average of 3 kB (unbalance ration of 3). By extrapolat-ing on this later case, the available bandwidth for each FRL is225 MB/s, still above the required 200 MB/s. With these measurements in realistic conditions, we seethat the D2S provides the required bandwidth for the Front-end data sources as well as traffic balancing. This contributesto good working conditions for the next stage of the eventbuilding process.Fig. 7 Switch efficiency for unbalanced data sourcesIV.  DEPLOYMENT SCENARIOEach FED builder has 8 output ports. The system is scal-able with the number of used ports. A DAQ reading out alldata sources but at 1/8th of maximum trigger rate is obtained(1 MB @ 12.5 kHz) by connecting one RU to each FEDbuilder and connecting these RUs to a RU builder slice.By connecting a second RU to each FED builder and a sec-ond RU builder slice (odd events assigned to output 1 and evenevents to output 2 for example), a DAQ with 1/4th capacity isobtained (1 MB @ 25 kHz).This principle works up to the full DAQ capacity (1 MB @100 kHz) by connecting 8 RUs to each FED builder and usingthe event number as base for output assignment.The D2S architecture allows a progressive deployment ofthe rest of the DAQ devices. It is also possible to use differenttechnologies when new slices must be installed and henceprofit from the future technological evolutions.V.  SUMMARYAfter reviewing the requirements of the readout of theCMS Data Acquisition system as well as the main characteris-tics of the data producers, the architecture of the Data to Sur-face (D2S) system is presented. The average amount of dataproduced varies among the data sources whereas the operationof the event builder with high efficiency requires that all inputscarry the same amount of data. The D2S is designed to solvethis problem by providing a first stage in the event buildingprocess. The D2S concentrates several data sources into an 	   			 !	 	"		 	 	01 02 03 04 05 06 07 08 09 010 00 0 .25 0.5 0.75 1 1 .25R M S /averag eMaximum Utilisation (%)EVB  tra ff ic01020304050607080901000 0.25 0.5 0.75 1 1.25RMS/averageMaximum Utilisation (%)balanced (2.0)unbalance ratio=2 (1.33 2.66)unbalance ratio=3 (1.0 3.0)Maximum Utilisation (%)output channel and multiplexes the event data to differentstreams in the second stage of the event building process. TheD2S output channels therefore provide more evenly distrib-uted data sizes to the second stage of the event builder. More-over, the multiplexing allows for a scalable design for thesecond stage of the event builder, resulting in a system that canbe procured and installed in phase with the requirements aris-ing from the performance of the accelerator and the experi-ment itself.VI.  REFERENCES[1] The S-LINK64 bit extension specification: S-LINK64A. Racz, R. McLaren, E. van der Bijhttp://hsi.web.cern.ch/HSI/s-link; The S-LINK Interface SpecificationO. Boyle, R McLaren, E. van der Bijhttp://hsi.web.cern.ch/HSI/s-link[2] Read-out Unit Working Group WEB pageshttp://cmsdoc.cern.ch/cms/TRIDAS/horizontalthen click on “RUWG” in the main banner[3] FED-Kit design for CMS DAQ systemE. Cano et al. in these proceedings[4] http://www.myri.com

CMS data to surface transportation architecture

CMS data to surface transportation architecture

Abstract

Similar works

Full text

Available Versions

CERN Document Server