The CMS Data Acquisition system is designed to build and filter events originating from approximately 500 data sources from the detector at a maximum Level 1 trigger rate of 100 kHz and with an aggregate throughput of 100 GByte/s. For this purpose different architectures and switch technologies have been evaluated. Events will be built in two stages: the first stage, the FED Builder, will be based on Myrinet technology and will pre-assemble groups of about 8 data sources. The next stage, the Readout Builder, will perform the building of full events. The requirement of one Readout Builder is to build events at 12.5 kHz with average size of 16 kBytes from 64 sources. In this paper we present the prospects of a Readout Builder based on TCP/IP over Gigabit Ethernet. Various Readout Builder architectures that we are considering are discussed. The results of throughput measurements and scaling performance are outlined as well as the preliminary estimates of the final performance. All these studies have been carried out at our test-bed farms that are made up of a total of 130 dual Xeon PCs interconnected with Myrinet and Gigabit Ethernet networking and switching technologies

/UC, San Diego /INFN, Legnaro /CERN /UCLA /Santiago de Compostela U. /Lisbon, LIFEP /Fermilab /MIT /Boskovic Inst., Zagreb

Brett, A.

Cano, E.

Cittolin, S.

Erhan, S.

Gigi, D.

Glege, F.

Gomez-Reino Garrido, R.

Gulmini, M.

Gutleber, J.

Jacobs, C.

Maron, G.

Meijers, F.

Meschi, E.

Oh, A.

Orsini, L.

Pieri, M.

Pollet, L.

Racz, A.

Rosinsky, P.

Sakulin, H.

Schwick, C.

UNT Digital Library

Available on CMS information server CMS CR 2006/033CMS Conference Report15 June 2006CMS DAQ Event Builder Based on GigabitEthernetM. Pieri , G. Maron , A. Brett , E. Cano , S. Cittolin , S. Erhan  , D. Gigi , F. Glege ,R. Gomez-Reino Garrido , M. Gulmini	 , J. Gutleber, C. Jacobs, F. Meijers, E. Meschi, A. Oh,L. Orsini, L. Pollet, A. Racz, P. Rosinsky, H. Sakulin, C. Schwick, J.Varela , J. Branson ,S. Murray , I. Suzuki , R. Arcidiacono , K. Sumorok , V. Brigljevic  AbstractThe CMS Data Acquisition system is designed to build and lter events originating from approxi-mately 500 data sources from the detector at a maximum Level 1 trigger rate of 100 kHz and with anaggregate throughput of 100 GByte/s. For this purpose different architectures and switch technologieshave been evaluated. Events will be built in two stages: the rst stage, the FED Builder, will be basedon Myrinet technology and will pre-assemble groups of about 8 data sources. The next stage, theReadout Builder, will perform the building of full events. The requirement of one Readout Builder isto build events at 12.5 kHz with average size of 16 kBytes from 64 sources. In this paper we presentthe prospects of a Readout Builder based on TCP/IP over Gigabit Ethernet. Various Readout Builderarchitectures that we are considering are discussed. The results of throughput measurements and scal-ing performance are outlined as well as the preliminary estimates of the nal performance. All thesestudies have been carried out at our test-bed farms that are made up of a total of 130 dual Xeon PCsinterconnected with Myrinet and Gigabit Ethernet networking and switching technologies.Presented at CHEP06, Mumbai, India, 13-17 February 2006University of California, San Diego, California, USAINFN - Laboratori Nazionali di Legnaro, Legnaro, ItalyCERN, Geneva, SwitzerlandUniversity of California, Los Angeles, California, USAUniversidad de Santiago de Compostela, Santiago, SpainLIP, Lisbon, PortugalFNAL, Chicago, Illinois, USAMassachusetts Institute of Technology, Cambridge, Massachusetts, USARudjer Boskovic Institute, Zagreb, CroatiaFERMILAB-CONF-06-449-E1 THE DATA ACQUISITION SYSTEM OF THE CMS EXPERIMENTThe CMS experiment is one of the 4 experiments that are being installed at LHC, the 14 TeV centre of mass energyproton-proton collider that is scheduled to start operating at CERN in the year 2007.The DAQ system plays a key role in CMS. The beam crossing rate at LHC will be 40 MHz and it will be impossibleto write all interactions to mass storage. The rate of events that will need to record for ofine processing andanalysis is of the order of 100 Hz. Therefore we must be able to select interesting events with a rejection powerof O( ﬀﬂﬁ). This is achieved in two steps: a hardware Level-1 trigger which has a maximum accept rate of 100kHz and a high level software trigger (HLT) with a rejection of O( ﬀﬂﬁ). In CMS all events that pass the Level-1trigger are sent to a computer farm (Filter Farm) that performs physics selections, using the ofine reconstructionsoftware, to lter events and achieve the required output rate.The design of the system is kept as modular as possible in order to expand the system when the luminosity increasesand maintain the exibility of implementing parts of the system when new technologies will be available or newrequirements will be identied. The design of the CMS Data Acquisition System and of the High Level Trigger isdescribed in detail in the Technical Design Report [1].EVM  Readout Builder 864x64EVM  Readout Builder 264x64FBEVM2 641 64RUBU8x818x8 8x802x8Event Manager Readout Builder 164x64Filter Farm data netwrokData Links (2.5 Gb/s x 1024) FED Builder (8x8 switches x 64) 1 512FRLTTCTTSFEDGTPGTPeDetector Front-End Drivers ( FED x ~700 )Front-End Readout Links (FRL x 512) USCSCXFigure 1: Sketch of the CMS DAQ system.Figure 1 shows a simplied sketch of the CMS DAQ system. The data ow is from top to bottom in the gure.At the top are drawn the detector Front-End Drivers (FEDs) from which data are read into the 512 Front-EndReadout Links (FRLs) that are able to merge the data of up to four FEDs together. The outputs of 8 FRLs arethen pre-assembled in the FED builder and data are sent to one or more Readout Builders (RU Builder slices). TheRU Builders are independent systems in charge of building the full events, performing the physics selections andforwarding the selected events to mass storage. The number of RU Builders in the system will grow up to 8 at thedesign luminosity of 10cm ﬃsec ﬃ that will be reached after a couple of years from the start-up. Figure 2 showsa larger picture of one slice of the RU Builder and indicates its various components. In the baseline congurationa RU Builder is made up by 64 Readout Units (RU) and 64 Builder Units (BU) connected together by a switchingnetwork. An Event Manager (EVM) supervisions the data ow in the RU Builder. From the BUs all events aresent to the Filter Units (FU) through another switching network, the Filter Data Network (FDN). In the baselineconguration the number of FUs per RU Builder is 256.2  "! #%$&('%! $*)  +-,/.1032/('%! $4)56-7 8:9;(7<>= ? @A@CB(=D = 8CEBCFGF8(=H	IKJJﬂLNM 7 ? OCPQFH	IKJSR>T ? 7 UCB(= FV T 9W X ;(=ZY-FH[\ ]^_5 `H	IKJ*a 6b< D5 cH ? 7 d B(=efegH	R FQh1? d EiBCFj BC;CUA8 T d[ OC? d FR>T ? 7 UCB(=[ OC? d Fk:l(m:nAop-q(n(qCrm(st u vwNxtyNxz{|}~~z4A (ŁG 	>	 	 3C(%	  	C 	(  	4((A	 ﬂ¡3 (¢ £ (	>¤4A ¥1	§¦	¢ £  	K(	Z¨©¤4C ¥1	Z¦ª/«Q¬ª®­¬­°¯±¬²°«Q¬³´µ¶ ·§¸¹¸¯¸¹¸«º¸¹¸¯¸®»­Figure 2: One Readout Builder slice.2 EVENT BUILDERAt the design luminosity the Level-1 trigger accept rate is 100 kHz and the data size of events is approximately 1MByte. Consequently the required performance of the RUs and the BUs is an in and out data ow of 200 MByte/s.The aggregate throughput from the detector to the Filter Farm is almost 1 Tbit/s and is very demanding in terms ofnetworking.2.1 NetworkingAt present there are two candidate network technologies for the data transfer in the event builder: Myrinet [2] andGigabit Ethernet (GbE). They have some different characteristics: Myrinet has a data link speed of 2 Gbit/s insteadof 1 Gbit/s of GbE, it has a lower latency, it implements hardware ow control and, while the Myrinet interfacecards exploit the onboard CPUs, Gigabit Ethernet uses a lot of computing resources on the hosts when one usesa reliable protocol such as TCP/IP. Finally with Myrinet it is relatively easy to implement custom drivers in theon-board CPU.For the FED Builder and the data transfer from the Front-End electronics, located in the underground cavern next tothe detector, to the surface [3] Myrinet was chosen because, in addition to the aforementioned reasons, it providesber optic links and modest cost. The data transfer in the RU Builders can be based on Myrinet or GbE while theFDN should use Gigabit Ethernet because it will not be possible to equip all FUs with Myrinet network interfacesand also the FU input throughput is only ¼ 50 MByte/s and one inexpensive on-board GbE link is largely sufcient.2.2 FED BuilderThe FED Builder pre-assembles events fragments of an average size of 2 kBytes, coming from the FRLs, intolarger fragments and distributes them to the multiple RU Builder slices. For random trafc the network efciencyis approximately 50%, due to head-of-line blocking. A throughput of 230 MByte/s per node is achieved for thenominal fragment size of 2 kBytes by using two Myrinet links. The output average fragment size is 16 kBytes butit could be increased, if necessary, to 32 kBytes by having 16 ½ 16 FED Builders.2.3 Readout BuilderFor the RU Builder both networking options are still open. We have already shown that, by implementing abarrel-shifter type [4] trafc shaping, we were able to approach 100% link utilization with Myrinet [1]. With thisoption one Myrinet port on the RUs and BUs would be enough for the RU Builder task. The other option weare considering is the use of the TCP/IP protocol over Gigabit Ethernet. In this case, we connect the PCs in the3RU Builders with two links, referred to as ‘rails’ in the following, in order to reach the needed throughput of 200MByte/s per node. We will show that already with present PCs we are able to satisfy this requirement.3 TCP/IP OVER GIGABIT ETHERNETThe choice of TCP/IP over Gigabit Ethernet has the advantage of using completely standard hardware and software.TCP/IP is a reliable protocol and we do not need to worry about packet loss at the application level which typicallyoccurs when operating close to wire speed. The main drawback is the considerable usage of machine resources forits operation.In order to efciently use TCP/IP in the Event Builder we had to optimize its use in the Linux operating system. TheDAQ software is built upon the XDAQ [5] framework and the Asynchronous TCP/IP transport software package,ATCP[6], is part of it. ATCP has been developed to decouple the DAQ applications from the networking software.It also avoids the head-of-line blocking when more than one host is trying to send data to the same network interfaceof another host. In order to achieve good performances the following design choices were made:i) ATCP puts all messages to be sent in different queues according to the destination and asynchronouslyprocesses them in another thread. To avoid blocking it writes(reads) into(from) a given socket until it blocksand as soon as it blocks it passes to another socket and continues.ii) We use Ethernet Jumbo frames. By increasing the maximum transmission unit (MTU) from the standard1500 bytes to 7000 or 8000 bytes we observe an increase in performance of approximately 50% for largefragments.iii) We need to implement multi-rail operation. In principle it would be possible to use port bonding softwaresuch as the one provided by Linux or the Intel ANS teaming driver. Even if they seem to work well, wediscarded this option because of the port bonding support requirement in the switches. What we do instead,is to use different physical networks depending on the source and destination hosts. For example in the RU-BU communication, we can send data from even hosts to even hosts through network A, from even hosts toodd hosts through network B, from odd hosts to even hosts through network B and from odd hosts to oddhosts through network A.iv) We have different requirements when sending event data and control messages that steer the Event Builderoperation, the latter need to be fast and have low throughput. There are two options in TCP/IP: to set theNagle algorithm [7] on or off. When Nagle algorithm is on and there are not many messages in the pipelinein can happen that the message is delivered with a very large latency. On the other hand, when the Naglealgorithm is off we have observed an important decrease with time in throughput for a socket that sendssmall packets of variable size. The solution is to establish different connections for the data and controlmessages and to set Nagle algorithm on for data and off for control messages.4 GBE EVENT BUILDER ARCHITECTUREIn this chapter we will review the performance of various GbE RU Builder architectures. All studies and integrationtests for the DAQ system are carried out in the Pre-series system. It corresponds to approximately one of the eightslices of the nal system and is composed of over 100 dual 2.6 GHz Xeon PCs. There are 64 PCs equipped withdual port Myrinet LANai-X and quad port Gigabit Ethernet NICs that can be operated as RUs or BUs and 16 PCswith only one additional on-board GbE interface that can be used as Filter Units or Event Manager. All of themare interconnected by means of a Force10 E1200 switch [8].4.1 Baseline ConfigurationIn the baseline conguration in each RU Builder slice there are three layers of PCs: RUs, BUs and FUs (seeFigure 2). The Readout Units have Myrinet input and GbE output, the BUs have GbE input and output and the FUshave only GbE input with a throughput of 1/4 compared to RUs and BUs. The Filter Unit output to mass storageis negligible. Given the relatively large consumption of CPU resources by TCP/IP compared to Myrinet, the mostloaded part of the system in this conguration is the Builder Unit layer.Figure 3 shows the results of tests in the baseline conguration. We consider three different congurations:4¾ 30 RUs ½ 30 BUs where the Builder Units discard the events as soon as they are built;¾ 12 RUs ½ 12 BUs with all Builder Units sending data to 4 Filter Units each;¾ 30 RUs ½ 30 BUs with 4 of the BUs connected to 4 FUs each while the others discard the events as soon asthey are built. In this case the througput is measured separately for the BUs discarding events and those thatsend events to the Filter Units.A throughput on the BUs of ¿ 170 MByte/s at the nominal fragment size of 16 kBytes is reached. We also veriedthat with faster PCs (3.2 GHz CPUs) we have an increase in the throughput of approximately 30%, proportional tothe CPU speed increase. Further improvements are expected from the dual core CPUs that are now available.Fragment Size (Bytes)103 104Throughput/node (MByte/s)05010015020025030030 RU x 30 BU12 RU x (12 BU + 4 FU)30 RU x (4 (BU + 4 FU) + 26 BU) - BU with FU30 RU x (4 (BU + 4 FU) + 26 BU) - BU without FUFigure 3: Throughput per node as function of the fragment size for the baseline RU Builder.4.2 Folded Event BuilderIn addition to the baseline conguration, in the DAQ TDR [1] the conguration where a RU and a BU operate onthe same PC was considered (‘folded’ conguration). In this case only 64 PCs would be needed for the RUs andthe BUs, the full duplex features of links and switches would be exploited and the throughput on each PC wouldbe doubled. The total design throughput in each node should be 200 MByte/s Myrinet input plus 200 MByte/sGbE input and 400 MByte/s GbE output. We performed some tests of the folded RU Builder but in case of GigabitEthernet the throughput is still too low because of CPU limitations. Nevertheless we were able to verify the scalingof the ATCP transmission up to 60 RUs x 60 BUs, which is almost the nal size of the RU Builder. Without inputto the RUs and output from the BUs an in and out throughput of ¿ 180 MByte/s per node at the nominal fragmentsize of 16 kBytes was reached which is consistent with the measurements in the baseline conguration, assumingthat the limitation comes from the aggregate TCP throughput per host, i.e. ¼ 350 MByte/s.4.3 Trapezoidal ConfigurationConsidering the fact that the Builder Units are the most loaded part in the GbE RU Builder and that most of theirload is related to receiving and sending data with TCP/IP, we could make a modied trapezoidal conguration for5the RU Builder, sketched in Figure 4.Builder UnitFilter UnitSub-farmsBUFU11 1Global TriggerProcessorFEDDAQ linksFED Builders256128512ReadoutUnitsEventManager1 64RUDetector Front-EndsControlandMonitorReadout Builder NetworkBCNRCNLV1AaTTSTTCsTTSTPDFilter Farm NetworkFigure 4: Sketch of the trapezoidal RU Builder.The Builder Unit PCs layer, visible in Figure 2, is removed, the 64 Readout Units are directly connected over tworails with all the 256 Filter Unit PCs and events are built in those same PCs, referred as BU-FU. The number ofconnections from the RUs increases from 64 to 256 but the throughput on the BU-FUs and therefore on the switchoutput ports is smaller. This also reduces the requirements for the switch performance and allows for examplethe use of oversubscribed line-cards. The most loaded part of the system in this conguration is the Readout Unitlayer.With the pre-series system it is possible to test almost one quarter of the nal RU Builder slice, a 16 RU À 60BU-FU conguration.Figure 5 shows the results. We reach a throughput on the RUs of 240 MByte/s for the nominal fragment size of 16kBytes, with a network utilization of almost 100%.To approach the full scale test in terms of number of connections from a RU to the BU-FUs we run multiple BU-FUapplications (up to 4 per node). By doing this we obtained similar results up to a 16 RU À 240 virtual BU-FUconguration.These results show that there is no or limited decrease in performance when increasing the number of outputsockets from RUs to BU-FUs. The system scales with the number of BU-FUs and a trapezoidal RU Builder shouldsatisfy the design throughput requirements.In this trapezoidal conguration, the task of building the events is carried out by the Filter Units PCs and thisslightly reduces the CPU resources available for running the HLT algorithms. On the other hand, this load is rathersmall and we veried that, for an input throughput of 50 MByte/s, the FUs only use about 5-10% of their CPU tobuild the events.5 SUMMARYWe reviewed the requirements and the performance of the CMS two-stage DAQ Event Builder. By using TCP/IPover Gigabit Ethernet in the second stage of the Event Builder it is possible to achieve the design performance. Inthe trapezoidal conguration we are able to saturate 2 GbE links in our test-bed system and achieve a throughputof 240 MByte/s per node at the nominal fragment size of 16 kBytes. This is obtained using different sockets withdifferent options, static multi-rail communication and jumbo frames.6Fragment Size (Bytes)103 104Throughput/RU node (MByte/s)050100150200250300Trapezoidal 16 RU x 60 BU-FUFigure 5: Throughput per Readout Unit as function of the fragment size for the trapezoidal RU Builder.We are now in the process of nalizing the RU Builder design and we are carrying out tests of PCs in view of thepurchase of the nal hardware that should be ordered in summer 2006.We will then install and commission the DAQ system that should be ready for data taking in summer 2007.References[1] CMS Collaboration, CERN/LHCC/2002-26, CMS TDR 6.2, 15 December 2002.[2] Myrinet products from Myricom, Inc. Arcadia, CA, USA, see http://www.myri.com.[3] R. Arcidiacono et al. The 2 Tbps Data to Surface System of the CMS DATA Acquisition, 14 ÁÂ IEEE-NPSSReal Time Conference, 2005, Stockolm, Sweden.[4] E. Barsotti, A. Booth and M. Bowden, Effects of Various Event Building Techniques on Data AcquisitionSystem Architectures, Santa Fe Computing 1990:82-101.[5] V. Brigljevic et al. Using XDAQ in Application Scenarios of the CMS Experiment, Computing in HighEnergy and Nuclear Physics, 24-28 March 2003, La Jolla, USA.See also http://xdaqwiki.cern.ch[6] See http://xdaqwiki.cern.ch/index.php/Ptatcp[7] R. Stevens, Advanced Programming in the UNIX Environment, Addison Wesley, 1992.[8] See http://www.force10networks.com7

CMS DAQ Event Builder Based on Gigabit Ethernet

https://digital.library.unt.edu/ark:/67531/metadc881745/m2/1/high_res_d/897264.pdf

CMS DAQ Event Builder Based on Gigabit Ethernet

Abstract

Similar works

Full text

Available Versions

UNT Digital Library