Rácz, A

CERN Document Server

AbstractThis document is a first attempt to define the basic func-tionnalities of the TTS in the CMS DAQ. Its role is to adaptthe trigger pace to the DAQ capacity in order to avoid conges-tions and overflows at any stage of the readout chain. The dif-ferent possibilities for the TTS to measure the load on parts ofthe chain are examined. It clearly appears that one part of thechain needs fast reaction time (few tens of microseconds)whereas the rest of the chain can afford longer reaction time,available to nowadays processors.I.  INTRODUCTIONThe trigger throttling system (TTS) is a critical element ofthe CMS data acquisition system. Its role is to ensure that trig-ger logic is not overloading any of the electronic devices incharge of moving, processing, and storing the data from thevery front-end electronic down to the storage media. By design, the DAQ is able to process events at 100 KHz(maximum average). However, due to the random nature of thetrigger, the LV1-A signal can exceed these 100 KHz for sometime. Although the DAQ is able to handle higher instantaneousLV1-A frequencies, data loss and DAQ failure may occur ifthe LV1-A frequency is not monitored and regulated accordingto the DAQ load. As inputs, the TTS receives warnings/status from all thedevices that can suffer from overload and as output, the TTSproduces a throttling signal that will reduce the trigger rateaccording to strategies to be defined later. These strategies areout of the scope of the present document.II.  FRONT END RESOURCESAccording to the front-end electronic logical model[1], thefront-end derandomizers (see Fig. 1.) are the first devices tooverflow when too fast LV1-A are issued. Space and powerconstraints in the front-end lead to small derandomizer depthand hence, these queues are very sensitive to bursty LV1-Aeven if the 100 kHz average is met. In general, these deran-domizers behave like a first-in-first-out queue: the input fre-quency is directly the LV1-A and the output frequency(detector dependent) ranges from 3us to 7us (see table in theappendix).Fig. 1. Front-End logical structureSimulations (see Fig. 2.) have been made showing the rela-tion linking the depth, the service time and the probability tooverflow. The best way to avoid overflow is to design forrobustness: that means implement a big memory depth alongwith a short service time compared to the average time intervalbetween two triggers (10us). When this is not possible sincesevere constraints on power consumption for example, otheralternatives exist to minimize the overflow probability and tolimit the consequences of it when it happens.Fig. 2. Graphs showing the overflow probability versus the service time and the queue depthA.  Overflow consequencesPractically, an overflow means that a data related to a giventrigger is overwritten in the readout channel or depending onthe implementation, a data corresponding to a trigger cannotbe introduced in the readout channel. At the level of the deran-domizer, as the data is usually not time-stamped, the data ofthe next event will be readout as being the data of the currentevent: the data belonging to two different beam crossings aremixed and this is a real disaster! But the worst is not yetthere... After an overflow, that channel (or group of channel) ismisaligned with respect to the other channels of the detectorand will continue to provide data belonging to a different trig-ger: the misaligned channel must be re synchronized to be inTrigBunch CntTTC systemEvent CntRF CLKEC ResetBC ResetLV1ATest SignalsInitializationFast ControlsParametersFront End UnitPreamp(shaper, ADC, …)DelayDerandomizerMuxFormatting* may be part of the FEFE Back Pressure4 6 8 10 12 14 16 1810−910−810−710−610−510−410−310−210−1100d: Fifo size (in events)P(n=d)s=2us s=3us s=4uss=5uss=6uss=7uss=8uss=9uss=10us1 2 3 4 5 6 7 8 9 1010−910−810−710−610−510−410−310−210−1100s: service time in usP{n=d)d=4d=6d=8 d=10 d=12d=14d=16d=18Trigger Throttling System for CMS DAQA. RaczCERN, Div. EP, Meyrin CH-1211 Geneva 23 Switzerlandstep with the others. A resynchronization on-the-fly of a givenchannel (or a group of channels) would be the most efficientregarding the system availability. Technically-wise, there arethree ways to perform this operation:• the Timing Trigger and Control system (TTC)[2] mustprovide the right bunch and event number to the mis-aligned channel to be used at restart time. Due to the ran-dom nature of the trigger, the TTC cannot know whatevent number will be associated to which bunch number.So the only possibility is to send only the current eventnumber reached at the end of the orbit gap of the LHCbeam. During the duration of the gap (127BX or 3.175us), there will be no event and at the first bunch of the neworbit, the bunch counter restarts at zero by definition• the second way uses the special commands of the TTC.When the misalignment is detected and quantified, onecould use 4 special commands to increment/decrementon-the-fly the event or bunch counter of a given channel.The misalignment monitoring process would detect some-time after the return-to-sync of the channel• finally, the third way is to use a general reset broadcastedby the TTC that will flush all the data in all the front-ends(pipeline, derandomizers,....), initialize the bunch andevent counter to zero and hold the system until the begin-ning of the next LHC orbitThe first alternative affects only a given channel but thatchannel can be unavailable up to an entire LHC orbit (~88us).The second alternative offers the smallest impact on the rest ofthe DAQ and on the un-availability time. The third one is thebrute force, affects the whole DAQ, can result in an un-avail-ability time of one orbit but the simplest to implement! Thetwo first set severe requirements on the front-end functionnali-ties at the current design stage. If none of these solutions are possible, the current run mustbe stopped in order to perform the resynchronisation. Then anew run can be started. If the overflows are frequent, this willreduce by a big factor the availability of the DAQ and conse-quently, the physics performances of CMS. B.  Overflow detectionThere is no way to detect an overflow in the front-endwithout implementing dedicated hardware that chases this spe-cific condition. Should the overflow happen (whatever we do,it will!), the status word for that event must include this infor-mation and the front-end electronic must be able to recoverfrom that situation without a general reset. In other words, anoverflowed detector must not loose the synchronicity of itsevent/bunch counter with respect to the other detectors.C. Overflow preventionIn the front-end area, the storage resources are generallysmall. Therefore, the reaction time of a feed back loop must beshort, i.e. within a few trigger intervals. Let us look at a simplereaction loop model (see Fig. 3 .)to better visualize the issues.Fig. 3. Overflow modelRealistic durations can be given to the parameters:• Sensor transmit time (T1) = 30us• TTS processing time (T2) = 4us• TTS transmit time (T3) = 1us• Trigger setup time (T4) = 5usThe total response time is 40us or 4 events in average(assuming an input rate of 100 kHz), the high limit sensor mustset at N-4, (N is the buffer capacity). During these 40us, 4events will be readout (assuming an output rate of 100 kHz)and 4 events will enter the buffer. But these are only averagevalues. The buffer would overflow if 9 triggers or more aregenerated. According to the Poisson distribution, the probabil-ity to have 9 or more triggers when an average of 4 is expectedis still 2.1%. If by design, the output service time is shorter(i.e. 7us, ~6 events can be removed from the buffer), the situa-tion is more comfortable: 11 triggers during 40us are needed tooverflow the same buffer. The Poisson probability of this situ-ation is 0.3%. The main problem with this solution is to use asignificant part of the buffer as safety margin and of course theneed of a complex alarm transmission network.A second alternative to reduce the overflow probability isto reduce the reaction time. If hardware emulators (statemachines) are used to know the buffer status at any time, theoverflowing trigger can be inhibited and hence the overflowavoided. Applied to the previous model, T1 is now 0us: thetotal response time is 10us. If the threshold is set to N-1, 3 trig-gers are needed to overflow the buffer. The Poisson probabilityof having 3 events or more when 1 is expected is close to 8%!That means that the threshold must be set lower: for N-2, theoverflow probability is 1.9% and for N-3, 0.4%. With thisalternative, a complex network is avoided but emulation hard-ware has to be developed. Regarding the buffer usage, the situ-ation is still not satisfactory.A last alternative is to setup rules on the trigger: the idea isto identify the overflowing trigger sequence for each front endsystems and implement in the trigger logic a sequence monitorthat will regulate the burstiness of the trigger and hence inhibitthe killing trigger of a sequence that overflows one front endMonitoredBufferTriggerTTSHigh limitsensorT1T2T3T4CommandNetworkFree spacebefore overflowrateii IFreeT ≤∑=41InputOutputsystem. For example, these rules can be: at least 2 bunch cross-ings between 2 consecutive triggers, no more than 8 triggersper LHC orbit, etc.... These rules are of course highly depen-dent on sub-systems and require a deep knowledge of them.The drawback of this option is that the global trigger effi-ciency may be reduced by an inacceptable factor. Studies andsimulation should allow the estimation of this inefficiency.When numbers smaller than a percent are written, usually,one could think that the occurrence of that event is rare andmarginal. In the case of the front-end, with an average timebetween triggers of 10us, 1% overflow probability means oneoverflow every millisecond of beam (!!!) and that is dramatic:if, by design, the only solution to recover from a misalignmentis a reset of the front end chips, events located in the delay lineand derandomizers are lost. Let’s take a typical derandomizerdepth of 8 events: 8 events are lost every millisecond (assum-ing no event in the delay line). For an average of 100 possibleevents during 1 ms, 8 of them are lost... In other words, this1% overflow probability creates 8% inefficiency for the DAQ!Other queue depth and service times are represented in thegraphs below (see Fig. 4 .).This simple calculation does not consider neither the timeto detect the overflow, time during which the data are cor-rupted and hence useless nor the fact that if the front-ends of asub-detector are independent from each other, the overflowprobability of that sub-detector is the overflow probability fora single front-end multiplied by the number of front-ends inthat sub-detector.Whatever the overflow reduction techniques will be, it isvital to maintain its probability in the 10-3 region or wellbelow (if independent front-ends) and to implement efficientrecovery procedures Fig. 4. Graphs showing the mean time between overflow and the DAQ efficiency loss for different derandomizer depth and for different service time when the recovery procedure is a general resetDuring 1999, a TRIDAS workshop1 addressed extensivelythis critical front-end issue: the table of the appendix summa-rizes the situation of every front-end regarding the buffer over-flow. More information is available in the proceedings2.Without making any assumptions on the final choice, it is verylikely that a combination of the last two alternatives will beused. For obvious technical reasons, the physical location of thestate machines and the trigger rule sequencer is within the glo-bal trigger logic.III.  READOUT UNIT RESOURCESThe storage devices of the Readout Unit (RU) are the nextelements of the acquisition chain subject to overflow. Thesestorage elements are the Detector Dependent Unit or DataConcentrator Card (DDU/DCC) and the Readout Unit Mem-ory (RUM) (see Fig. 5 .).Fig. 5. Readout Unit block diagramThe physical location of the DDU is in the undergroundcounting rooms whereas the RUM is located in the surfacebuilding: 100m (200m cable path) are in-between. The DCCprovides more or less 2 KB of data per trigger: this makes 200MB/sec. average data rate, assuming 100 kHz average triggerrate. Currently, as far as the protocol is known, the event dataare removed from the DCC by a transparent data link at 400MB/sec.: this significantly higher throughput is needed toabsorb the fluctuations in the event size and also avoids theoverflows in the final DCC storage buffer when nominalworking conditions are met. If the data transfer is interrupteddue to some failure (hardware or software), a classical ava-lanche of “full flags” will be issued.Regarding the RUM, the situation is quite different fromthe front-end: almost no constraint on the memory size, I/Oprocessors able to run complex monitoring/recovery tasks,standard fast network access (i.e. Gigabit Ethernet). By design,the memory is able to handle up to 100000 events (1 second ofdata taking). Given this capacity, the reaction time can be in1.Front End and Readout Unit Workshop June 15-16, 1999 at CERN6.0 8.0 10.0 12.0 14.0 16.0Fifo size (events)10-410-310-210-1100101102103104Mean time between overflow (seconds)Service time 7usService time 5us6.0 8.0 10.0 12.0 14.0 16.0Fifo size (events)10-810-710-610-510-410-310-210-1DAQ efficiency lossService time 7usService time 5us2.http://cmsdoc.cern.ch/ftp/distribution/Meetings/TriDAS.workshops/99.06.15/Proceedings.htmlSwitch Data LinkDetector data linkFEDRUIRUMRUORUSRCRUI busRUO busRUDDU/FED/DCCthe region of the processor times to signal a problem to theTTS.So regarding the Readout Unit, assuming no failure in thedata transfer, there is no intrinsic weakness requiring a specialcare.IV.  ROUTING NETWORK RESOURCESAs far as the architecture is defined, the routing networkwill be assembled from commercial units provided by the tele-com industry. Although the final product is far from being cho-sen, they all include (with no exception) built-in monitoringcapabilities and hence the TTS can be notified in the event ofcongestions. V.  FILTER FARM RESOURCESFor what is concerning the filter farm, a distinctionbetween the global load of the farm and the individual load ofa single Filter Unit (FU) must be made.A. Filter Unit loadThe Filter Unit is made out of a Builder Unit (BU) and acommercial-off-the-shelf computer or cluster of computers(called “workers” in the following) running dedicated code.The BU receives the data to process from the routing networkunder the control of the Builder Manager (BM). The task ofthe BM is to assign a given event (or a set of events) to an indi-vidual BU. When an event is fully re-assembled, one of the“workers” starts the data processing. When the process is com-plete, the worker starts another event and so on. As the eventsare processed, the event queue of the BU is emptied. The BUcan then requests another set of events. In principle, the loadon the BUs and the workers should be balanced but given theevent size distribution, some devices can be more loaded thanothers. Monitoring processes will run concurrently to ensurethe proper operation of the FU. Should an overload occur(storage or processing overload), the FU can signal the prob-lem to the “Farm Manager” or the BM who, in turn, will takecorrective actions (to be defined later).To know the load of the farm, status of each individual FUscan be collected but this will result in an important traffic onthe Computing Service Network (CSN). A global load mea-surement can be performed at the BM level.B. Filter Farm loadAs described in the above paragraph, the BM grants a par-ticular event of the RU to a particular FU. As the RU can han-dle 100000 events, the BM has a pool of 100000 “names” withwhich it will name the outcoming event from the RU. When aname is in-use, it cannot be assigned to another event: it isbusy until the event analysis completion. At the end of theevent processing, the concerned FU returns a status messagewith the “name” of the processed event. This name can againbe re-used for another event of the RUs. With this mechanism,the BM can easily measure the global load of the farm bycounting the number of names in-use. VI.  CONCLUSIONThe role of the CMS Trigger Throttling System has beendescribed in global terms. The situation regarding the differentstages of the data acquisition chain is quite different. Intrinsi-cally, the most problematic parts are the front-end systemswhere full custom solutions must be developed. For the rest ofthe chain, standard and well known solutions can be used. After this first analysis, it has been shown that the TTS can besplit logically and physically into two parts:• a first one featuring quick reaction time, custom hardware,located in the global trigger logic• a second one with slower reaction time, running on theBM processorsFinally, depending on the overflow recovery procedure, theDAQ availability time can be reduced by an inacceptable fac-tor.VII.  REFERENCES[1] Front End Logical Model in CMS DAQS. CITTOLIN, J.F GILLOT, A. RACZCMS Technical note CMS-TN 96/015available on http://cmsdoc.cern.ch[2] Timing Trigger and Control reference WEB sitehttp://www.cern.ch/TTC/intro.htmlVIII.  APPENDIXBuffersize(events)Servicetime(us)Overflow probabilityEmulation WarningsignalTriggerrulesPixelDetectorNA NA ? not possible (all the front-ens are inde-pendentstatus bit in the data stream (3 levels of warning)orbit gap extended from 127BX to 128BX and empty orbit every secondTrackers 6 7 0.8%a(APV6)a.extracted from fig.1APV logic itself or FPGA (see CMS note 1999/028no at least 3 BX between consec-utive triggers, other rules seems inefficientPreshower 8 7.2(5.4)0.2%(4.10-5)easy state machine yes no more than N+7 triggers during N x service timeECAL 16 7 1.10-5 easy emulation if neededno (considered to be too slow)noneHCAL ? ? estimated to zerobb.If the trigger rule is appliednot needed not needed no more than 22 triggers per orbitMuon DT ? ? very low not possible (occu-pancy dependent)? noneMuonRPC? ? very low not possible (occu-pancy dependent? noneMuonCSC~85 0.3/3/26 0.3event/24hours? ? ?

Trigger throttling system for CMS DAQ

Abstract

Similar works

Full text

Available Versions

CERN Document Server