In this paper we describe the LHCb Timing and Fast Control (TFC) system. It is different from that of the other LHC experiments in that it has to support two levels of high-rate triggers. Furthermore, emphasis has been put on partitioning and on locating the TFC mastership in one type of module: the Readout Supervisor. The Readout Supervisor handles all timing, trigger, and control command distribution. It generates auto-triggers as well as controls the trigger rates. Partitioning is handled by a programmable patch panel/switch introduced in the TTC distribution network between a pool of Readout Supervisors and the Front-End electronics. I

Chlopik, A

Guzik, Z

Jacobsson, R

Jost, B

R. Jacobsson

B. Jost

A. Chlopik

Z. Guzik

CiteSeerX

The LHCb Timing and Fast Control system

CERN Document Server

The LHCb Timing and Fast Control systemR. Jacobsson, B. JostCERN, 1211 Geneva 23, SwitzerlandRichard.Jacobsson@cern.ch, Beat.Jost@cern.chA. Chlopik, Z. GuzikSoltan Institute for Nuclear Studies, Swierk-Otwock, Polandarek@ipj.gov.pl, zbig@ipj.gov.plAbstractIn this paper we describe the LHCb Timing and FastControl (TFC) system. It is different from that of the otherLHC experiments in that it has to support two levels of high-rate triggers. Furthermore, emphasis has been put onpartitioning and on locating the TFC mastership in one type ofmodule: the Readout Supervisor. The Readout Supervisorhandles all timing, trigger, and control command distribution.It generates auto-triggers as well as controls the trigger rates.Partitioning is handled by a programmable patchpanel/switch introduced in the TTC distribution networkbetween a pool of Readout Supervisors and the Front-Endelectronics.I. INTRODUCTIONLHCb has devised a Timing and Fast Control (TFC)system[1] to distribute information that must arrivesynchronously at various places in the experiment. Examplesof this kind of information are:• LHC clock• Trigger decisions• Reset and synchronization commands• Bunch crossing number and event numberAlthough the backbone of the timing, trigger and controldistribution network is based on the CERN RD12 system(TTC)[2], several components are specific to the LHCbexperiment due to the fact that the readout system is differentfrom that of the other experiments in several respects. Firstly,the LHCb TFC system has to handle two levels of high-ratetriggers: a Level 0 (L0) trigger with an accept rate ofmaximum 1.1 MHz and a Level 1 (L1) trigger with an acceptrate of maximum 40 - 100 kHz. This feature reflects itself inthe architecture of the Front-End electronics, which consistsof a L0 part and a L1 part (see Figure 1). The L0 Front-End(FE) electronics samples the signals from the detector at a rateof 40 MHz and stores them during the duration of the L0trigger processing.  The event data are subsequently de-randomized before being handed over to the L1 FEelectronics. The L1 FE electronics buffers the data during theL1 trigger processing, de-randomizes the events before itzero-suppresses the data, and finally feds the data into theDAQ system for event building.Secondly, the TFC architecture has been designed withemphasis on partitioning[3]. A partition is in LHCb a genericterm, defined as a configurable ensemble of parts of a sub-detector, an entire sub-detector or a combination of sub-detectors that can be run in parallel, independently and with adifferent timing, trigger and control configuration than anyother partition.Furthermore, the aim has been to locate the entire TFCmastership of a partition in a single module. The triggerdecision units are also considered as sub-detectors.Read-out Network (RN)RU RU RU6-15 GB/s6-15 GB/s50 MB/sVariable latencyL2 ~10 msL3 ~200 msControl & MonitoringLANRead-out units (RU)Timing&FastControlLevel-0Front-End ElectronicsLevel-1VDET    TRACK   ECAL     HCAL   MUON    RICHLHC-B DetectorL0L1Level 0TriggerLevel 1Trigger40 MHz1 MHz40 kHzDatarates40 TB/s1 TB/s1 MHzFront End LinksTrigge r Level 2 & 3Event FilterSFC SFCCPUCPUCPUCPUSub-Farm Controllers (SFC)StorageThrottleFront-End Multiplexers (FEM) LANThrottleFigure 1: Overview of the LHCb readout system.II. USE OF THE TTC SYSTEMThe TTC system has been found to suite well the LHCbapplication. The LHC clock is transmitted to all destinationsusing the TTC system and Channel A is used as it wasintended, i.e. to transmit the LHCb L0 trigger decisions to theFE electronics in the form of a accept/reject signal at 40 MHz.Channel B supports several functions:• Transmission of the Bunch Counter and theEvent Counter Reset (BCR/ECR).• Transmission of the L1 trigger decision (~1.1MHz).• Transmission of Front-End control commands,e.g. electronics resets, calibration pulse triggeringetc.Table 1: Encoding of the Channel B broadcasts. “R” stands forreserve bit.7 6 5 4 3 2 1 0L1 Trigger 1 Trigger type EventID 0 0Reset 0 1 R L1EvIDL1FEL0FE ECR BCRCalibration 0 0 0 1 Pulse type 0 0Command 0 0 R R R R R RThe information is transmitted in the form of the shortbroadcast format[4], i.e. 16 bits out of which two bits arededicated to the BCR/ECR and six bits are user defined.From  the TTC bandwidth  it  follows that a  maximum of~2.5 MHz of broadcasts can be transmitted. The eight bits areencoded according to Table 1. A priority scheme determinesthe order in which the different broadcasts are transmitted incase they clash.III. TFC COMPONENTS SPECIFIC TO LHCBThe TFC architecture is shown in Figure 2. It incorporatesa pool of TFC masters, Readout Supervisors[5], one of whichis interfaced to the central trigger decision units and that isused for normal data taking. The other Readout Supervisorsare reserves and can be invoked for tests, calibrations anddebugging. The reserve Readout Supervisors also allowconnecting local trigger units.The TFC Switch[6] distributes the TTC information to theFront-End electronics and the Throttle Switches[6] feed backhardware throttle signals from the L1 trigger system, the L1de-randomizers and components in the data-driven part of theDAQ system, to the appropriate Readout Supervisors.The Throttle ORs[6] form a logical OR of the throttlesignals from sets of Front-End electronics.A GPS system allows time-stamping the local eventinformation sampled in the Readout Supervisor.TTC rx TTC rxTTC rxL 1EFEchipFEchipL1  bufferADCADCADCADCADCD SPControlFEchipTTC rxL0EiF EchipF EchipTTCrxL 0EF ipF ipFEchipTTCrxL 1EF c hipFEc hipL1 b uffe rDCA DCCA DCDSPControlC lo ck fano utBC  and BC RLHC  clo ckL0  trigg er L1  trigger  R eado ut Super viso r  R eadout S uper vis or Loca l  trigger   (optiona l)L0 L1  R eadout Sup er vis orL0 L1        TFC switch       L1  Throttle            switch      L0 Throttle             switchSD1 TTC tx SD2  TTC tx SDn TTC tx L0 TTC tx L1 TTC txOptic al coupler s Optica l co uplers Optica l coup lers Optic al coupler sTTC rxL1EFEc hipFEc hipL1 buffe rA DCA DCA DCA DCA DCDSPControlFEc hipTTC rxL 0EiFEc hipFEc hipTTC rxL0EEc iEc iF EchipTTC rxL 1EFEc ipFEchipL1  bufferDADCADCAADCD SPControlD AQ DAQThrottle ORThrottle ORTTC  system1717        L1 trig ger sy stem  GP S r ece iverFigure 2: Overview of the TFC architecture.IV. THE READOUT SUPERVISORThe Readout Supervisor has been designed with emphasison versatility in order to support many different types ofrunning mode, and modifiability for functions to be added andchanged easily. Below is a summary of the most importantfunctions. A complete description can be found in Reference[5].The TTC encoder circuit incorporated in each ReadoutSupervisor receives directly the LHC clock and the orbitsignal from the TTC machine interface (TTCmi). The clock isdistributed on the board in a star fashion and is transmitted toall synchronous destinations via the TTC system.The Readout Supervisor receives the L0 trigger decisionfrom the central L0 trigger Decision Unit (L0DU), or from anoptional local trigger unit, together with the Bunch CrossingID. In order to adjust the global latency of the entire L0trigger path to a total of 160 cycles, the Readout Supervisorhas a pipeline of programmable length at the input of the L0trigger. Provided no other changes are made to the system, thedepth of the pipeline is set once and for all during thecommissioning with the first timing alignment. The BunchCrossing ID received from the L0DU is compared to theexpected value from an internal counter in order to verify thatthe L0DU is synchronized. For each L0 trigger accept, thesource of the trigger (3-bit encoded) together with a 2-bitBunch Crossing ID, a 12-bit L0 Event ID (number of L0triggers accepted), and a “force bit” is stored in a FIFO. Theforce bit indicates that the trigger has been forced and thatconsequently the L1 trigger decision should be made positive,irrespective of the decision of the central L1 trigger DecisionUnit (L1DU). The information in the FIFO is read out at thearrival of the corresponding L1 trigger decisions from theL1DU.The RS receives the L1 trigger decision together with a 2-bit Bunch Crossing ID and a 12-bit L0 Event ID. The twoincoming IDs are compared with the IDs stored in the FIFO inorder to verify that the L1DU is synchronized. If the force bitis set the decision is converted to positive. The 3-bit triggertype and two bits of the L0 Event ID is subsequentlytransmitted as a short broadcast according to the format inTable 1. In order to space the L1 trigger decision broadcasts aL1 de-randomizer buffer has been introduced.The Readout Supervisor controls the trigger ratesaccording to the status of the buffers in the system in order toprevent overflows. As the distance and the high trigger rate,the L0 de-randomizer buffer occupancy cannot be controlledin a direct way. However, as the buffer activity is completelydeterministic, the RS has a finite state machine to emulate theoccupancy. This is also the case for the L1 buffer. In case anoverflow is imminent the RS throttles the trigger, which inreality is achieved by converting trigger accepts into rejects.The slower buffers and the event-building components feedback throttle signals via hardware to the RS. Data congestionat the level of the L2/L3 farm is signalled via the ExperimentControl System (ECS) to the onboard ECS interface, whichcan also throttle the triggers. “Stopping data taking” via theECS is carried out in the same way. For monitoring anddebugging, the RS has history buffers that log all changes onthe throttle lines.The RS also provides several means for auto-triggering. Itincorporates two independent uniform pseudo-randomgenerators to generate L0 and L1 triggers according to aPoisson distribution. The RS also has a unit running severalfinite state machines synchronized to the orbit signal forperiodic triggering, periodic triggering of a given number ofconsecutive bunch crossings (timing alignment), triggering ata programmable time after sending a command to fire acalibration pulse, triggering at a given time on command viathe ECS interface etc. The source of the trigger is encoded inthe 3-bit L1 trigger qualifier.The RS also has the task of transmitting various resetcommands. For this purpose the RS has a unit running severalfinite state machine, also synchronized to the orbit signal, fortransmitting Bunch Counter Resets, Event Counter Resets, L0FE electronics reset, L1 + L0 electronics reset, L1 Event IDresets etc. The RS can be programmed to send the commandsregularly or solely on command via the ECS interface. TheBunch Counter and the Event Counter Reset have highestpriority. Any clashing broadcast is postponed until the firstbroadcast is ready (L1 trigger broadcast) or until the nextLHC orbit (reset, calibration pulse, and all miscellanouscommands).The RS keeps a large set of counters that record itsperformance and the performance of the experiment (dead-time etc.). In order to get a consistent picture of the status ofthe system, all counters are samples simultaneously intemporary buffers waiting to be read out via the onboard ECSinterface./+&FORFN/7ULJJHUJHQHUDWRU7ULJJHUFRQWUROOHU7KURWWOHV5HVHWFRPPDQGJHQHUDWRU56)URQW(QG//'$4(&6LQWHUIDFH(&677&HQFRGHU&K$%//'HUDQGRPL]HUFigure 3: Simplified logical diagram of the Readout Supervisorshowing the basic functions.The RS also incorporates a series of buffers analogous to anormal Front-End chain to record local event information andprovide the DAQ system with the data on an event-by-eventbasis. The “RS data block” contains the “true” bunch crossingID and the Event Number, and is merged with the other eventdata fragments during the event building.The ECS interface is a Credit Card PC through which theentire RS is programmed, configured, controlled, andmonitored. Note that in order to change the trigger and controlmode of the RS for testing, calibrating and debugging it is notnecessary to reprogram any of the FPGAs. All functionality isset up and activated via parameters that can be written at anytime.7ULJJHU56 5656 56 567)&6ZLWFK7KURWWOH256ZLWFK3DUWLWLRQHOHPHQW3DUWLWLRQHOHPHQW3DUWLWLRQHOHPHQW3DUWLWLRQHOHPHQW3DUWLWLRQHOHPHQWFigure 4: The TFC architecture simplified to show an example ofpartitioning.A. The TFC Switch and PartitioningA good partitioning scheme is essential in order to carryout efficient commissioning, testing, debugging, andcalibrations. The LHCb TFC partitioning is shown by anexample in figure 4, in which the TFC architecture in Figure 2has been simplified. The TFC Switch allows setting up apartition by associating a number of partition elements (e.g.sub-detectors) to a specific Readout Supervisor (Figure 5).The Readout Supervisor can then be configured to control andtrigger the partition in whatever specific mode that isrequired. In the example in figure 4, the partition elements 2 –5 are running with the central RS, which is interfaced to thecentral triggers. Partition element 1 is simultaneously runninga stand-alone run with a separate RS. The three other ReadoutSupervisors are idle and can be reserved at any time for otherpartitions. Note that the TFC Switch is located before the TTCoptical transmitters (TTCtx) and that it is handling theencoded TTC signals electrically.The configuring of the TFC Switch is done via thestandard LHCb ECS interface incorporated onboard: theCredit Card PC.,1387&+$11(/6287387&+$11(/6&RQWURO,QWHUIDFH3RRORI5HDGRXW6XSHUYLVRUV3DUWLWLRQ$ 3DUWLWLRQ%)URQW(QGVJURXSHGE\77&W[2SWLFDOFRXSOHUVWRSDUWLWLRQHOHPHQWV7)&6ZLWFK77&LQIRUPDWLRQDV77&HQFRGHGHOHFWULFDOFigure 5: The principle of the TFC Switch.From the architecture of the system it follows that the FEelectronics that is fed by the same TTCtx is receiving thesame timing, trigger, and control information. Hence theTTCtx define the partition elements. The TFC Switch hasbeen designed as a 16x16 switch and thus allows the LHCbdetector to be divided into 16 partition elements. To increasethe partition granularity an option exists whereby four TFCSwitches are deployed in order to divide the LHCb detectorinto 32 partitions (Figure 6).7)&6ZLWFK 7)&6ZLWFK7)&6ZLWFK 7)&6ZLWFK)URQW(QG5HDGRXW6XSHUYLVRUVFigure 6: Four TFC Switches put together to increase the partitiongranularity to 32.A crucial point concerning the TFC Switch is that allinternal paths from input to output must have equalpropagation delays. Otherwise, the partition elements willsuffer from timing alignment problems using differentReadout Supervisors. Measurements performed on the firstprototype of the TFC Switch shows that it will be necessary toadd adjustable delays at the outputs due to strongly varyingpropagation delays in the 16:1 multiplexers used.B. The Throttle Switches and the Throttle ORsThe function of the Throttle Switches is to feed back thethrottle information to the appropriate Readout Supervisor,such that only the Readout Supervisor in control of a partitionis throttled by the components within that partition. Figure 4shows an example of how they are associated. The logicaloperation of the Throttle Switch is to perform a logical OR ofthe inputs from the components belonging to the partition(Figure 7). The system incorporates two Throttle Switches, aL0 and a L1 Throttle Switch. The sources of L0 throttles areessentially the components that feed the L1 trigger system.The sources of L1 throttles are the L1 de-randomizers and theevent building components.,1387&+$11(/6287387&+$11(/6OR OR&RQWURO,QWHUIDFH3RRORI5HDGRXW6XSHUYLVRUV3DUWLWLRQ$ 3DUWLWLRQ%7KURWWOH6ZLWFK)URQW(QGVHWFJURXSHGE\7KURWWOH25VLHa7KURWWOH6ZLWFKHV7KURWWOHVLJQDOVFigure 7: The principle of the Throttle Switches.For monitoring and debugging, the Throttle Switches keepa log of the history of the throttles. A transition on any of thethrottle lines trigger the state of all throttle lines together witha time-stamp to be stored in a FIFO.The configuring and the monitoring of the ThrottleSwitches are done via the standard LHCb ECS interface.The Throttle ORs group throttle lines belonging to thesame partition elements. They are identical to the ThrottleSwitches in all aspects except that they OR all inputs and haveonly one output.VI. CONCLUSIONSThe LHCb Timing and Fast Control (TFC) system and theuse of the TTC system are well established. The ReadoutSupervisor incorporates all mastership in a single module andit provides a lot of flexibility and versatility. Partitioning iswell integrated through the TFC Switch and the ThrottleSwitches.The architecture and the components have been putthrough two reviews and the system is now in the prototypingphase.V. REFERENCES[1] R. Jacobsson and B. Jost, “The LHCb Timing and FastControl system”, LHCb 2001-016 DAQ.[2] RD-12 Documentation on WWW(http://www/cern.ch/TTC/intro.html) and referencestherein.[3] C. Gaspar, R. Jacobsson, B. Jost, “Partitioning inLHCb”, LHCb 2001-116 DAQ.[4] B. Jost, “The TTC Broadcast Format (proposal)”,LHCb 2001-017 DAQ.[5] R.Jacobsson, B. Jost, Z. Guzik, “Readout SupervisorDesign Specifications”, LHCb 2001-012 DAQ.[6] Z. Guzik, Richard Jacobsson, and B. Jost, “The TFCSwitch specifications”, LHCb 2001-018 DAQ.

http://lhcb-doc.web.cern.ch/lhcb-doc/presentations/conferencetalks/postscript/2001presentations/leb2001_paper1.pdf

The LHCb Timing and Fast Control system

Abstract

Similar works

Full text

Available Versions

CiteSeerX

CERN Document Server