Baden, A

Grassi, T

Hazen, E S

Rohlf, J

Wu, S

CERN Document Server

THE CMS HCAL DATA CONCENTRATOR: A MODULAR,STANDARDS-BASED IMPLEMENTATIONE. Hazen, J. Rohlf, S. Wu, Boston University, USAA. Baden, T. Grassi, University of Maryland, USAAbstractThe CMS HCAL Upper Level Readout system processesdata from 9300 detector channels in a system of about 26VME Crates. Each crate contains about 18 readout cards,whose outputs are are combined on a Data ConcentratorCard, with real-time synchronization and error-checkingand a throughput of 200 Mbytes/s. The implementation ismodular and based on industry and CERN standards: PCIbus, PCI-MIP and PMC carrier boards, S-Link and LVDSserial links. A prototype system including front-end emula-tor, HTR cards and Data Concentrator has been prototypedand tested. A VME motherboard provides a standard plat-form for the data concentrator. Implementation details andcurrent status are described.HRCOn−detector Front End:  QIE ADCGigabit Optical Link TxFanoutOpticalTTC18 HTR Cardsper VME crateHTRHTRHTR3 Channels/fiber@ 1.6 Gbit/sDCCLocal TTC Fanout (BCR, ECR, CLK)80 Mbyte/sLVDS SerialDAQ200Mbytes/sAverageL1 CMSCalorimeterTriggerVitesse800Mbit/sCopperTTCFigure 1: HCAL DAQ Crate1 OVERVIEWThe CMS HCAL trigger/DAQ system consists of about(26) 9U VME64xP crates (Fig. 1) with up to 18 HCALTrigger Readout (HTR) modules, one Data ConcentratorCard (DCC), and one HCAL Readout Controller (HRC).Front-end data is carried from the on-detector front-endelectronics to the crate by 100m optical fibers, each car-rying 3 front-end channels. LVDS data links are used totransfer data from the HTR modules to the DCC and for lo-cal fanout of TTC (Trigger, Timing, Control) signals. Theprimary DAQ output is via an S-Link/64[1] carrying an av-erage data volume of 200 Mbytes/s from each crate.2 HCAL TRIGGER READOUT CARDThe HTR module is a 9U VME module (Figure 2) equippedwith optical receivers, TTCrx circuitry, outputs on serialLevel 1 PipelineSerialLinkBoardLevel 2 PipelineLevel 1 PipelineLevel 1 PipelineTriggerPrimitivesToLevel 1TriggerLevel 1+2DataToDCCFEData8 or 16 Fibers24 or 48 ChannelsTTCRx Fanout BC0, ClockLocal BC0, ClockFigure 2: HCAL Trigger Readout CardLVDS (Channel Link) and a custom mezzanine card. Theoptical inputs receive data from the HCAL front-end elec-tronics, with one charge sample per bunch crossing (BX).The high-speed serial inputs require special board layouttechniques. The CMS HCAL is a trigger detector, thusthe HTR includes two data pipelines: the trigger pipeline,which assigns Front-End data to a BX and sends them tothe CMS regional trigger, and the DAQ pipeline where theFE-data are pipelined, triggered and sent to the Data Con-centrator Card.916816MIP bitL1-TPGFilterET &Compr.(LUT)E T,compr9µwindowLineariz.LUTESLB-PMC(ECALdesign)OTERXDataCh CRecov.CKSystem CKGOLDeserFibertoFiberAlignmentLocalClockSynch Test RAMCh BCh AFigure 3: HTR Input and Level 1 PipelineThe HTR input processing and Level 1 Pipeline is shownin Figure 3. The raw fiber data stream is deserialized, thensynchronized to the local clock. A programmable delay ofup to a few clocks is used to align data from different inputfibers. A test RAM can substitute for the input data stream.Finally, the 3 channels carried on one fiber are demulti-plexed. Each channel is then fed to a linearizing look-uptable which converts raw input data to a 16-bit linear en-ergy value. Next a finite-impulse response (FIR) filter isused to subtract the pedestal and assign all the energy toa single bunch crossing. This performs the same functionas a traditional analog shaper, but has the advange of beingeasily reprogrammable. Finally, the energy is convertedtoand compressed to 8 bits according to a non-lineartransformation specified by the CMS level 1 calorimetertrigger, and a comparison is done to see if the signal mayrepresent a muon. This compressed output plus a muon IDbit is sent to level 1. The final synchronization and serialtransmission is performed by a Synchronization and LinkBoard (SLB) described in detail elsewhere[2]. The latencyof the level 1 pipeline is critical; it must be less than 23 BX periods. Currently the theoretical minimum for theHTR implementation is 16 BX periods.	     ﬀ ﬁﬃﬂ "!  #$%'&(ﬃ)* +-,./.	021/345ﬃ67 89(;:'<=7>@?/A B CD EGFIH JKKﬂL  ﬁ;ﬂ&M N"N"OPrimitives$7PQ1RS TSUS"VW6XYS  ZO,7ﬂ[! ﬁG\ +ﬃ]_^\MﬂK* `ﬃa@bZeroSup.,7ﬂ[! ﬁﬀ\ +ﬃ]_^,ﬂK* `9 c@?/A B C@D EﬀFH JKKﬂL  ﬁﬀﬂFigure 4: HTR Level 2 (DAQ) PipelineThe HTR Level 2 Pipeline is shown in Figure 4. Firstis a pipeline of programmable depth which stores data dur-ing the CMS level 1 latency period (a fixed value). Thencomes a “derandomizer” buffer into which data is copiedat each level 1 accept. The derandomizer can hold up to10 charge samples (one per BX) per event although cur-rently we anticipate only processing 5 samples. Note that agiven charge sample can in principle participate in multipleevents, so the pipeline-to-derandomizer copy logic musthandle overlapping events. From the derandomizer, datais linearized by a LUT, filtered by an FIR filter similar tothat in the level 1 pipeline, and a threshold is applied forzero-supression. At this point either the output of the filter,the raw data or both may be inserted into the output datastream.A similar pipeline is used to store the level 1 triggerprimitives, synchronized with the corresponding level 2data. Finally the data is packaged in a variable-length blockformat along with any error information from the inputlinks and transmitted using an LVDS serializer to the dataconcentrator.3 DATA CONCENTRATOR CARDThe Data Concentrator Card is composed of a VME moth-erboard, six LVDS link receiver boards and a PMC-typelogic board. The motherboard is a VME64x 9Ux400mmsingle-slot module. The motherboard[3] (Fig. 5) supportsVME access up to A64/D32, and contains three bridgedPCI busses. Six PC-MIP[4] mezzanine sites are arrangedin groups of three on two 33MHz 32-bit PCI busses. A third33MHz 64-bit PCI bus is bridged to the VME bus using aTundra Universe II VME-to-PCI bridge.5VTPC−MIPPC−MIPPC−MIPPC−MIPPC−MIPPC−MIP33MHzPCI bus 133MHzPCI bus 2PCI bus 3BridgeBridgeControlLocalJTAGMMTM/T33MHz 64 bitTTTTTTUniverse IIVME−PCIVME64xPMC(Standard)PMC (triple)Figure 5: VME 9U MotherboardA single large logic mezzanine board has access to allthree PCI busses for high-speed application-specific pro-cessing, and an additional standard PMC site is available.A local control FPGA on the motherboard provides accessto on-board flash configuration memory, a programmablemulti-frequency clock generator, and JTAG.The LVDS link receiver boards[5] (Fig. 6 use ChannelLink[6] technology from National Semiconductor. Eachboard contains three independent link receivers which canoperate at 20–66MHz (16-bit words). Buffering for 128K32-bit words is provided for each link with provision todiscard data if buffer occupancy exceeds a programmablethreshold. Event building, protocol checking, event num-ber checking and bit error correction are performed in-dependently for each link. A PCI target interface pro-vides single-word and burst access to the data stream, plusnumerous monitoring registers. A single PCI burst readserves to build an event from fragments found in each ofthe three input buffers. The expected event number (loweight bits) is provided as part of the PCI address, and amis-match causes an error bit to be set in the link trailer.AlteraACEX 1K100FPGA256k x 36SynchronousZBT SRAMPCI33MHz32 bitLVDS RxDS90CR285ALVDS RxDS90CR285ALVDS RxDS90CR285AFigure 6: PC-MIP 3-Channel Link Receiver BoardThe logic mezzanine board (Fig. 8) contains the coreDerand.BufferDerand.BufferBufferHTRL1A 40MHzPCI32/33EventBuilder SLINK256kb FIFO>500 eventsLSCDerand.DerandomizerBuffer Protectedagainst overflowby trigger rules logic:  no bottlenecksame as HTR processingLVDS link speedLinkTxLinkTxLinkTxLinkRxLinkRxLinkRxDCCPCI busses(2) 33MHz 32 bit>4000 events8Mb Buffer~100MHzProcessingBusy/Ready Level 2 Datato Readout UnitOverflowWarningFigure 7: HCAL DAQ Bufferingdata concentrator logic. The prototype was implementedusing a Xilinx XC2V1000 for the logic, plus three AlteraEP1K30 for three PCI bus interfaces.The event builder logic merges two data streams fromthe two PCI busses, and re-orders the incoming data sothat the various sub-types (Level 1, Level 2...) are in con-tiguous blocks in the output stream. An on-board TTCrxstores level 1 accepts (L1A) into a FIFO which drives theevent builder. For each L1A, the data decoder triggers aPCI burst read on the PCI-1 and PCI-2 interfaces simul-taneously. As data is transferred it is sorted into varioussub-types and summary and monitoring information is col-lected. Each sub-type is pushed into a unique FIFO. Afterthe end of the event has been processed (block trailer re-ceived from LRB) an end-of-event marker is pushed intoeach of the FIFOs. The event builder reads data from eachof the sub-type FIFOs in turn, inserting protocol words asneeded. The DCC logic is designed to operate continu-ously at the full speed of the two input PCI busses, namely33MHz d 32 bits d 2. The event builder and output logic mustthus run at an average rate of at least 66MHz (32-bit words)or 264MBytes/sec.The event builder output is sent in parallel to several des-tinations. Each output path contains a filter which can beprogrammed to select specific portions of events or a spe-cific subset of events (prescaled, specially marked, etc.).1. The DAQ Output. Every event is sent via SLINK-64to the CMS DAQ. The detailed contents of each eventmay be controlled by configuration registers.2. The Trigger Data Output. The trigger primitives sentto the CMS L1 trigger are also sent to via SLINK-64to a special “trigger DAQ” system for monitoring ofthe trigger performance.3. The Spy Output. A selected subset of events is sent toa VME-accessible memory for monitoring and diag-nostics.EventBuilderTTCRxDataDecoderLevel 2FIFOLevel 1FIFOSummaryFIFOPCI 1DataDecoderLevel 2FIFOLevel 1FIFOSummaryFIFOPCI 2DAQFIFOTriggerFIFOSpyFIFOMonitoring (via VME to CPU)L1AFIFOFigure 8: DCC Logic PMCError detection and recovery are a primary considera-tion in a large synchronous system and the DCC containslogic dedicated to this purpose. Figure 7 shows the mainDAQ data pipeline and buffering in the HCAL readout sys-tem. Hamming error correction is used for the LVDS linksbetween the HTR and DCC. All single-bit errors are cor-rected and all double-bit errors are detected by this tech-nique. Event synchronization is checked by means of anevent number in the header and trailer of each event, whichare checked by the LRB logic against the TTC event num-ber. Buffer overflow is avoided by the expedient of dis-carding the data payload and retaining only header andtrailer words when the LRB buffer occupancy exceeds aprogrammable level. Additionally, an “overflow warning”output is provided which is delivered to the CMS triggerthrottling system to request a reduction in the rate of L1A.Data transfers from the LRB to DCC logic are protected byparity checks on the PCI busses. The event builder operatesat a processing speed sufficient to handle 100% occupancyof the two PCI busses. After the event builder is a largememory, which can contain several thousand average-sizeevent.The main bottleneck (speed limitation) in the DCC isthe two 32/33 PCI busses through which all data mustflow. The theoretical maximum bandwidth for one of thesebusses is 33MHz x 4 or 132 Mbytes/s per bus. In prac-tice we expect to achieve about 100Mbytes/s, for a total of200Mbytes/s throughput on the two busses. This is exactlythe maximum average data volume permitted on one inputof the CMS DAQ switch.CLK& LHC EmulatorTTC (fiber)S-LinkHRCDCC FEEHTR HTRS-LinkTTCrx TTCrxTTCrxTTCvxVME BusData (G-Link)LVDSrx LVDStx LVDStxTTCtxTTCviL1ACh A Ch ACh B Ch BFigure 9: HCAL Readout Demonstrator4 PROTOTYPE TESTINGA “demonstrator” (first prototype) of the entire systemis being built (see Figure 9). The HTR demonstrator isa 6U VME module with 4 G-Link receivers running at800Mbyte/s and an Altera APEX family FPGA for the pro-cessing logic. The (second) prototype and production HTRmodules will be 9Ux400mm VME modules using CERNGOL links. The DCC demonstrator is built on the 9UVME motherboard as described above, and is quite closein hardware configuration to the anticipated production de-sign. A custom front-end emulator (FEE) which simulatesLHC timing and produces dummy front-end data is usedto provide simulated input data to the HTR for testing. AG-Link based optical S-Link is used to transport data fromthe DCC demonstrator to a VME CPU for verification.As of this writing, a simplified demonstrator using oneFEE, one HTR, one DCC and S-Link to CPU has been suc-cessfully tested for use in a high-rate radioactive source testat Fermilab. Data was transferred through the entire chainwithout error at a continuous rate of 80 Mbytes/s. The S-Link data is received on the CPU in a large DMA buffer(400+ Mb) and when full written to disk for off-line analy-sis.We expect to complete the full demonstrator shortly,though only highly simplified FPGA code will be imple-mented in the HTR and DCC.5 SUMMARYA demonstrator of the CMS HCAL DAQ has been assem-bled and testing has begun. The data concentrator makesextensive use of standard interfaces and busses, and was as-sembled from “multifunction” components developed sep-arately. This resulted in significant savings by sharing de-velopment costs between multiple projects. The design ofthe full-fuction prototypes will continue through the re-mainder of 2001, with a working prototype system ex-pected in 2002.6 REFERENCES[1] “The S-LINK 64 bit extension specification: S-LINK64”,A. Racz, R. McLaren, E. van der Bij, EP Division, CERN.see http://hsi.web.cern.ch/HSI/s-link/[2] See http://cmsdoc.cern.ch/carlos/SLB/.[3] See http://ohm.bu.edu/%7Ehazen/my d0/mb9u/[4] “PC*MIP Specification”, VITA 29. VMEbus InternationalTrade Association Standards Organization[5] See http://ohm.bu.edu/%7Ehazen/my d0/TxRx/[6] The National Semiconductor family of LVDS point-to-pointserial links. See for example the transmitter data sheet at:http://www.national.com/pf/DS/DS90CR285.html

The CMS HCAL Data Concentrator: A Modular, Standards-Based Implementation

Abstract

Similar works

Full text

Available Versions

CERN Document Server