Bal, F

Jost, B

Müller, H

Toledo, J F

CERN Document Server

Readout Unit for the LHCb experimentJose F. Toledo,  Politechnical University of ValenciaHans Muller, Francois Bal, Beat Jost, CERN, Geneva, SwitzerlandAbstractThe Readout Unit (RU) for the LHCb data acquisitionand trigger system is a programmable eventbuilderinterface, taking input from up to four Gbit/s data linkreceivers, and outputting to single PCI nodes of a readoutnetwork. It has been conceived as a versatile andprogrammable prototype module that uses the CERN S-Link as link-receiver protocol [1], 9U- IEEE960 powerand mechanics, and PCI bus protocols for the readoutnetwork. The FPGA-based receiver stage mergesincoming event-fragments into subevent blocks that arestored in an intermediate subevent buffer. Subevents areforwarded by an FPGA-based eventbuilder interface logicthat is directly connected via the PCI bus with theprotocols of the readout network. An embedded I/Oprocessor provides remote control over all input andoutput functionalities, via a 100Mbit/s LAN connection tothe experiment control system.1.  PURPOSE OF THE READOUT UNITThe Readout Unit (RU) is being developed to fulfil thefunctionality of the first stage of the LHCb dataacquisition system [2] [ Figure 1].Figure 1:   LHCb DAQ systemThe overview below [Figure 2] shows it's top levelarchitecture between S-Link receiver inputs and PCIoutputs. The role of the merger is to collect event-fragments from the input  FiFos and store them in thesubevent buffer. The subevent builder assembles theseinto subevent blocks. For complete event building,subevents are transferred via the subevent builderinterface to a  PCI bus  compliant readout  network.The RU is also equipped with an embedded CPUwhich is  connected via  a LAN to the remote control andmonitoring  system.Figure 2:  architecture of Readout Unit1.1 Bandwidth  requirements for LHCbThe nominal requirements for a Readout Unit are givenby the LHCb level-1 trigger rate and event sizes per inputlink of the RU. The detector with the highest occupancywill generate max. 1 kByte per event including someformatting overhead.  With four input  links and a 40 kHzlevel-1 rate, this corresponds to a 160 Mbyte/s sustainedthroughput requirement for the RU. The requirements forthe individual parts of the RU are summarised in [Table1] below. Each link input requires 40 Mbyte/s inputbandwidth,  the output bandwidth  is chosen  264 Mbyte/sby using 64-bit 33-MHz PCI (528 Mbytes/s for 66 MHz).PCI output 64bit@33MHz ( 66 MHz)Subevent Buffer 1 MbytePeak input per link 40 Mbyte/sIntegral   BW 160 Mbyte/s      Table 1:  Bandwidth  Requirements2.  THE RU MODULEThe Readout Unit is a 9U motherboard  with modularI/O connections for  data  and control  [Figure 3]. Theinput stage receives data fragments from front-end linksvia four S-Link receivers. Input FiFos are used toderandomize input data and  to convert the output  to a64-bit  format. The subevent merger (SEM) logic scansthe FiFos, which share a 64 bit very high bandwidth bus,and  writes their data to the subevent buffer (SEB),together with a directory entry. The output stage is theeventbuilder interface (EBI) which combines the datafrom  the SEB into subevents which are then transmittedFiF oM ergerSubev en tbufferSubeventbui ld erIn terface PCImerg er busS link  Rx S linkoutCPUL AN to DCSO v er v iew  R ea d o u t U nitout6 464( M ux  appl ications o nly)3 2P CI (aux)   L HC b D A Q  syste mTrigge r Level 2 & 3E vent  FilterRea dout  Network (RN)StorageRU RUSFCControl & MonitoringSFCCPUCPUCPURUCPUCPUCPU2-4 GB/s4 GB/s20  MB/sVa riable latencyL2 ~10  msL3 ~200 msLANSub-Fa rm Controllers  (SFC)Readou t  un its  (RU)Timin g&Fas tControlFron t-End  Ele ctronicsVDET    TRACK     ECAL    HCAL     MUON     RICHLHCb Dete cto rL0L1Level 0TriggerLevel 1Trigger40 MHz1 MHz40 kHzFixed latency 3 .2 msVaria ble la te ncy <256 msDatarates40 TB/s1 TB/sFront-End Multiplexers  (FEM)1 MHzFront- End LinksLevel 1  buffers th isp ro jec tX on /offD A Qvia  the PCI bus to the readout network (RN). Themonitoring and control unit (MCU) provides remotecontrol and access to data via a LAN connection. Anauxiliary PCI slot allows for extended applications of  theRU, such as the use of a Timing and Controls (TTC)receiver card [3].Figure 3:  Units and connections of Readout Unit module2.1  The subevent merger (SEM)The SEM logic scans the  64-bit-wide FiFo outputs in around-robin fashion and transfers event data blocks to thedual-port subevent buffer (SEB). After writing a block tothe SEB, a pointer to its location is added to the directorytable in the SEB.As shown dotted in [Figure 3] there is a multiplexeroption to transmit the input data unchanged directly to asingle S-link transmitter output. In normal operationhowever subevents are built from the stored event-data,which are then sent as one block either to the PCI, or theSlink output.2.2 Subevent buffer (SEB)The SEB is physically a dual-port memory between theSEM and the EBI. Most of its capacity1 is used for storingdata, a small part is used a pointer directory. In order tosupport LHCb’s “phased readout protocol” the SEB mustbe accessible like memory via the PCI network interfacefor selective readout. For the LHCb “full readoutprotocol”, all events residing in the SEB are transferred tothe readout network, using a fully event-driven data flowconcept.2.2  Eventbuilder Interface (EBI) The EBI logic is the RU’s output stage. It generatessubevents from the data in the SEB by identifying allblocks which belong to the same event number. Suchidentified subevents are framed and reformatted as singleoutput entities called subevents. The EBI is also part ofthe full eventbuilding system of the DAQ system andtherefore transfers subevents according to a chosenreadout protocol and network technology. For thisprototype, the PCI standard industry bus is used asconvenient and well-supported network interface betweenthe RU's FPGA logic and the network. The full or phased                                                          1 1 Mbyte  is the nominal SEB capacity for the  RU prototypeLHCb eventbuilding protocols can be implemented usingPCI either for message passing or for memory-likeaccess.2.3  The Monitoring and Control Unit (MCU)A commercial plug-in processor card [4] is used forremote monitoring and control via a LAN connection tothe Experiment Control System (ECS). Tasks of the MCUinclude monitoring the SEB buffer status, error handlingand reporting and remote control of the operatingparameters of the RU. Apart from this, the MCU can alsoinitialise the PCI bus, re-load FPGA configurationbitmaps, access status registers inside the FPGAs,implement high-level data transport protocols, or emulateRU functions before implementing them in FPGAhardware. The MCU firmware will be developed using across-compiler (for Intel i960) on host PCs with calls tolow level monitor services.2.4 The Front-end Link Receivers (FLR)For the RU prototype we opted for the use of the 32 bit S-Link protocols of the ATLAS experiment at the linkreceiver side. The use of S-link mezzanine cards allowsfor a free choice of either using short (copper) or long(fiber) technologies for the input links. At the S-linkconnector the same 32 bit framing protocol is availablefor all link technologies.2.5  Network Interface (NI) This is a network-specific PCI card which adapts a givenreadout network technology to the PCI output bus of theRU.Since standard PCI cards are mechanically not compatiblewith 9U crate mechanics, a final version of the RUrequires that this card is either available as PMCmezzanine or that its logic can be placed on the RUmotherboard. An example of a PMC network interfacecan be found in [5]. For tests with standard PCI cards, asimple PCI-PMC adapter card may be used.3.  READOUT PROTOCOLSThe transfer of data between the RU's and the 2/3rdlevel CPU farm computers requires a transfer protocolbetween the RU’s EBI logic and the 2nd/3rd level farmCPU's [6].The full readout protocol of LHCb is meant to besimple: a continuous, write-only data transfer ismaintained from source to destinations. The phasedreadout protocol only transfers subevents which havebeen pre-selected by the 2nd level decision. Thisdrastically reduces the amount of data to be transferredover the network, however its implementation, i.e.selective readout, is complex. Both readout protocols areto be studied with the Readout Unit prototypes.  i n p u t  s u b -e ve n t e ve n t- N etw o r k  F E M   L A N /E C S m o n ito r in g  &  co n tr o l  u n it         SE MR Nm e rg in gs u b e ve n t b u f ferP C I- 64b u ild erS E Bi n ter fa ce  S - l ink l i n ksEB IF R XS -l inkF EL i n ter fa ceN IA U XP C I -32(T T C )( FP G A )(F P G A ) (D P M )M C U( e m b e dd e d C P U )R J 45I N p u t sta g e (S E M ) O U T p u t sta g e (E B I )  X o n/o f f    N IM    N IM  R e s e t ( M u x  a pp licatio n s o n ly )3.1 Full readoutUnder the full readout protocol, the EBI output logicautonomously writes subevents via the PCI bus and thereadout network (RN) to a destination CPU. Thetransmission via the RN can be either a memory mappedPCI-to-PCI transfer between sources and destinations(transparent), or a DMA between the SEB and adestination buffer (buffered I/O). The choice dependsalso on the possibilities of the RN.Subevents are chained event-blocks in the SEB, withpointers stored in the SEB directory.  Due to the use ofcircular buffers there is no need to implement a garbagecollection mechanism. Buffer overflow in the SEB isprohibited via a Xon/Xoff return signal, generated by theinput stage of the SEB logic.3.2  Phased readoutUnder this protocol, only small subsets of events aretransmitted for making a 2nd level decision whilst the restof the event is queued in the SEB. Since the remainder ofthe data is only transferred after a positive level-2decision, the bandwidth across the RN network can beconsiderably reduced: non-accepted events are discardedfrom the SEB. For this protocol, the EBI logic needs tounderstand a set of messages like selective accept andreject from the RN.4.  SUBEVENT BUILDINGSubevents are built by the EBI output logic from theevent-fragments stored in the subevent buffer. The event-data fragments stored in the SEB are identified by theirevent numbers which is used by the EBI output stage tobuild and frame subevents. The process of event buildingfrom a buffer is very tolerant for arrival spreads andallows in particular  receiving several fragments of thesame ID via the same link.Figure 4: Link input format convention4.1  Input Data formatsA very simple input format convention is adopted forthe Readout Unit prototype [Figure 4]. The framingoverhead is minimal and consists of two 32-bit words inthe header and one in the trailor. The start and end of theinput block is identified via a hardware flag bit. In orderto ease subevent building, the event identifier is containedin the header word. The block size and the error status are contained in thetrailor word. Four LSB bits are used by S-link.3.2  Subevent framingA recursive and simple way of generating subeventblocks has been adopted. The data blocks of event framesbelonging to the saem event number are concatenatedbehind the same  header word as the input frames [Figure5]. This requires that data blocks have their ownconsistent format which is transparent to the RU. The sizeand status fields for subevents are recalculated andappended as trailor word. An error block is insertedbetween the last data block and the trailor only in case oferrors. The latter is indicated in the status word.Figure 5: Recursive subevent framing of input events5. TECHNICAL IMPLEMENTATIONThe RU module 's technical layout is shown is shownbelow [Figure 6]. Figure 6: RU technical implementation5.1 Internal bus connectionsAt the input side, the 32 bit S-Link words areconverted via interleaved 32 bit FiFos to a 64 bit SEB busDA Q Event NumberSource Id(24)Size(16)32-bit wordsDATA BLOCKtype(4) (  run. monitoring, errorblock, etc.. )Stat(8)Flag bit4 bits:Used by S-Link hardware for error report4 bits: reserved for futureapplicationsD a t a  f ra m e  f orm a t  for  R U  r ec eiverS31 15 0reserved(4)control   trailorheaderty peEvent Number nTypeSource Id aStatus  a32-bit wideBlock aBlock bBlock cEvent Number nData Size a + Data Size b + Data Size c +Output frame  building(always expressed in 32-bit words)Incoming frames+ {Size of Error Block}Error BlockBlock aOnly created if an error is reportedin the Status wordBlock mergingSize  aEvent Number nTypeSource Id bStatus bBlock aSize  bEvent Number nTypeSource Id cStatus  cBlock aSize cStatus X Size XTypeSource Id XInitially all zeros!Initially all zeros!Block bBlock cBlock a(Erro r B lo ck)R ead o u tN e tw o rkIn te rfa ceFi FoFi FoFi FoFi FoS E MS E BEBIP C I -6 4  323232326464646464S-LinkS-LinkRx L A NReadout Networkfas t  in te rleav ing  log icS u b ev e n t M U X  o u tp u t  (d u a l  p o rt  m em o r y )F P G A F P G A(m e z za n in e )(m e z za n in e )S E M  L O G ICE B I L O G ICM C UM o n it o r in g  &  c o n tro l l o g ic A U XF PG A P C I-3 2  8  bi t   i9 6 0  c o n tro l  b u s32X o n /X o ffPC I83 26 4S E M  b u s6 4 b i t@ 5 0  M H z(m e z za ni ne s )(TTC) D P M  b u s e s  A  +BB u ffe rM C Ub uswhich is read out by the SEM FPGA with a clock rate of50 MHz to provide a raw bandwidth of 400 Mbyte/s.Input data are stored in a 64-bit-wide dual port memoryinterfaced between the SEM input and the EBI outputstage. The latter consists of two parallel FPGAs (EBI andAUX) which share the 64-bit SEB bus at their inputs andthe 64-bit PCI bus at their outputs. The reason for theparallel use of two FPGA's, apart from lower cost of theFPGAs is the increased PCI bandwidth which can beachieved with two tandem PCI masters2. The MCU isconnected to the 64-bit PCI bus and its 32-bit memorybus is connected to the EBI FPGA. The latter allows thatthe MCU can map the SEB into its local memory space.A dedicated, 8-bit-wide i960 bus interconnects all FPGAswith the MCU. This allows for FPGA configuration andaccess to control of status registers via the MCU, andfurther via a remotely connected server. The MCUprovides a secondary, 32-bit PCI port which may be usedfor auxiliary PCI cards.5.2  Motherboard layout The RU prototype is implemented as 9U motherboardwhich only takes power from an IEEE-960 cratemechanics. All 6 mezzanines for data input (4) and fordata output (2) are located on either front-or backpanel ofthe motherboard. A preview on the module (currentlyunder design) is shown below [Figure 7]Figure 7: A preview of the 9U motherboard5.3 Board design toolsThe schematic capture was completed using CadenceConcept. The 12-layer board is designed using CadenceAllegro.For critical buses design like for the 64 bit FiFo bus andthe 66-MHz PCI bus, a signal integrity study usingCadence SpecctraQuest and IBIS models from different                                                          2 In a future RU version we expect that a single FPGA can beused.vendors was carried out. The results of this study alsoinfluenced the placement of components in the PCB[Figure 8].Figure 8: Floorplanning with Cadence Allegro5.4 FPGA technologyLucent Technologies’ ORCA 3TP12 device [7] hasbeen chosen for both FPGAs (EBI and AUX) in theoutput stage. This device has a 64-bit 66-MHz embeddedmaster/target  PCI hard core interface, as well as an 8-bitembedded hard core interface to i960 processors. Thelatter interfaces directly to the MCU’s local bus andallows for remote configuration, program readback andaccess to internal FPGA. Approximately 35k equivalentlogic gates are available for the EBI logic.The SEM FPGA in the input stage (ORCA 3T55) isalso a member of Lucent Technologies’ 3T family. Thedifferences with the previous device are the available userlogic size (up to 55k equivalent gates). This device hasalso an i960 interface but no PCI core [8].5.5 FPGA development toolsLike for previous design projects we use a VHDLdesign methodology for ORCA FPGAs from LucentTechnologies. After a VHDL simulation with CadenceLEAPFROG, the VHDL output is synthesised byExemplar’s LEONARDO to generate an EDIF outputwhich is used by Lucent's Foundry tools to place androute all required FPGA resources. The bitstream outputis programmed into a Flash Eprom, which during power-up transfers its contents to all FPGAs in the Readout Unit.MCU SLINKSLINKSLINKSLINKSEB5V->3.3VPMC 64 bitPMC 32 bit orSlinkSEBEIB-AUXRJ45IEEE960connectorAux. I/O areaLANFPGA FPGAFPGAresetFront panel mezzanineFlashClock64 bit SEB busXon/Off6. STATUS AND OUTLOOKThe Readout Unit project is a joint activity of theCERN EP-ED electronics group and the LHCbexperiment. A series of meetings [9] have led to thedesign decisions for the first prototypes to be built in1999. Since technology choices for input links, readoutnetwork and control system are pending, we have optedfor a modular approach using S-Link mezzanines for thelink input, PCI for the readout network and T-base 100LAN technology for the control system.This module is meant to be reproduced in smallquantity and serves mainly the purpose for developing thefinal RU concept. A second-generation module will bestarted when technology and protocol choices have beentaken. REFERENCES1. For information on S-Link, see  the Slink homepagehttp://www.cern.ch/HSI/s-link/2. LHCb collaboration, LHCb Technical Proposal,CERN/LHCC 98-4, Chapter 133. Estudio y Desarrollo de Nuevos Sistemas deTemporizacion, Trigger y Control para los FuturosDetectores de LHC del CERN,  doctoral thesis on theTTCsr mezzanine  Jorge Ferrer, CERN /Politechnical University of Valencia (to appear inEnglish )4. PXECORE embedded CPU mezzanine module fromCompuLab. See http://www.compulab.co.il5. Design of a high performance PCI interface for anSCI network, COMPUTING & CONTROLengineering journal, December 1998, F.Mora et al.6. The LHCb Trigger and Data Acquisition System,presented at the IEEE Real Time Conference June14-18, 1999 Santa Fe, NM RT 99, Santa Fe, 15 June1999.   B.Jost et al.7. Lucent Technologies’ ORCA 3PT12 data sheet.http://www.lucent.com/micro/fpga/hardware.html.8. Lucent Technologies’ ORCA 3C/T FPGA data sheet.http://www.lucent.com/micro/fpga/hardware.html.9. For history and design documents, see the ReadoutUnit home page.http://nicewww.cern.ch/~hmuller/lhcb.htm

Readout unit for the LHCb experiment

Abstract

Similar works

Full text

Available Versions

CERN Document Server