Antchev, G H

Cano, E

Chatellier, S

Cittolin, Sergio

Erhan, S

Gigi, D

Gutleber, J

Jacobs, C

Meijers, F

Nicolau, R

Orsini, L

Pollet, Lucien

Rácz, A

Samyn, D

Sinanis, N

Sphicas, Paris

CERN Document Server

AbstractIn the context of large data acquisition systems (DAQ)for LHC experiments, flexible and uniform interfacesbetween subsystems is a key feature to ensure best pos-sible system integration/maintainability and a clear sys-tem upgrade policy. This approach is widely used inCMS at the border between sub-detectors front-end andthe DAQ. Common requirements and reconfigurablehardware used in CMS DAQ are presented in this arti-cle.1.  IntroductionA basic block diagram of the CMS DAQ is shown fig-ure.1. Its constitutive elements are the detector front-end, the trigger logic, the readout units (RU), the eventbuilder, the event filter units (EFU), the computing ser-vices, and the event manager. Fig. 1. CMS DAQ block diagramThe front-end electronic acquires data every bunchcrossing and the trigger logic selects the data to be sentto the readout units. A readout unit is constituted of 1DPM and several Detector Dependent Units (DDU). ADDU is controlling the associated front-end electronicand receives the trigger selected data. As a DDU isreceiving data on several data links, a data packet isbuilt locally (DDU event). The DDU event is kept avail-able for the DPM. As a DPM is reading out severalDDUs, a DPM event is also built. Then the DPM event(also called Event Fragment) is sent through the eventbuilder to the requesting EFU. The association of theRU with an EFU is the task of the event manager.2.  Interfaces and standardizationThe DAQ is the natural convergence point of the dataproduced by the sub-detectors. In the case of CMS,there will be about 12 different data sources providingdata to the DAQ. Interfacing these sources with theDAQ is a critical point given the overall size and com-plexity of the final system (on-detector electronic,counting room electronic and DAQ). Reducing thediversity in the electronic devices is a valuable approachregarding the system integration (especially during theinitial debug phase) and later the maintenance/upgradeoperations.Looking at hardware interfaces between heterogeneoussystems, one can find a generic architecture oftenencountered in DAQ applications (see figure 2).Fig. 2. Generic interface architectureAs soon as a "popular" hardware has to drive (or bedriven by) a specialized one, this generic architecture isused (with some variations).The DDU, part of the Readout Unit (see figure 3), canbe seen as the interface between the DAQ and the sub-detector readout systems. Below the DDU, no sub-detector specific hardware is foreseen in the DAQ archi-tecture. The task of the DDU is to deal with a given sub-detector with its specificities on one side and makeavailable the data on the Front End Bus (FEB) accord-ing to its specification/protocol on the other side. Moreinformations about the DDU functionnalities are avail-able in [1].The principle of a common DDU for all sub-detectors inCMS is proposed for a long time now. A common func-tional specification document [1] is adopted by all CMSdata producers. The next step is to study the feasibilityThrottleTriggerLevel 1ManagerEventTriggerDetector  FrontendComputing ServicesReadout UnitsEvent BuilderEvent Filter UnitsSwitch FabricStorage Device On-the-fly Processingstandard Interface to a mass-market Custom I/OsInterface Fast message Generic hardware for DAQ applicationsG. Antchev, E. Cano, S. Chatellier, S. Cittolin, S. Erhan, D. Gigi, J. Gutleber, C. Jacobs, F. Meijers, R. Nicolau, L. Orsini, L. Pollet, A. Racz, D. Samyn, N. Sinanis, P. SphicasCERN, Div. EP, Meyrin CH-1211 Geneva 23 Switzerlandof a common hardware platform that can suit the sub-detectors needs. Architecturally speaking, the part of theDDU in touch with the FEB can be common for everysub-detectors both at the functional and physical level.Regarding the sub-detectors, two families can be distin-guished: analog detectors and digital detectors. Con-cerning the analog detectors (Pixel, Silicon andMSGCs), a unique hardware development is underwayand made by the Rutherford Appleton Laboratories [2].The standardization is eased by the fact that the detec-tors are using the same data link and/or Front End chip[3]. About the digital detectors, the situation is not aseasy due to the fundamental differences between detec-tors. However, the CMS DAQ group designed a versa-tile board that has been first used for specific DAQapplications. But during the design phase, special carehave been taken to allow other applications outside theDAQ area to be implemented: especially DDU-typeapplications were considered.In the next parts of the article, the generic block diagramof a DDU will be shown and compared to the block dia-gram of the versatile DAQ board. Then a quick over-view of the different applications running on that boardwill be listed and finally, the next generation of versatileboard, currently in the assembly/debug phase, will bepresented.Fig. 3. Readout Unit Structure3.  Generic DDUAs reminded before, the DDU is in charge of interfacinga sub-detector environment with the central DAQ envi-ronment. One part of the DDU deals with the sub-detec-tor specificities and the other part deals with the so-called FEB. This later part must be common for all sub-detectors on the functional level and can be common onthe physical level. Architectural/functional flexibility isa major feature of the common part as the DAQ will fol-low the technological evolution of the telecommunica-tion industry during its life time. Therefore,reconfigurable logic (FPGAs) and a layered approachare used wherever it is possible.Generically speaking, the DDU must acquire the data ofthe FES, control/manage the FES and make availablethe data on the FEB after a specific processing (if any).The data encapsulation format and the communicationprotocol with the DAQ will be unique through CMS.The DDU generic block diagram is shown Fig.3. Fig. 4. Generic DDU block diagram4.  Generic DAQ platformA DAQ hardware platform has been designed anddeveloped last year in order to provide a prototypingfacility for testing other DAQ boards (mainly the DualPort Memory [4]) and evaluate architectural optionswithin the CMS DAQ project. 4.1 DescriptionThe architecture of the board features total reconfig-urability thanks to the on-board FPGA (50Kgate-equiv-alent) 100% available to user-application. The FPGA[5] is linked on one hand to a commercial PCI interface[6] (initiator and target) and on the other hand to a 77pins user port through bi-directional TTL drivers.Thedrivers are fully controlled from the FPGA allowingcustom I/Os to be implemented. The FPGA has alsoaccess to 32 KBytes of nvRAM. Finally, the sameaccess port to the TTC signals defined by RAL has beenimplemented on the board.Switch Data LinkFast Data Link or BridgeReadout Unit OutputFrom EVM  Detector Dependent Units Detector Data Links From EVM RUM RUSReadout UnitFront End Driver Readout Unit InputFMULong Distance (~200m) Dual Port Memory FED  BUS RUI  BUS RUO  BUS Data LinkDual Port Protocol FPGA RAM TTC Port FPGADetectorPort Detector Processing FEB Interface Chip Fig. 5. DAQ generic platform block diagramThis architecture allowed numerous applications to bedeveloped. Detailed description and engineering docu-ments are freely available on the WEB [7] to anyoneinterested by such hardware.4.2   Applications• DPM exerciserIn the previous version of the DPM named P25++,the data input port was a custom development andthe output port was a standard PCI. The genericboard has been programmed to readout datathrough the output port (the generic board was initi-ator of the DMA) and to reinject these data into theinput port. The data were cycling at 128 MB/sec.sustained (97% of the maximum possible perfor-mances of PCI 32bit/33MHz)• DDU emulatorAlways with the P25++, the generic board has beenconfigured to emulate a DDU. Under the control ofthe PCI bus, the FPGA was translating on the fly aPCI data stream (DMA) to the input format of theDPM. The throughput was again close to the maxi-mum PCI speed.• FERA to PCI interface [8]The generic board is in charge of transfering thedata on the FERA bus (Fast Encoding and ReadoutADC developed by Lecroy Corp.) to the PCI bus bymeans of DMA. The PCI destination address iscontrolled via a register in the FPGA.• Trigger interface for CMS Muon chamber [9]For test beam applications, the board is used tointerface the chamber readout system on PCI withthe beam signals (spill, trigger). The busy and vetosignals are controlled by the board.• Fast Messaging network based on IEEE-1394 [10]Still under design, it is an extension of the previousapplication to several PCI based readout units. Thenetwork is Firewire at 400Mbit/sec. The triggers aretagged and broadcasted to all readout units toensure the correct event fragments to be mergedduring the event building process. For that, a transi-tion module has been developed to house theFirewire hardware: it plugs directly into the userconnector. • Myrinet protocol monitor [11]For LHC DAQ systems, one major piece of equip-ment is the event builder. The most likely imple-mentation of the event builder is based on a multi-stage switch fabric. If the behavior of an individualswitch can be measured and simulated, the behaviorof a switch fabric is much more difficult to study. Inorder to understand what is going on in the hard-ware, a protocol monitor is often needed. Thisdevice allows to measure the basic parameters ofthe device and allow to define the simulation con-stants needed to tune the software model. More-over, it helps to interpret/explain somemeasurements when the architecture and the testsetup is getting complex, and it is the case! ForCMS, one candidate is a switch from Myricom [11].A spy/monitor based on the generic board is underdevelopment. A pre-processor board is plugged onthe user connector to allow the interfacing with thelow-level layers of the switch protocol. The FPGAextracts the features of the messages (destination,length, status, flow-control, time tags...) and storesthem into the nvRAM. A visualization softwarethen acquires them and process them for displayand statistics.Given the comments of the different users and the tech-nology improvements, the need for hardware flexibility(especially for the user connector), more storage capac-ity and faster FPGA/PCI interface triggered the devel-opment of a new generation.5.  New generation of the DAQ platformCompared to the previous version, it features:•  PCI 66MHz, 64 bits capability• On-board 32 Mbytes SDRAM• 800MBytes/sec. internal bandwidth• 300Kgate-equivalent FPGA• 148 pins high performance port for custom hard-ware• JTAG support for BSCAN and FPGA configurationThe choice of the GT64120A from Galileo [12] as PCIinterface enables the fastest speed on PCI (524 MBytes/sec. peak) and the usage of huge, inexpensive memoryControlData PLX PCI 9080 Altera 10K50V 274 I/Os TTL DriversUser I/OsTTC Port nvRAM chips (100 MHz SDRAMs). The FPGA is a Virtex 300(the new family from Xilinx [13]) providing the equiva-lent of 300K logic gates. Fig. 6. Core board block diagramDue to BGA technology used for the two previous chip,IEEE-1149.1 (JTAG) had to be implemented for test/debug purposes. The FPGA and its configurationEEPROM are compliant to IEEE-1149.1 and integratedin the JTAG chain as well. Finally the JTAG chain isavailable on the user connector to allow the test/debugof the application specific hardware extension. Eachchip can be removed separately from the chain if theshifting time becomes "long". This feature allows alsoindividual debug if needed. The access to the JTAGchain is either local from an on-board tap or from thePMC connectors, hence allowing remote test/debug/reconfiguration.To increase the flexibility at the hardware level, insteadof providing a connector with fixed drivers and termina-tions (solution adopted in the first generation), all theremaining free pins of the FPGA are available on highperformance/high density connectors (Matched Imped-ance Connector family from AMP [14]) where a spe-cific extension board can be plugged. Hence, theplatform can match exactly the requirements of the spe-cific application by developing only an extension mod-ule.The form factor of the core board is PMC. With itsextension, the PMC specs. are violated but still allowsthe operation in a standard PMC slot. Mictor edge toedge connectors are available and would allow to matchthe PMC form factor but the cost and minimum quanti-ties of such connectors are dissuasive for prototyping!Again, the RAL TTC access port has been implementedbut now, the FPGA has the capability to drive these linesif the application requires it: bidirectional LVDS driversfully wired to the FPGA have been implemented on-board. This feature opens a full range of new applica-tions that were not possible with the first generation ofgeneric boards. As with the first generation, engineeringfiles along with component specifications are freelyavailable at [7]. When the board will be fully operation-nal, access to CAD files under Cadence will be alsogiven.5.1 Foreseen and potential applications• The first application to be ported on that genericboard will be a 200m-400MByte/sec. data link.Such performances are required to implement theso-called "DAQ link" in CMS (this link goes fromthe DDUs located in the underground countingrooms up to the RUM located in surface buildingsalong with the switch and the filter farm).• Protocol analyser/monitorA faster and more powerful Myrinet analyser maybe ported on the new platform. In the near future,Myrinet will more than double the speed of theirlink and the first board will be saturated immedi-ately with the high speed traffic.• Level 1 Accept generator/Control networksAs the DAQ test setup evolves towards the finalarchitecture, trigger generation devices and fastcontrol networks must be developed in order toexercise the setup under more realistic conditions.The versatility of the new generation enables thesedevelopments.• Front-End data fan-in/fan-outFor low/high occupancy detectors, data may need tobe multiplexed/demultiplexed before/after the FEB.This is needed to use the FEB bandwidth in an effi-cient way or reduce the load to a small part of theswitch: this can be done by such board, re-design-ing only the extension.• Finally, the board can be the core of a unique digitalDDU (to reach the same situation as with analogdetectors in CMS) or a general prototyping plat-form to get familiar with high speed PCI, to imple-ment exotic interfaces to PCI... Whatever you canimagine!6.  SummaryIn a first part, steering principles for sub-detector/DAQcommunication are presented. In order to ease systemintegration, the concept of a uniform interface for allsub-detectors has been adopted for CMS. A set of mini-mum requirements has been established and a commonimplementation for these minimum requirements is pro-posed. This standardized approach will ease the integra-tion, the debug, the maintenance and the upgrade of the148 linesJTAGPCI BusJTAGDataJTAG66MHz, 64 bitControlJTAGCPU BusEEPROM TTC  Galileo GT-64120A SDRAM Port  UserConnectorXilinx Virtex XC300 system with minimal resources consumption (financialand human).In a second part, reconfigurable hardware platformsdeveloped by the CMS DAQ group are presented.Thanks to the on-board programable logic, these plat-forms are used by the sub-detectors but also by othergroups to develop their specific applications. An over-view of these applications is given.Finally, the latest generation of the platform is pre-sented. The versatility of the platform has beenincreased by using extension boards where applicationspecific hardware can be added. A 200m-400MB/s datalink based on this platform will be the first practicalapplication.7. References                                                                                                                                [1] CMS DDU design specificationsCMS Note 1999/10A. RACZ CERN - EP/CMD[2] CMS Tracker FED-PMC Pagehttp://hepnts1.rl.ac.uk/CMS_fed/Default.htm[3] APV6 User Manualftp://ftp.te.rl.ac.uk/apv6/user_manual/apv6_user_manual_2.0.ps[4] CMS Dual Port Memorysee LEB99 ProceedingsD. GIGI CERN - EP/CMD[5] Altera WEB pageshttp://www.altera.com[6] PLX WEB pageshttp://www.plxtech.com[7] DDU Designers pageshttp://cmsdoc.cern.ch/~racz/DDU/[8] A versatile F.E.R.A to PCI interface for the GASPexperiment - M. Bellato, R. Ponchia - INFN internalreport[9] A reconfigurable trigger interface for small DAQsystems - M. Bellato - INFN internal report[10]A readout control network for small DAQ systemsbased on IEEE-1394 standard - M. Bellato, R. Pon-chia - INFN internal report[11]Myricom WEB pageshttp://www.myri.com[12]Galileo Technology WEB pageshttp://www.galileot.com[13]Xilinx WEB pageshttp://www.xilinx.com[14]AMP MICTOR Familyhttp://www.amp.com, then do a search on "mictor"!

Generic hardware for DAQ applications

Generic hardware for DAQ applications

Abstract

Similar works

Full text

Available Versions

CERN Document Server