Bohm, C

Engström, M

Hellman, S

Jakobs, K

Quast, G

Schäfer, U

Silverstein, S

Thomas, J

CERN Document Server

Use of Programmable Logic in a Pipelined Trigger for ATLASK. Jakobs, G. Quast, U. Schäfer, J. Thomas, University of Mainz, GermanyC. Bohm, M. Engström, S. Hellman, S. Silverstein, Stockholm University, SwedenAbstract  We present an overview of the Jet and Energy Sumprocessor for the ATLAS Level-1 Trigger. The designsystem is based on the use of commercial off-the-shelfcomponents, including large programmable logic devices.The architecture of the system is discussed, includingimplementation details that reduce latency and boardcomplexity, and take advantage of the inherent flexibilityin the design. The results of technology demonstratorprograms confirm the feasibility of key elements of theprocessor design.1.  INTRODUCTION   The programmable logic industry has seen rapiddevelopment in recent years. Currently availableprogrammable logic devices offer hundreds of thousandsof usable gates, hundreds of user-definable I/O pins,system performance well over 100 MHz, andcompatibility with a variety of I/O signal standards. Theseadvances have been accompanied by reductions in price,making in-system programmable logic a seriousalternative for applications which have previously favoredASIC solutions.   The advantages of using programmable logic are wellunderstood. Prototyping cycles take hours or days, ratherthan weeks for ASICs. Belatedly discovered errors can beeasily fixed, and system functionality can be modified orenhanced as requirements change. Programmable logicdevices are fully tested at the factory, effectivelyproviding 100% device yield. Finally, in-circuitprogrammability allows “design multiplexing”, wheredifferent functions (such as data taking and diagnostics)may be performed in the same logic, reducing the numberof gates needed.   The Jet/Energy Sum processor system for the ATLASFirst Level Calorimeter Trigger is a multi-crate triggerprocessor system designed to be built using FPGAs andother commercial off-the-shelf components. FPGAs areused to calculate transverse and missing energy in thecalorimeter, identify energy clusters for use in jetcalculations, send information for use by the Level-1 andLevel-2 triggers, and read out time slice data to DAQ fortrigger validation purposes.In this paper, we present an overview of the proposedsystem, followed by implementation features whichenhance performance, simplify the design, and exploit theflexibility of the system. Finally, we present results fromtechnology demonstrator programs that have been used tovalidate key features of the system.2. SYSTEM OVERVIEW2.1 The ATLAS Level-1 TriggerThe organization of the ATLAS Level-1 trigger ispresented in Figure 1.  Data from the calorimeters andmuon detectors are transmitted to respective triggerprocessors, which identify and count electron, tau and Jetcandidates, calculate total and missing transverse energy,and reconstruct muon trajectories. The processors sendresults from every bunch crossing to the Central TriggerProcessor (CTP) which makes a Level-1 Accept (L1A)decision based on them. If the CTP issues a L1A, theprocessors read out more detailed information on thetypes and locations of event features to the Region ofInterest (ROI) Builder for use by the Level-2 Trigger.             .1x.1 (8b)          .2x.2 (9b)Figure 1: Organization of the ATLAS Level-1 Trigger   In the calorimeter trigger, analog trigger tower sums arefirst received and digitized by the CalorimeterPreprocessor system before being passed to the e/τ ClusterProcessor and the Jet/Energy Sum processor. Eight-bitelectromagnetic and hadronic trigger tower energies with0.1x0.1 η/φ granularity and the correct bunch crossing aretransmitted to the Electron/Tau cluster processor, while 9-Cal Preprocessorµe, τ Jet, ΣΕΤTrigger, Timing & Ctl.Central Trigger Proc.Front End Electronics Level-2 TriggerRegion of InterestBuilder(LVL1/LVL2)Calorimeters Muon Detectorsbit Jet elements with 0.2x0.2 granularity are sent to theJet/Energy Sum Processor. 2.2 The Jet/Energy Sum ProcessorThe Jet/Energy Sum trigger space consists of 960 Jetelements in the range –3.2 < η < 3.2, as well as extraenergy sums from the forward calorimeters. The system issubdivided into four quadrants in φ, which are eachcovered by a separate set of Jet/Energy Sum Modules(JEMs). Jet elements near the border of adjacentquadrants are duplicated in the preprocessor and sent toJEMs in both quadrants to provide the environmentalinformation needed by the Jet algorithm.   Each JEM (Figure 2) receives 9-bit Jet elementenergy sums in 25 ns intervals via 400 Mb/s serial links(National Semiconductor LVDS). The demultiplexedinput data are transmitted at 40 MHz to Energy SumFPGAs, which sum the electromagnetic and hadronicenergies. The ET, EX, and EY, of each Jet element arecalculated, and sums of these are transmitted to addertrees on a separate FPGA, which merges the results andtransmits them to the Sum Merger Module.                                                                                 Jet                                                                                 ETFigure 2: The Jet/Energy Module (JEM)  The Energy Sum FPGAs also distribute the 10-bit ETvalues of each Jet element via 80 MHz point-to-pointlinks to Jet FPGAs on the same module and toneighboring modules, providing each JEM with thenecessary environmental information to perform the Jetalgorithm.Board-level Jet and Energy Sum results are transmittedvia point-to-point links from each module to a Jet MergerModule and a Sum Merger Module in each crate. Thecrate-level results are merged and reduced to trigger bits,which are sent to the Central Trigger Processor.The input and output data of each module, as well asthe types and positions of clusters identified in the Jetalgorithm, are stored in pipelines on the Energy and JetFPGAs, to be read out to the DAQ and ROI Builder upona L1A decision.3. DESIGN FEATURES3.1  The Energy Summation AlgorithmThe Energy Sum FPGAs process input data using thealgorithm shown in Figure 3. Separate 9-bitelectromagnetic and hadronic energies for each Jetelement are received at 40 MHz. Noise-reductionthresholds are applied, after which the electromagneticand hadronic parts of the Jet element are summed. Lookuptables produce values of ET, EX, and EY for globalsummation, and the 10-bit Jet element transverse energysum is multiplexed to 5 bits at 80 MHz and fanned out tothe Jet FPGAs.                                    10                               5EM                                                              80MHz To Jet                                                                      FPGAsHad                                                           To                                                                  Summation                                                                  LogicFigure 3: The Energy Summation AlgorithmThe Energy Sum algorithm is foreseen to beimplemented on a Xilinx Virtex device [1], with typicallyfour Jet elements processed by each FPGA. An additionalVirtex device merges the results of adder trees in theEnergy FPGAs to produce ET, EX, and EY values for theentire module.Xilinx Virtex devices were chosen for the Energy Sumalgorithm implementation for a number of reasons. Virtexoffers large amounts of logic and I/O resources atrelatively low cost. Large amounts of distributed andblock memory are available for lookup tables and otherfunctions, and DLLs are available for internal andexternal clock synchronization. Maximum system speedsare far higher than the 40 and 80 MHz necessary for thealgorithms, relaxing the implementation.3.2  The Jet AlgorithmThe Jet algorithm is implemented in 64 Jet FPGAs,each of which covers a 2x8 Jet element area in ηxφ. Thepositions of potential Jet cluster candidates are identifiedby summing the energies of Jet elements in 0.4x0.4windows sliding in increments of 0.2, and identifying  LVDS  LinksReadout to DAQ and ROIBETR/OJetETETETVME/JTAGΣETThrMUXEXEYF/OΣThrETlocal maxima. These are referred to as ROI or declusterpositions. Jet clusters are identified by comparingtransverse energy sums within windows of size 0.4, 0.6, or0.8 that exceed predefined thresholds and are associatedwith an ROI position. All three Jet window sizes arecalculated in parallel, and 8 combinations of energythreshold and window size are available for trigger menus.The Jet cluster sizes (dark areas) and their associated ROIpositions (bold outlines) are shown in Figure 4. For Jetclusters of size 0.6, the ROI position can be in one of fourpositions in the cluster.                         0.4 x 0.4                         0.6 x 0.6                         0.8 x 0.8Figure 4: Jet cluster definitionsEach Jet FPGA receives 10-bit ET information from 55Jet elements providing the 5x11 environment necessary toconstruct all of the needed cluster sums. To process all550 bits of data on a single device, the information mustbe received at 80 MHz, halving the number of input pinsnecessary.The data continues to be processed at 80 MHz after ithas been received in the Jet FPGA. One reason for doingthis is to reduce the latency of the Jet algorithm, which isone of the most time-critical paths in the ATLAS Level-1trigger. The second reason for doing this is that it allowsadditions and comparisons to be performed by so-called“5-bit serial” calculations, where 10-bit operations arecarried out in two consecutive 5-bit steps. The use of 5-bitserial calculations reduces the necessary logic and routingresources by nearly a factor of two, and also raises themaximum speed of the algorithm.Large savings in logic and routing resources are alsoachieved by carefully eliminating duplicate andunnecessary calculations in the algorithm. For instance,the results of a single magnitude comparison between twoadjacent decluster positions can be used by both positionsto determine whether either is a local maximum.Similarly, only Jet windows that are actually associatedwith a local maximum need be tested against energythresholds. The baseline device for the Jet FPGA implementationis the Altera FLEX 10k250 [2]. The 10K series of FPGAshas a global routing architecture which is well suited forlarge designs with high clock rates and the large degree ofdata fanout and routing inherent in the Jet algorithm. It isplanned to investigate alternative devices, includingXilinx Virtex and Altera Apex, before producing the finalsystem.3.3 Backplane Signal TransmissionThe Jet/Energy Sum system has a full-custom 9Ubackplane. The baseline choice of card-edge andbackplane connectors are 5+2 row, 2mm HM connectorswhich are an industry standard and commonly used inCompactPCI. The crate and front panel hardware areplanned to be based on the IEEE 1101.10 standard.All Jet element data are transmitted from the EnergySum FPGAs to the Jet FPGAs via point-to-point linksusing low-voltage CMOS levels. This includes signalsbetween FPGAs on the same board as well as data sharingbetween neighboring JEMs via short backplane links.Because the CMOS levels are compatible with the JetFPGA inputs, no signal translators are necessary on thereceiving end of the links. It is also considered to bepossible to drive all 80 MHz signals directly from theEnergy Sum FPGA output pins. This strategy promotessignal uniformity within the system, and simplifies thedesign of the JEM by eliminating large numbers of driverand receiver devices.Transmission of JEM results to the Jet and Sum MergerModules is also performed via point-to-point links.Studies of appropriate signal levels and terminationschemes are currently in progress.3.3  Making Use of ReprogrammabilityIt is planned to take advantage of the in-systemreprogrammability of the FPGAs in the system.Configuration details-such as energy thresholds andlookup table contents-will be controlled by writing toregisters implemented in the FPGA designs. However,more significant changes in functionality-such asmodifications to the Jet algorithm-may be easier to makeby downloading a different FPGA design altogether. It isconceivable that several FPGA configurations may beselected for different luminosity or noise conditions, forexample.The most important use of FPGA reprogrammabilitymay be the ability to use special diagnostic configurationsto perform fast and complete testing of all signal paths inthe system. Such configurations could be routinely loadedand run during pauses in data taking, for example, duringbeam fills. They could also be used to quickly pinpoint thesource of errors discovered by trigger monitoring andvalidation programs.4.  DEMONSTRATOR PROGRAMSTechnology demonstrator systems have been built andtested to confirm the feasibility of key elements of thedesign.4.1 “Module –2” Technology DemonstratorThe JEM Module -2 was designed to study issues raisedduring the review of the Level-1 Technical Design Report[3]. These issues included the reliability of 80 MHz datatransmission between FPGAs and across backplane links,the performance of algorithms at running at 40 and 80MHz, and the reliability of high-speed serial links usingdifferent cable and connector options.Each Module –2 board includes a Xilinx 4013XL andan Altera 10K50V FPGA with point-to-point linksbetween them, and to a 2mm-pitch HM backplaneconnector which allows two such modules to be connectedby a small 2-slot backplane. Another set of direct andbackplane connections between the FPGAs pass throughan ALVC buffer chip clocked at 80 MHz. Each of the fourdata paths is 10 bits wide. The timing of each FPGA andthe ALVC buffer are independently controlled usingadjustable delays.Each board also includes a pair of HP Glink transmitterand receiver devices for link stability tests. A VMEinterface implemented on a separate Altera 10K30 FPGAis used for control and monitoring. A 40 MHz clock isprovided by an external source.Data transmission tests using the Module –2 systemshowed error-free CMOS data transmission over all paths,both with and without the use of intermediate ALVCdrivers. Upper limits on bit-error rates were set at ~10-14.The timing margin for 80 MHz data was measured to be9.6 ns with buffers, and 7.3 ns with direct transmissionbetween FPGAs. Sample “5-bit serial” calculations weretested at 80 MHz and found to work as expected.Early experience with the use of serial links wasgained, both with HP Glinks, and later with LVDS linksthat were added to the board using a small adaptor PCB.The results of these tests have been superseded byexperiences with the Mainz Link Test Board.4.2 Mainz Link Test BoardThe Mainz Link Test Board was designed to gainexperience with National Instruments LVDS serial links,and to identify and solve previously observed problemswith link stability over long cables and differential PCBtracks.The Link Test Board is a 3U card that fits in a J1 VMEbackplane slot. The board contains 8 LVDS receiverchips, which are read out by two Xilinx Virtex XCV100devices. A VME interface is implemented on a XilinxCPLD. Modified JEM Module –2 boards were used asdata sources for tests.Tests showed that an active pre-compensation circuit atthe LVDS data source improves link stability and allowsthe cable range to be extended. Stable transmission of fourlinks over 15m of category 5 twisted-pair cable plus 55cm of differential PCB tracks was achieved using a pre-compensation circuit based on the Motorola MC10EL89differential fanout gate [4]. Commercial off-the-shelf pre-compensation solutions for the final system are currentlybeing investigated.5.  OUTLOOKThe Module –2 and Link Test Board programs haveshown that the key design features of the Jet/Energy Sumprocessor are sound. Plans are currently underway for thefirst of two prototype systems planned before production.The Jet/Energy Sum “System –1” will be a fullyfunctional prototype system with reduced channel count.It is planned to be a single crate 6U system with JEMs,merger modules, and timing and VME modules connectedvia a custom backplane. It is hoped to have the systemfully operational by summer 2000.“System 0” will be a single crate, full size prototypesystem. It should be nearly identical to the productionhardware, with only minor changes necessary.Construction and testing of System 0 should be completedby the end of 2001.In addition to the planned prototype systems, additionaltechnology demonstrators may also be produced toevaluate and gain experiences with new devices whichappear on the market.6.  CONCLUSIONSAn overview of the Jet and Energy Sum processor forthe ATLAS Level-1 Trigger has been presented. Thedesign system is based on the use of commercial off-the-shelf components, including large programmable logicdevices. The design includes implementation details thatreduce latency and board complexity, and use FPGAreprogrammability to enhance system flexibility anddiagnostic capability. Technology demonstrator programshave confirmed the feasibility of key elements of theprocessor design, and two fully functional prototypesystems are planned before the start of final production.7.  REFERENCES1. http://www.xilinx.com/2. http://www.altera.com/3. The ATLAS Level-1 Calorimeter Trigger TechnicalDesign Report (1998)4. http://scgproducts.motorola.com/

Use of programmable logic in a pipelined trigger for ATLAS

Use of programmable logic in a pipelined trigger for ATLAS

Abstract

Similar works

Full text

Available Versions

CERN Document Server