Hauser, R

Legrand, I

CERN Document Server

Presented at the HPCN 95 Conference, Milan, Mav l995Reiner Hauser & losif LegrandA Real-Time Application for the CS-2CERN-CN-95-05llIllIlllllllllIlllllllllllllllllllllllllCERN LIBRARIES, GENEVAB ‘/Q 5 5 A g OCR Output<’ 6/<x\f mf fJ$· L5March 1995CN/95/5QERN-COMPUTING AND NETWORKS r>1v1$10NThis work was sponsored by ESPRIT project P7255 - GPMIMD ll OCR Outputof dividing the trigger process into several clearly separated steps, where eachconnected by a high—bandwidth switching network. Atlas follows the approachrates both the second- and third-level trigger. All processors and data buffers areThe CMS solution uses a large general-purpose processor farm which incorposlightly different approaches for implementing the second-level trigger system.The two main experiments proposed for LHC, Atlas [3] and CMS followparallel systems like the CS-2 can compete with custom-composed architectures.years in the field of processors and networks. Our goal is to investigate how closedtake advantage of the expected technological progress during the next five to tena high-speed network switch like ATM, Fibre Channel or SCI. The goal is tochitectures [1][2] to farms of commercial workstation processors connected viapossible architectures are under study, ranging from special-purpose systolic arbandwidth to about 100 GByte/sec. For the second-level trigger a number ofware and is expected to reduce the event rate to 100 kHz and the total dataThe first level trigger will be implemented in highly parallelized custom hard‘interesting’ events and discard uninteresting data.amount of data is usually reduced by using several levels of triggers, which selectsition and triggers using currently existing computers and networks. The largeMHz and the resulting amount of data represent a challenge for data acquiwill go into production in about 10 years. The high bunch crossing rate of 40The Large Hadron Collider (LHC) is a projected accelerator at CERN which1 Introductionin the next few years.today, which makes them promising candidates for a complete solutionare already able to reach about 25 to 30 % of the required performancecessor system, the Meiko CS—2. Measurements show that such systemsan irnplementation of the second-level trigger on a commercial multiproprobably consist of a parallel processing implementation. We describethe high bandwidth and short latency constraints, the solution will mostwhich can be stored on mass storage for later off-line analysis. Due totrigger system is used to reduce these data in real-tirne to an amountorders of magnitude larger than those produced by existing systems. Anext·generation collider LHC will generate amounts of data which areAbstract. Future high energy experiments planned at CERN for theDESY, 15738 Zeuthen, GermanyCERN, 1211 Geneva 23, SwitzerlandReiner Hauserl and losif LegrandzA Real-Time Application for the CS-2*i.e. the time it takes to make the final decision. OCR Outputfor these buffers depends obviously on the latency of the whole trigger system,they should be transfered to the third-level trigger or discarded. The size requiredare stored in buffers until the second-level trigger has made the decision whetherassume that packets are smaller than 64 bytes/event. The complete raw datawhich are sent to the Rol task are usually only a few bytes. ln the following weThis is typically in the order of a few kBytes/event. The computed featuresThe feature extraction tasks receive the data from the detector for one Rol.collects the data from all the Rols to compute the final decision.to a given Rol are collected into the Ro] task. Finally, the so called global tasktor. In our implementation all the features from different subdetectors belongingThe so called feature extraction tasks work on a single Rol ofa given subdetectake advantage of the inherent parallellism of the problem.algorithm for the second level trigger has been structured into several parts toInterest (ROI}, which denote areas where interesting features were found. Theto the second level trigger, the first level trigger selects so called Regions oftecture in mind. To avoid the transfer of all of the raw data from the detectorsThe current study has been carried out with the Atlas second level trigger archi2 The Atlas Second-Level Trigger SystemFig. 1. Structure of Atlas trigger architecturel|¥100Ml]/SData Sluragc10*-102rpnfzwrl.\l.}1v.\hlc¤ l limujourr l,xc1uBu|ldmp lr} HBAI Hmuv-1¢»·I.\I,Z~·=~ M ~¤·x l wm l~·~··~~~l|}"-HVmm xuow lR·\(`KI‘~Jl) I wilwLatencw Rate IH:}etc. Fig. 1 shows the general structure of the Atlas trigger architecture.trigger system, depending on the required bandwidth/latency/processing powerdifferent processor technologies and network switches in different parts of thestep parallelizes and reduces the bandwidth further. This allows them e.g. to uselatency increases to about 25 ps. OCR Outputorder of 10 us on the lowest level. When using the channel interface the startuptime is usually dominated by the startup latency of the network which is in thecan be achieved. However, for small packets (less than 200 Bytes) the transfernel communication. For large packets, transfer rates of more than 40 MByte/secFig. 3 shows the results of some bandwidth measurements for DMA and chan—intervention on the receiver side.allows e.g. data transfers to a given address on a remote processing node withoutferent processors. There is also a direct memory access (DMA) interface, whichlibrary It has an application interface built on logical channels between difbraries like MPI [6], but there is also a low—level interface, called the Elan WidgetThe communication facilities can be used from high-level message passing liallows the main CPU to continue with other operations.The actual data transfer is handled by a communication processor [5], whichcan be initiated by a normal user program without requiring any system call.The distinguishing feature ofthe CS-2 is its internal network switch. Transfersmemory each and a processor clock speed of 40 MHz.ESPRIT project GPMIMD-ll consists of 32 nodes, mostly with 32 MByte ofa high-speed, low-latency network. The system installed at CERN as part of theSolaris operating system on each processing node. The nodes are connected byThe Meiko CS—2 is a distributed memory machine, using SPARC CPUs and the3 The CS-2 at CERNFig. 2. Structure of second-level trigger program(`alorimclrrSUIIRI\|1|»deIcvIn>r\ Rui l`a>k (Hnbal Taslvrdifferent tasks.thc nodes represent computation task and the edges the data How between theFig. 2 shows the structure 0i` the corresponding program as a graph, wherenatural for the programmer. Buffers can be allocated as needed and just sent to OCR Outputthen with direct DMA transfers. The channel implementation is much moredifferent methods described above: first with the virtual channel based method,The communication part of the program has been implemented using the twoor receiving a packet, and then use polling to test for completion if necessary.munication with their computation. They always initiate a request for sendingAll parts of the program take advantage of this property to overlap their comThe communication library of the CS-2 allows for asynchronous transfers.current event. It takes typically about 30 as to perform its task.execution time for the global task is dependent on the number of Rols for thefrom different Rols, until all the data for a given event have been received. TheThe global task is structured in a similar way. A central loop collects packetstypically executes in less than 10 ps. The results are then sent to the global task.have arrived, the actual procedure for the Rol algorithm is called. This routinememory. Only pointers to them are passed around. When all expected packetsfor a given Rol have been collected. Buffers are never copied when they are inThe Rol task receives packets and puts them into a list until all the datacan keep up.we have an easy way of measuring the event rate with which the whole systempossible. Since the communication channels include a flow control mechanism,programs generate random data and feed it into the rest ofthe system as fast asextraction algorithms are not actually implemented in the nodes; instead theextraction, Rol and global task is mapped to a single processor. The featuregorithm on the Meiko CS—2 multiprocessor system. ln our setup each featureWe have implemented the global decision part of the second level trigger al4 ImplementationFig. 3. Bandwidth measurements 0n the CS—2S1ze(By1es)0 5000 10000 15000 20000 25000 30000 35000 40000 45000 5000020 Fw25} ¢¢0 T v_45 {DMATrar1s1er r·onanner Transfer Mode Vtransfered to each sending processor. OCR Outputthe addresses for buffer pool: for 2048 buffers about 8 kByte of data has to betems. This is due to the increased communication which is necessary to sendof the latter is usually better for small systems, while it decreases for large systation, one with 512 buffers and one with 2048. As one can see, the performancetake up some considerable time. There are two columns for the DMA implemenbe neglected and that the polling of a large number of incoming channels maygetting larger. This indicates that the overhead of the channel library may notplementation over the channel version increases when the system as a whole isimplementation. A more interesting point is that the speedup of the DMA imAs one can see the DMA implementation is usually faster than the channelsubdetectors/Rols. Table 1 shows the results.used. As expected, the achievable rate decreases with an increasing number of20 to 40 kHz, depending on the parameters and the communication methodfrom two to four. For the channel implementation, event rates vary from aboutThe number of subdetectors varied from two to four, and the number of Rolscan be calculated for the given setup.the system and computing the mean value for one event. From this a event rateprocessor node. The measurements were done by sending one million events intomined by the current setup ofthe CS-2. Each basic task was assigned to a singlenumber of processors needed (see Fig. 2). The maximum numbers were detersubdetectors and the number of Rols. These two parameters determine the totalThe system was run with a varying number of parameters for the number of5 Resultsby a special flag in the packets).the processor just checks if anything new has arrived in the buffers (indicatedbuffers sent without further intervention on the receiver side. On the receiver sideoverhead for the necessary synchronisation is amortized over the large number offrom the receiver. This reduces the overhead for a single packet transfer. Theafter it has used all the buffers, does it have to wait for a new list of addressesbuffers without further permission and without waiting for an acknowledge. Onlyis sent to the sender node. The sender is allowed to transfer data to the receiveon the receiver side are preallocated, then a list of the addresses of these buffersThe DMA implementation uses a different approach: a large number ofbuifersone final function call to complete the transfer.that the programmer polls each outstanding transmission separately and doestransfer the data to that buffer. Furthermore the communication library requiresbetween the communication processors to get the destination address and thenThe disadvantage of this scheme is that is requires several packet exchangesrequest, then continue with some other task.receiver may also choose an arbitrary buffer and issue an asynchronous receivethe receiver node without caring about the buffers on the destination side. TheElan Widget Library, Meiko Computing Surface Documentation, 1993.MPI: A Message-Passing Interface Standard, Version 1.0, 1994.1993.Cornmunications Processor Overview, Meiko Computing Surface Documentation,1994.CMS The Compact Muon Solenoid - Technical Proposal, CERN/LHCC/94-38,Collider at CERN, CERN/LHCC/94-43, 1994.Atlas · Technical Proposal for a General Purpose Experiment at the Large HadronD. Belosloudtsev et al., to appear in NIM, Feb 1995.J. Badier et al., IEEE Trans. Nucl. Sci. 40, 1993.Referencesof two, bringing the system quite close to the required limit.together should increase the performance ofthe application by at least a factorprocessor nodes, thereby doubling the processing performance. These two factorsthan 3 ps. In the near future the CERN machine will be upgraded to 100 MHzat the end of this year. The latency for DMA transfers will be reduced to lessA new version ofthe communication processor, the ELAN-2 will be availablethe required rate.Meiko CS-2, achieving a throughput which is only a factor three to five belowpart ofthe LHC second level trigger on a commercial multiprocessor system, theWe have shown that is possible to implement an algorithm for the global decision6 DiscussionTable 1. Timing measurements: achieved event rate in kHz17.7 13.8 2121.123.5 18.3 1727.733.4 25.2 1330.224.2 1625.2 17.830.2 22.9 1327.634.5 1031.5 27.524.9 20.9 1127.932.6 26.929.741.234.8 33.7ROIs Subdetcct0rs}DMA (512 bufs) DMA (2048 bufs) Channcl|Pr0cess0rs uscd

A real-time application for the CS-2

Abstract

Similar works

Full text

Available Versions

CERN Document Server