Ardilla Perez, L.

Balzer, M.

Caselle, M.

Dritschler, T.

Kopmann, A.

Mohr, Hannes

Rota, L.

Schilingaryan, S.

Schuh, T.

Weber, M.

English

KITopen

EVALUATION OF GPUS FOR HIGH-LEVEL TRIGGERS IN HIGH ENERGY PHYSICSPresenter: Hannes Mohr - KIT, IPE Contributors: L. Ardilla Perez, M. Balzer,  M. Caselle, S. Chilingaryan, T. Dritschler,  A. Kopmann, L. Rota, T. Schuh, M. Weber 29TH SEPTEMBER, TWEPP 2016OUTLINE▸ Implement track trigger using GPUs ▸ Use established methods for seeding ▸ Present our own version of the Hough transformation ▸ Compare different GPUs/vendors ▸ Investigate data transfer/latencies ▸ Estimate impact of technological advancesOUR GOAL IS TO ACHIEVE COMPETITIVE RESULTS, WHILE GAINING FLEXIBILITY2CMS DETECTOR*Image CMS Collaboration3SILICON TRACKERBASELINE GEOMETRY - 6 LAYERS 5 DISKS4*Image CMS CollaborationLuminosity [cm-2s-1]YearPeak luminosity Integrated luminosityUPGRADE TIMELINE 5*Image CMS CollaborationTHE LS3 UPGRADE▸ Current CMS trigger won’t be able to handle: ▸ Increased data rates ▸ Increased pile-up ▸ Currently proposed solution: ▸ Data reduction on detector ▸ Raise latency of trigger from 3.4 to 12.5 μs ▸ L1 track trigger ?!6EXPECTED DATA RATES▸ Readout at 40 MHz, BX every 25 ns  ▸ 6 μs each for L1 Trigger and Global Trigger ▸ L1 Tracking to combine Track seeding and Fitting 7Detector  HitStub  BuildingL1 TrackingGlobal Trigger≈104Tb/s 50Tb/s 1Tb/sBack EndFront EndSTUB BUILDING▸ Applies momentum cut to hits ▸ Delivers estimate on track bend ▸ Drastically decreases number of hits by a factor of 1008CURRENT APPROACHES▸ Associative Memory approach (ASICs) ▸ Time-multiplexed FPGA Hough transformation ▸ …CURRENT APPROACHES USE SPECIALIZED HARDWAREWHAT ABOUT GPUS?9Nvidia Tesla K40c vs. XILINX VIRTEX-7 XC7VX1140T (both 28nm)COMPARISON GPU VS. FPGAI/O BW [GB/S]MEMORY BANDWIDTH [GB/S]GFLOPS*GFLOPS/WATT*GFLOPS/$*0 0,25 0,5 0,75 1GPU FPGA163482881,821,4420,810,24504085010*SINGLE PRECISION,FLOATING POINTQUALITATIVE COMPARISON OF STRENGTHSGPU FPGARapid development cycles and high flexibility Huge I/O BandwidthLarge bandwidth to external memoryDeterministic timings/runtimesHigh Floating-point performanceHigh bit-level performance11Equivalent to FPGA approach, used by collaboration of KIT and UK Track Trigger Group* ‣ Uncompress data ▸ Perform Hough transformation ➤ Uses module bend information ▸ Apply layer condition ▸ Reject or return track candidatesPartitioning of tracker 9 x η 32 x ɸHOUGH TRANSFORMATION 12*An FPGA-Based Track Finder for the L1 Trigger of the CMS Experiment at the High Luminosity LHC (DOI: 10.1109/RTC.2016.7543102)HOUGH TRANSFORMATIONCalculate possible  (φ0, q/pt) pairs for each hit▸ Make histogram in  Hough spaceq/ptφ0φ(r)= m(q/pt)r+φ013HOUGH TRANSFORMATIONCalculate possible  (φ0, q/pt) pairs for each hit▸ Make histogram in  Hough spaceq/ptφ0φ(r)= m(q/pt)r+φ014HOUGH TRANSFORMATIONCalculate possible  (φ0, q/pt) pairs for each hit▸ Make histogram in  Hough spaceq/ptφ0φ(r)= m(q/pt)r+φ015CLUSTERING POINTS ARE TRACK CANDIDATESq/ptφ-φ0TTBar event - PU 14016FILTERING BY LAYER CONDITIONq/ptφ-φ0TTBar event - PU 14017OUR IMPLEMENTATIONGPU  implementation specifics: ▸ Optimized for minimum latency ▸ Computes q/pt-bins in parallel ▸ Almost no dependence on number of stubs18▸ Kernel scheduling ▸ Kernel launch time ▸ Allocation of shared memory ▸ …?NEEDS SPINNING KERNELOVERHEADS…Invocation and setup of kernels is too costly, we need to keep it running continuously19BENCHMARK KERNEL RUNTIME - SPINNINGStubstime [μs]CURRENTLY NOT POSSIBLE IN OPENCL: CACHE CAN’T BE FLUSHED FROM KERNEL!CUDA - Tesla K40c2021WHAT ABOUT DATA TRANSFER?DMA SETUP - CPU ONLY STARTS THE KERNEL 22DMA MEASUREMENT SETUPFPGA GPUPRELOAD DATA1 GPU:request dataKERNEL RUNS CONTINUOUSLY23DMA MEASUREMENT SETUPPRELOAD DATA1 GPU:request data2 FPGA:Data Transfer2 GPU:Poll for dataSIMULTANEOUSLYKERNEL RUNS CONTINUOUSLY24FPGA GPUDMA MEASUREMENT SETUPPRELOAD DATA1 GPU:request data2 FPGA:Data Transfer2 GPU:Poll for data3 GPU:ComputesSIMULTANEOUSLYKERNEL RUNS CONTINUOUSLY25At the moment we don’t write back into the FPGA,  estimate another 2 μs penalty (conservative estimate)FPGA GPUTime [μs]0,002,004,006,008,00POLLING TRANSFER TOTALMean MaxDMA BENCHMARK: POLLING - SPINNING KERNEL➤ Start transfer ➤ Poll for data ➤ Write back resultread and write 160 stubs (64 bits each)latency limitestimated response time**26Time [μs]0,002,004,006,008,00TRANSFER COMPUTATION TOTALMean MaxDMA BENCHMARK - HOUGH TRANSFORMATION➤ Read/Uncompress data ➤ Compute ➤ Poll160 stubs, 1 sector estimated response time**27latency limitINCREASING THROUGHPUT▸ Computation time is higher than data transfer ▸ We can hide the transfer behind the computationINTERLEAVED APPROACH▸ Start data transfer for current data set ▸ Do calculations on previous dataset (lies in register memory) ▸ Poll new data (should take less time)INCREASES THROUGHPUT AT COST OF LATENCY28DMA BENCHMARK: INTERLEAVED HTTime [μs]0,001,252,503,755,00POLLING COMPUTATION TOTALMean Max➤ (poll) Read/Uncompress data ➤ Ask for data ➤ ComputeData older, throughput higherestimated response time**29NEW APPROACH - HEXAGONAL HOUGH-SPACE▸ Hexagonal bins in hough space ▸ Suppresses fake candidates ▸ Runtime comparable ▸ only 1 possible bin per row ▸ less algorithmic branchingCAVEAT: NEEDS MORE BINS (FACTOR OF 2)30COMPARISON OF FAKE RATES - REGULAR VS. HEXAGONALResults for TTBar Dataset PU140, whole detector, 1 eventTRACK CANDIDATES0 22 44 66 88 110REGULARHEXAGONALnumber of true tracks (20)31Preliminary ResultsDMA BENCHMARK: HEXAGONAL HTTime [μs]0,002,004,006,008,00POLLING COMPUTATION TOTALMean Max➤ (poll) Read/Uncompress data ➤ Ask for data ➤ Computeestimated response time**32latency limitCONCLUSIONSPerformance: ‣ Computational time of around 4 μs ‣ Transfer time of around 2 μs                    Surpassed our expectations Development is faster More complex algorithms are possible: ‣ Example: hexagonal approach Data transfer using standard interfaces is challenging 33OUTLOOK▸ Need to process multiple sectors per card in future ▸ Look at performance of newer cards  ▸ High Bandwidth Memory, already in consumer model cards,  promises 2-4x better throughput ▸ Investigate new transfer technologies ▸ PCIe 4.0 (2x faster) ▸ nv-link (5-10x faster)MOORS LAW IS OUR FRIEND!34THANK YOU!QUESTIONS?35CUDA 7.5 - TESLA K40C 36CUDA 7.5 - Tesla K40CCUDA 8 - TESLA K40C 37

Evaluation of GPUs for high-level triggers in high energy physics

https://publikationen.bibliothek.kit.edu/1000064174/4004855

Evaluation of GPUs for high-level triggers in high energy physics

Abstract

Similar works

Full text

Available Versions

KITopen