Ardila, L.

Caselle, M.

Chilingaryan, S.

Dritschler, Timo

Ebersoldt, A.

Hurst, J.

Karcher, N.

Kopmann, A.

Sander, O.

Stockmann, T.

English

KITopen

www.helmholtz.deT. Dritschler,  M. Caselle, L. Ardila,  S. Chilingaryan,A. Ebersoldt, J. Hurst, N. Karcher, A. Kopmann, O. Sander, T. Stockmann(*)(Commercial) DMA technologies to realize a flexible DAQ system for Tera-Scale experimentsKarlsruhe Institute of Technology (KIT)(*)Forschungszentrum Jülich2MT Annual Meeting 2021Challenges for modern DAQ Systems § Advancements in Detector Technologies lead to highly complex DAQ systems§ This complexity results in heterogeneous systems with many different componentsComplex data paths and heterogeneous architecturesDetectorReadout Event building Processing StorageASICs? FPGAs? CPUs?GPUs?OpticalLinks?PCIe?InfiniBand?Ethernet?3MT Annual Meeting 2021Challenges for modern DAQ Systems § Development and maintenance of complex DAQ systems is expensive and time consumingComplex data paths and heterogeneous architecturesDetectorReadout Event building Processing Storage§ Using commercially available standards and „off-the-shelves“ components might mitigate these efforts§ Can we design scalable and maintainable DAQ using mostly commercial components?4MT Annual Meeting 2021BufferKernel Space Application SpaceSystem BusData transfer inside of conventional computing systemsCPU involvement in data movement5MT Annual Meeting 2021PlaceBufferKernel Space Application SpaceSystem BusDataPlaceFetchFetch§ Data arrives and is placed in device buffer§ CPU Fetches data from buffer§ CPU Places data into Driver memory§ CPU Fetches data from Driver§ CPU Places data into application spaceData transfer inside of conventional computing systemsCPU involvement in data movement§ Repeat for each Data!6MT Annual Meeting 2021DMAEngineKernel Space Application SpaceSystem BusData§ CPU „reserves“ memory in application space (Pinning)§ CPU configures the device’s DMA engine, effectively „Mapping“ the pinned memory to the device§ Data arrives and is placed right into destination memoryPinConfigureMapDirect Memory Access (DMA) enabled data transferFreeing up CPU and reducing copy efforts§ Enables microsecond latenciesat full interconnect bandwidth!To Network7MT Annual Meeting 2021RDMA enabled GPGPU computingEfficient use of GPUs for On-Line and In-Line data processingSystem BusDMAEngineData§ DMA is possible for GPU memory as well§ Analogous to main memory DMA§ Pin → Map → Place§ Significantly reduces latency!IO DeviceIO 8MT Annual Meeting 2021InfiniBand, the ‚de facto‘ standard for RDMACommercial High-Speed, Low-Latency data transfer© MellanoxSupport both: InfiniBand AND Ethernet!9MT Annual Meeting 2021Next generation RDMA capable computing acceleratorsClosing the gap between networking and computing§ Commercially available components§ Software Programmable§ Low development effort§ Easy to upgrade to newer generations, once they become available§ (R)DMA Capable§ Optimized for Machine Learning and AI applications!Xilinx ALVEO SmartNIC:Up to 1.1M LUTsUp to 100Gbps EthernetNvidia EGX A100:Ampere GPGPU,200Gbps Network,Ethernet + InfiniBand© Xilinx© Nvidia10MT Annual Meeting 2021Communication between heterogeneous componentsOne protocol for all the most common components?§ Ethernet is one of the most common commercially available interconnectsEthernet ✓ Ethernet ✓ Ethernet ✓PC / CPU GPU FPGA Network InfrastructureEthernet ✓§ Is there a way to benefit from RDMA using Ethernet?11MT Annual Meeting 2021RoCE (RDMA over Converged Ethernet)§ RoCE uses UDP packets and encapsulates InfiniBand “instructions” into the UDP payload§ Soft- and Hardware implementations available!Enabling RDMA benefits for conventional Ethernet networks12MT Annual Meeting 2021KIRO: KIT RDMA Programming LibraryIntegrating RMDA capabilities into software© MellanoxClone it on Github!https://github.com/ufo-kit/kiro§ KIRO Server/Client:§ Unidirectional data transfer from Server to Client (Clients “Pull” from Server)§ Fixed-Size memory segment§ Supports multiple connected clients per server§ KIRO Messenger (Beta):§ Layers bi-directional point-to-point messaging on top of KIRO§ Messengers can connect to multiple peers§ Fully RDMA enabled arbitrary sized memory exchange (“Push” and “Pull”)§ Works for both: InfiniBand and RoCE!13MT Annual Meeting 2021Novel DAQ architecture using commercial HPC ComponentsHigh-performance distributed ML for physics experimentsEthernet fabrics (hundreds Gb/s)DetectorsOn-line data path (by commercial devices)…FPGAAIE T H…FPGAAIE T H…FPGAAIE T HModular readoutEthernet…§ Versatile DAQ optimized for detector and AI applications § High-performance ML inference on modern programmable hardware platforms§ Novel heterogeneous FPGA/GPU architecture based on emerging Ethernet protocolsFast FeedbackHPCGPUsAIAlveoAIHighFlex AI14MT Annual Meeting 2021Machine Learning Application ExamplePANDA Tracking and Event SelectorForward Tracking Stations• 1 & 2, 3 & 4, 5• 3&4 inside 2 Tm dipole fieldGraph Neural NetworkIn Total:§ Connection of all hits from one layer with all hits in next layer§ 6 charged tracks§ Track finding efficiency  96%Most probable connections surviveWaleed Esmail15MT Annual Meeting 2021Machine Learning Application ExamplePANDA Tracking and Event SelectorPANDA online event selectorBest network: CNNEfficiency improvement for different physics channelsCNN vs. Classical methodPeiyong JiangSoftware trigger selects physical interesting channels from background for data storageAll data points above 100% efficiency vs. conventional approach → Indicates Neural Network performs better accross all metrics100%§ Heterogeneous system§ Unifiable communication§ Proposed for PANDA DAQ16MT Annual Meeting 2021Heterogeneous FPGA – GPU architecture for beam physicsHigh-performance distributed ML for physics experimentsEthernet fabrics (hundreds Gb/s)FPGAAI ETHFPGAAI ETHFPGAAI ETHModular readoutHPCGPUsAIAlveoAIHighFlex AI§ Versatile DAQ optimized for detector and AI applications § High-performance ML inference on modern programmable hardware platforms§ Novel heterogeneous FPGA/GPU architecture based on emerging Ethernet protocolsDetectors………B am diagnosticsSynchrotronAcceleratorKAPTUREKALYPSO…EthernetFast FeedbackOn-line data path (by commercial devices)17MT Annual Meeting 2021Hardware implementation Reinforcement Learning on modern programable deviceDeep Deterministic Policy Gradient (DDPG)Actor consists of four fully-connecteddense layers, each layer consists of 64neurons using a rectified linear unit(ReLU) as activation function in FPGAActorCritic neural network and the policygradient are implemented as bare-metalapplication on ARM processorCriticAs seen in:  „Accelerated Deep Reinforcement Learning for Fast Feedback of Beam Dynamics at KARA“   A. Ebersoldt18MT Annual Meeting 2021Control of the complex beam with MLMachine Learning toward Autonomous AcceleratorsRF SystemactionTHz detectorKAPTUREHighFlex cardAI deployed in the FPGA§ Closed feedback loop at KARA: § Detection of signals with THz detectors and KAPTURE @ 500 MPulse/s§ Data processed by Reinforcement Learning on FPGA§ FPGA action as special RF signal modulation is sent to the kicker cavity§ Goal: total latency of control feedback loop << 1 ms§ Target applications: KARA, FLUTE, ARES and more§ Status: first beam control on FPGA developed withinAMALEAà will continue in ACCLAIM (Helmholtz Innovation fund) MicroTCA A. EbersoldtFeedbackEthernet19MT Annual Meeting 2021DigitalConversionRFFPGA~DACADCUp-conversion4 … 8 GHzDown-conversion±400 MHz24Baseband frequency comb generationDigitization &signal processing153Storage / BackendServer𝑓|a|𝑓!𝑓" 𝑓#…|a|𝑓! 𝑓𝑓" 𝑓#…Application: RoCE in FPGA for the ECHo experimentECHo Experiment to measure the Neutrino massJonas Hurst, Nick Karcher, Oliver Sander20MT Annual Meeting 2021RoCEMaster Thesis: „Entwicklung eines breitbandigen Datentransfers (DMA) für die Messdatenakquise“  (Jonas Hurst, Nick Karcher, Oliver Sander)Xilinx Zynq US+(64bit ARM + FPGA)Development BoardApplication: RoCE in FPGA for the ECHo experimentXilinx ERNIC IP Core implementation and integrationKIRO21MT Annual Meeting 2021ConclusionApplication of industrial RDMA standards for modern DAQ§ Computing Accelerator integration → Full support for HPC and on-line processing § Readily available Hard- and Software → Drastically reduces development efforts§ Single, commercial components → Easy maintenance and easy upgrades to newer versions§ Highly versatile → Unify communication between all components: CPU, GPU, FPGABenefits:Drawbacks:§ Integration into existing infrastructure requires (complex) retooling§ Difficult to deploy and operate from within virtualized software environments§ Full performance can only be reached when using dedicated networking hardware

(Commercial) DMA technologies to realize a flexible DAQ system for Tera-Scale experiments

https://publikationen.bibliothek.kit.edu/1000133471/149775005

(Commercial) DMA technologies to realize a flexible DAQ system for Tera-Scale experiments

Abstract

Similar works

Full text

Available Versions

KITopen