State-of-the-art technology supports the High Energy Physics community in addressing the problem of managing an overwhelming amount of experimental data. From the point of view of communication between the detectors’ readout system and computing nodes, the critical issues are the following: latency, moving data in a deterministic and low amount of time; bandwidth, guaranteeing the maximum capability of the link and communication protocol adopted; endpoint consolidation, tight aggregation of channels on a single board. This contribution describes the status and performances of the NaNet project, whose goal is the design of a family of FPGA-based PCIe network interface cards. The eﬀorts of the team are focused on implementing a low-latency, real-time data transport mechanism between the board network multi-channel system and CPU and GPU accelerators memories on the host. Several opportunities concerning technical solutions and scientiﬁc applications have been explored: NaNet-1 with a single GbE I/O interface, and NaNet-10, oﬀering four 10GbE ports, for activities related to the GPU-based real-time trigger of NA62 experiment at CERN; NaNet3, with four 2.5Gbit optical channels, developed for the KM3NeT-ITALIA underwater neutrino telescope

Ammendola, R.

Biagioni, A.

Cretaro, P.

Frezza, O.

Lamanna, G.

Scientific Open-access Literature Archive and Repository

DOI 10.1393/ncc/i2017-17073-xColloquia: IFAE 2016IL NUOVO CIMENTO 40 C (2017) 73Reconfigurable PCI Express cards for low-latency data transportin HEP experimentsR. Ammendola(1), A. Biagioni(2), P. Cretaro(2), O. Frezza(2), G. Lamanna(3),F. Lo Cicero(2), A. Lonardo(2), M. Martinelli(2), P. S. Paolucci(2),E. Pastorelli(2), L. Pontisso(4), F. Simula(2) and P. Vicini(2)(1) INFN, Sezione di Roma Tor Vergata - Roma, Italy(2) INFN, Sezione di Roma Sapienza - Roma, Italy(3) INFN, Laboratori Nazionali di Frascati - Frascati (Roma), Italy(4) INFN, Sezione di Pisa - Pisa, Italyreceived 17 October 2016Summary. — State-of-the-art technology supports the High Energy Physics com-munity in addressing the problem of managing an overwhelming amount of exper-imental data. From the point of view of communication between the detectors’readout system and computing nodes, the critical issues are the following: latency,moving data in a deterministic and low amount of time; bandwidth, guaranteeingthe maximum capability of the link and communication protocol adopted; endpointconsolidation, tight aggregation of channels on a single board. This contribution de-scribes the status and performances of the NaNet project, whose goal is the designof a family of FPGA-based PCIe network interface cards. The efforts of the teamare focused on implementing a low-latency, real-time data transport mechanismbetween the board network multi-channel system and CPU and GPU acceleratorsmemories on the host. Several opportunities concerning technical solutions and sci-entific applications have been explored: NaNet-1 with a single GbE I/O interface,and NaNet-10, offering four 10GbE ports, for activities related to the GPU-basedreal-time trigger of NA62 experiment at CERN; NaNet3, with four 2.5 Gbit opticalchannels, developed for the KM3NeT-ITALIA underwater neutrino telescope.1. – IntroductionModern High Energy Physics experiments work at high interaction rates and an effi-cient trigger mechanism is often mandatory to pick potentially interesting events amongcrowded backgrounds. For the purposes of this discussion, two broad categories can beidentified in the varied scenario of trigger systems: on-line and off-line.The off-line triggers perform the analysis once data reaches the final destination,usually a PC-farm; in this case the primary obstacles to face are: i) database and storagemanagement, to guarantee an efficient organization of recorded data, ii) high bandwidthCreative Commons Attribution 4.0 License (http://creativecommons.org/licenses/by/4.0) 12 R. AMMENDOLA et al.Fig. 1. – NaNet PCIe Network Interface Card family architecture.(and low-latency as well) network adaptor, to handle large amounts of data in a fast way,iii) great computing power, to keep analysis time low and iv) endpoint consolidationcapability, to keep to a minimum the number of final devices and reduce the operatingcost of the server host infrastructure.In the context of on-line analysis, the data flow is processed along the path; here thechallenging parameters are: i) low and (almost) deterministic latency —full control ofthe time fluctuations in moving data towards the memory of the computing device—ii) high throughput of the trigger processor and iii) large computing power but witha deterministic execution behaviour. Available choices for data handling are: i) fullhardware module —ASICs or FPGAs— implementing quite simple but fast and valuabletrigger primitives, ii) exploitation of COTS hardware with trigger applications runningon CPUs or many-core accelerators, such as GPUs.A network interface component tasked with managing the data flow from the detectorreadout to the adopted computing device plays a key role in all the aforementioned cases.2. – NaNetThe NaNet [1] project aims at designing and building a family of FPGA-based PCIenetwork interface cards (NIC) providing a bridge between the readout of the detectorsand the computing nodes of a PC farm by means of a real-time data transport mechanism.The NaNet architecture is built upon the assurance of the complete control of the latency.This feature is achieved by implementing in hardware the data transfer tasks. In this way,latency fluctuations can be minimized — a core issue for an on-line trigger setup — untila completely deterministic behaviour for the communication management is reached,devoid of OS jitter effects. Consequently, hardware handling of the data path allows fora low-latency and high-bandwidth implementation of the data transport system.In fig. 1 an overview of the NaNet design is sketched. The I/O interface is responsiblefor the network stack and is the distinctive trait of the NaNet board variant matchedRECONFIGURABLE PCI EXPRESS CARDS FOR LOW-LATENCY DATA TRANSPORT ETC. 3Fig. 2. – Pictorial view of GPU-based Trigger.to the use case requirements. The Physical Link Coding and Protocol Manager IPs canbe modified to match the detector readout interface of each experiment allowing for areconfiguration of the channel. The former manages the line coding, alignment pro-cess, bit rate, transmission mode, media access and logical link control — i.e. 8B10B,10/100/1000 MAC, Altera Deterministic Latency, Altera 10GBASE-KR PHY and the10Gbps MAC [2] — while the latter is in charge of handling the communication protocol— i.e. standard UDP, Word-stuff and Time Division Multiplexing of proprietary APElinkand KM3link channels. The Data Processing module can perform application-dependentmodifications on the data stream on the fly — e.g. data reshuffling towards more device-friendly formats or (de)compression. The Transmission Control Logic (NaNet TCL)runs a protocol translation optimizing the PCIe transactions and generates the virtualaddresses for appropriate interaction with the Linux Kernel. The router module consol-idates the endpoints by multiplexing up to 10 data flows at 2.8 GB/s using a dimension-order routing algorithm. Finally, the Network Interface achieves Zero-copy networking,moving data to/from application memory by means of a DMA engine implementing theRDMA protocol for both CPU and GPU — this latter supports GPUDirect V2/RDMAby nVIDIA.3. – NaNet at the NA62 CERN Experiment: data transport for the RICHDetector GPU-based L0 TriggerInvestigating the feasibility of a GPU-based L0 trigger system for the NA62 exper-iment (GPU L0TP) led us to focus on the ring-shaped hit patterns reconstruction inthe RICH detector of the NA62 experiment [3]. Here the detector identifies pions andmuons with momentum in the range between 15 GeV/c and 35 GeV/c. Čerenkov lightis reflected by a composite mirror with a focal length of 17 m and focused onto twoseparated spots equipped with ∼ 1000 photomultipliers (PM) each.Data from the PMs are gathered by four readout boards (TEL62, Trigger Electronicsfor NA62), each sending primitives to the GPU L0TP (fig. 2) by GbE UDP streams.Communication main requirement is the deterministic response latency ofGPU L0TP; the available time budget forces both communication and computation tobe under 1ms.Refined primitives from the GPU-based computation are sent to the central L0 proces-sor, where information from other detectors is taken into account in the trigger decision.In 2015 the GPU-based trigger at CERN includes 2 TEL62 boards connected to aHP2920 switch and a NaNet-1 board with a TTC HSMC daughtercard plugged intoa server made of a X9DRG-QF dual socket motherboard populated with Intel XeonE5-2620 @2.00 GHz CPUs (i.e. Ivy Bridge architecture), 32 GB of DDR3 RAM and aKepler-class nVIDIA K20c GPU.4 R. AMMENDOLA et al.Fig. 3. – Multi-ring reconstruction.Such a system allows testing of the whole chain: the data events move towards theGPU-based trigger through NaNet-1 by means of the GPUDirect RDMA interface. Alldata landing within a configurable time frame are gathered and organized in a parametriz-able Circular List Of Persistent buffers (CLOP) in GPU memory. This time frame mustbe always shorter or equal to how long multi-ring reconstruction takes on the GPU, tobe sure that buffers are not overwritten before they are consumed if one wants to avoidmemory copies. Events are timestamped; those sharing a time-window but coming fromdifferent boards are fused into one event describing the PMs status in the RICH detector.The multi-ring reconstruction GPU kernel is based on the histogram algorithm: theXY -plane is divided into a grid and the distances from gridpoints to hits of the physicsevent are put into bins so that those whose contents exceed a threshold value identifyrings. The computing kernel implements the histogram fitter with a single step (i.e. overan 8×8 grid only) and is executed on a full CLOP buffer as soon as the NIC signals to thehost application that the buffer is available to be consumed. Data selection might benefitfrom taking into account the parameters of Čerenkov rings, whose values are determinedusing the coordinates of activated PMs.Results are in fig. 3; the CLOP sizes lie on the X-axis measured as number of receivedevents while on the Y-axis the latencies of different stages are shown. Events comingfrom 2 readout boards, for a gathering time of 400μs, and parameters like events rate(collected with a beam intensity of 4 × 1011 protons per spill), a CLOP’s size of 8 KB,time frame was chosen so that we could test the online behaviour of the trigger chain.Since the merge operation requires synchronization and serialization and does not exhibitmuch parallelism, as such it is an ill-suited problem to the GPU architecture. Again,high latency of this task suggests offloading its duty to a hardware implementation.4. – NaNet at KM3NeT-ITKM3NeT-IT [4] is an underwater experimental apparatus for detection of high energyneutrinos in the TeV–PeV range. It consists of an array of photomultipliers (PMTs)exploiting the Čerenkov effect produced by charged particles propagating in sea water.The final assessment of the experiment foresees the installation of 8 detection unitscalled tower [5], each one consisting of 14 floors vertically spaced 20 m apart and abaseline. Each floor is 8 m long and hosts 6 Optical Modules (OM) —each contain-ing a ten-inch photomultiplier (PMT) and the front-end module (FEM) to acquire andtransmit the pulse generated by the PMT—, 2 hydrophones to reconstruct in real-timeRECONFIGURABLE PCI EXPRESS CARDS FOR LOW-LATENCY DATA TRANSPORT ETC. 5the OM position and a board, called Floor Control Module (FCM). The FCM collectsOM-produced data and manages the communication between the on-shore laboratoryand the underwater devices through an optical fiber.In order to reconstruct the particle tracks, accurate knowledge of spatial and temporaldistribution of the hits is mandatory. This implies a continuous tracking of OMs positionand a system-wide distribution of a common timing. As regards the last constraint, theendpoint board in the laboratory is in charge of distributing a common clock all over thesystem, and data frames labelled with a “time stamp” are encapsulated in a synchronouslink protocol which embeds clock and data with deterministic latency.The customization for the KM3NeT-IT experiment of NaNet board is calledNaNet3 [6]; by means of a real-time data transport mechanism it is able to managemultiple FCM data channels, allowing for more cost-effective and efficient scaling in-frastructure. NaNet3 is implemented on the Terasic DE5-net board, equipped with anAltera Stratix V FPGA, four I/O channels with maximum capability of 800 Mbps eachand a PCIe Gen2 × 8 edge connector. The card implements a synchronous link proto-col with deterministic latency at the physical level and a Time Division Multiplexing(TDM) protocol at the data level (see fig. 1). A GPS clock acts as reference and is usedfor the optical link transmission from the on-shore board towards the underwater FCM.GPS signals —i.e. clock and IRIG data— are received from the two SMA connectorson board. Incoming TDM data —i.e. hydrophones and photomultipliers data— aretranslated in a packet-based protocol, properly handled —i.e. computing the destina-tion virtual address— and then sent to the computing node memory through a PCIeDMA write process. The outbound flow consists instead of slow-control data for the un-derwater apparatus; data addressed to the off-shore devices are stored in registers (onefor each underwater device) through PCIe target mode and a custom TX module is incharge of dispatching the messages to the proper channel. One data frame is transmittedevery 125 μs, with a GPS tagging every frame with a 12.5 ns precision timestamp.5. – ConclusionOur envisioned design of FPGA-based NICs with hardware-controlled latency andstream processing capabilities proved effective in two TRIDAQ contexts. Besides thehigh performance as a PCIe NIC, the key enabling features of NaNet provided acceler-ation both at communication protocol management and data stream processing stages,resulting in an integrated Network/Processing low-latency accelerator suitable for real-time (GPU) processing systems. The FPGA technology scaling endorses our currenteffort in further developing the design towards larger aggregated bandwidth and process-ing capabilities, to face the evergrowing requirements imposed by HEP experiments.REFERENCES[1] Ammendola R. et al., J. Instrum., 9 (2014) C02023.[2] “Altera Transceiver PHY IP Core User Guide.” https://www.altera.com/content/dam/altera-www/global/en US/pdfs/literature/ug/xcvr user guide.pdf.[3] Lamanna G., J. Phys. Conf. Ser., 335 (2011) 012071.[4] Margiotta A., J. Instrum., 9 (2014) C04020.[5] Nicolau C. A. et al., EPJ Web of Conferences, 116 (2016) 05011.[6] Ammendola R. et al., EPJ Web of Conferences, 116 (2016) 05008.

Reconﬁgurable PCI Express cards for low-latency data transport in HEP experiments

http://eprints.bice.rm.cnr.it/19746/1/ncc11274.pdf

Reconﬁgurable PCI Express cards for low-latency data transport in HEP experiments

Abstract

Similar works

Full text

Available Versions

Scientific Open-access Literature Archive and Repository