de Cuveland, J.

Hartmann, H.

Lindenstruth, V.

GSI Repository

CBM First-level Event Selector data management developments∗H. Hartmann1, J. de Cuveland1, and V. Lindenstruth11FIAS Frankfurt Institute for Advanced Studies, Goethe-Universita¨t Frankfurt am Main, GermanyThe First-level Event Selector (FLES) is a high perfor-mance computing cluster functioning as the central eventselection system in the CBM experiment. It combines datafrom a large number of input links to time intervals and dis-tributes them to the compute nodes, via a high-performancenetwork. Simultaneously, the FLES carries out online anal-yses and complete event reconstruction on the data. Datarates at this point are expected to exceed 1 TByte/s.The FLES system will consist on one hand of a scalablesupercomputer with custom FPGA-based input interfacecards and a fast event-building network and will be con-structed largely from standard components. On the otherhand special developed software allowing to process theincoming data in real-time builds up the FLES.A small scale, highly customizable platform, the Micro-FLES cluster was installed at GSI. Eight identical com-pute nodes provide a total of 192 logic cores and 512 GBmemory plus one head node for infrastructural services.This test system enables studies on the development of theFLES such as elaborating performant software for times-lice building.A timeslice is the fundamental data structure managingaccess to all detector raw data of a given time interval.In addition to existing timeslice building prototype soft-ware based on InfiniBand Verbs investigations of a morehigh-level interface to the network hardware have been per-formed using MPI. For this purpose a specialized microbenchmark test suite was developed simulating the FLEStimeslice building use case. Benchmark results for simul-taneous data transfer on the Micro-FLES are displayed inFig 1. When communication is established only betweenthree nodes, MPI’s performance compares to the maximumdata rate of point to point communication for InfinibandVerbs (green curve) on the Infiniband-FDR network. How-ever, the data rate decreases by 15% when all eight nodesof the Micro-Fles are participating in an any-to-any com-munication. Further tests on bigger compute cluster areneccessary to evaluate the achievable data rates for MPI ona big scale and are currently under investigation.In 2014 the FLES demonstrator system was upgradedsignifciantly to the Micro-FLES2. First, the Micro-FLES2was equipped with the latest Mellanox dual Connect-IBHCAs (mlx5), in addition to the existing Mellanox dualConnectX-3 cards (mlx4). Overall the new cards are fasterthan the old as shown in Fig. 2. A data rate of 6 GB/s can beachieved using only one of the four ports, already. Further-more, the new cards feature a 16x PCIe 3.0 interface and∗Work supported by BMBF (05P12RFFCP) and HIC for FAIRFigure 1: MPI benchmark on the Micro-FLES.Figure 2: Perfomance tests for the Micro-FLES2.therefore allow to saturize the full bandwith of both ports.An accumulated data rate of 18 GB/s can be achieved utiliz-ing all Infiniband ports. With this first upgrade the Micro-FLES2 can send data from node to node three times as fastas before (e.g., 18 GB/s instead of 6GB/s). The improvedperformance is essential for the development of timesslicebuilding software.Secondly, two further Mellanox SX6036 36-port 56Gb/sswitches were installed in order to realize different networksetups such as a fat tree. This helps investigating routingissues in the development of software when distributingthe incoming data. The previous existing switch was con-nected with full bidirectional bandwidth to both of the newswitches making them leaves of a fat tree. All first portsof mlx4 and mlx5 for each node were connected to leaf-switch1 and all second ports to leaf-switch2. Using thissetup the network structure and blocking ratio in case ofa fat tree can be configured dynamically via the providedinternet interface of the switches. The upgraded Micro-FLES2 provides better performance and a greater flexibiltyin testing different scenarios allowing to evaluate a greatervarity of possibilites for the final system – the FLES.GSI SCIENTIFIC REPORT 2014 MU-NQM-CBM-11DOI:10.15120/GR-2015-1-MU-NQM-CBM-11 47

CBM First-level Event Selector data management developments

http://repository.gsi.de/record/183818/files/MU-NQM-CBM-11.pdf

CBM First-level Event Selector data management developments

Abstract

Similar works

Full text

Available Versions

GSI Repository