The Compressed Baryonic Matter (CBM) experiment is designed to handle interaction rates of up to 10 MHz and up to 1 TB/s of raw data generated. With triggerless streaming data acquisition in the experiment and beam intensity fluctuations, it is expected that occasional data bursts will surpass bandwidth capabilities of the Data Acquisition System (DAQ) system. In order to preserve integrity of event data, the bandwidth of DAQ must be throttled in an organised way with minimum information loss. The Timing and Fast Control (TFC) system provides a latency-optimised datapath for throttling commands and distributes a system clock together with a global timestamp. This paper describes a prototype design of the system with focus on synchronisation and its evaluation

Becker, J.

Bähr, S.

Emschermann, D.

Fröhlich, I.

Müller, W. F. J.

Sidorenko, V.

Sturm, C.

English

KITopen

Prepared for submission to JINSTTopical Workshop on Electronics for Particle Physics2021Prototype Design of a Timing and Fast Control system inthe CBM ExperimentV. Sidorenko,𝑎 I. Fröhlich,𝑏 W.F.J. Müller,𝑏 D. Emschermann,𝑏 S. Bähr,𝑎 C. Sturm,𝑏 J.Becker𝑎 on behalf of CBM collaboration𝑎Karlsruhe Institute of Technology, Institute for Information Processing TechnologiesEngesserstraße 5, 76131 Karlsruhe, Germany𝑏GSI Helmholtz Centre for Heavy Ion Research,Planckstraße 1, 64291 Darmstadt, GermanyE-mail: vladimir.sidorenko@kit.eduAbstract: The Compressed Baryonic Matter (CBM) experiment is designed to handle interactionrates of up to 10 MHz and up to 1 TB/s of raw data generated. With triggerless streaming dataacquisition in the experiment and beam intensity fluctuations, it is expected that occasional databursts will surpass bandwidth capabilities of the Data Acquisition System (DAQ) system. In orderto preserve integrity of event data, the bandwidth of DAQ must be throttled in an organised waywith minimum information loss. The Timing and Fast Control (TFC) system provides a latency-optimised datapath for throttling commands and distributes a system clock together with a globaltimestamp. This paper describes a prototype design of the system with focus on synchronisationand its evaluation.Keywords: Detector control systems (detector and experiment monitoring and slow-control sys-tems, architecture, hardware, algorithms, databases); Control and monitor systems onlinearXiv:2110.12738v2  [physics.ins-det]  20 Dec 2021Contents1 Introduction 12 Concept of the Timing and Fast Control system 13 Firmware development 24 Implementation 35 Prototype evaluation 46 Conclusion and outlook 41 IntroductionThe Compressed Baryonic Matter (CBM) experiment aims to study rare probes produced in heavyion interactions. With the intended interaction rate of up to 10 MHz, the experiment will be equippedwith self-triggered front-end electronics and a free-streaming data acquisition (DAQ) system. 1TB/s of timestamped raw experimental data will be generated by the front-end electronics andpreprocessed in parallel by a layer of 200 FPGA-based Common Readout Interface (CRI) boards.These boards also serve as a gateway for the data to a computing farm hosting the First-Level EventSelector (FLES) network, where online event reconstruction takes place.Beam intensity fluctuations are expected in the experiment [1]. They are likely to causelocalised congestion of the DAQ system. Partial loss of raw data leads to collecting corruptedevents that cannot be reconstructed. A prior study demonstrated that collection of complete eventsin these conditions can be maximised using a centralised throttling mechanism [2]. The throttlingdecision is based on buffer and link occupancy in the DAQ network and is distributed to CRIboards to prevent uncontrolled data loss. A major prerequisite for throttling implementation is alow-latency control network with under 6 `s round-trip time [3].The current article presents a prototype design of a Timing and Fast Control (TFC) system. Thesystem is meant to provide the framework for throttling implementation as well as sub-nanosecondprecision clock and timestamp distribution to the experimental electronics. The paper starts withthe high-level concept of the TFC network. Afterwards, it covers firmware architecture and timedistribution mechanism. In the end, the evaluation setup is described and the results are presented.2 Concept of the Timing and Fast Control systemThe proposed TFC network has hierarchical structure with a single controller and focus on scala-bility. Network nodes are FPGA boards interconnected with bidirectional optical links.– 1 –Endpoint nodes collect and aggregate FIFO/network occupancy information. This informationis then forwarded by a hierarchy of Submasters to a central TFC Master node. This node is the rootof the system and is responsible for making the throttling decision. After that, a throttling commandis broadcasted downstream to Endpoints.Another purpose of the TFC system is experiment-wide distribution of a clock and a 64-bitglobal timestamp. Cascaded clock recovery is used to propagate the clock from Master to Endpoints.In this scheme, a Gigabit Transceiver (GTH) in each Submaster node recovers the clock from anupstream link and reuses it for all downstream connections (Figure 1). With the constraints of evennetwork depth and deterministic downstream latency, global clock reaches all Endpoints with thesame certain skew relative to the Master node.Figure 1: Cascased clock propagation scheme. Each subsequent node passes recovered clockthrough a jitter attenuator and reuses it for further transmission.3 Firmware developmentFirmware is developed in three variants, one for each TFC node role. Roles differ by the numberand direction of the TFC interfaces. Master node only has a number of downstream connections,whereas Submaster boards also require one TFC link in the Master-side direction. From thisperspective, Endpoints are the simplest nodes and need only one link to communicate with the TFCnetwork.To have a local notion of time, each TFC node runs a 64-bit timestamp counter at the basefrequency of the experiment - 40 MHz. The counter is connected to a dedicated finite-statemachine (FSM) that functions according to link direction. If the FSM serves a downstream link,it periodically transmits the local timestamp. In the case of an upstream link, the FSM receivesincoming timestamps and applies a correction to the local counter if needed. Detailed architectureof the prototype firmware cores is shown in Figure 2. Xilinx Multi-Mode Clock Manager (MMCM)primitives are used to synthesise various frequencies inside FPGA.Protocol routines and link abstraction from the user logic are handled by a Link AccessUnit (LAU). This module handles the necessary clock domain crossings with the transceiver andperforms link initialisation and protocol check for link status monitoring (Figure 3). In futureimplementations, LAU will also be responsible for shared link access for timing and fast controlmessages.– 2 –Figure 2: Architecture of Master and Endpoint TFC cores.Various components of any variant of the TFC core are controlled over Wishbone bus. Pro-vided that control software can access the same bus, the design allows to leverage the benefits ofhardware/software co-design.Major system clock frequency 𝑓𝑠𝑦𝑠 in the experiment is 40 MHz. Data transport logic operatesat a higher frequency 𝑓𝑡𝑟 = 120 MHz. To ensure deterministic phase of 𝑓𝑠𝑦𝑠 clock at the endpoint,a concept of subcycles was introduced. A counter runs at the higher of the frequencies and countsup to 𝑓𝑡𝑟𝑓𝑠𝑦𝑠, thus unambiguously labeling the edges relative to the slower clock. This way, alignmentof the slower clock with any edge of the faster clock can be characterised by a value of this counter.Based on this value, an Endpoint node can recover the phase of its system clock.4 ImplementationPrototype firmware has been developed on the BNL-712 platform [4]. These boards feature a KintexUltrascale FPGA, Si5345 jitter attenuator and 8 MiniPOD fiber optics modules that provide accessto 48 bidirectional optical connections.Clock is recovered from incoming links using the built-in clock-data recovery mechanism ofXilinx GTH primitives. This clock must be cleaned with a PLL before it can be used by firmwarecomponents because of high jitter. In our case, it is passed through the onboard Si5345 jiterattenuator. Upon a TFC link loss, the internal clock switchover mechanism of Si5345 is used tosmoothly transition to a local free-running reference. The phase shift rate of the output clock islimited at this point by the PLL bandwidth of 100 Hz. Thus, the clock remains uninterrupted at alltimes regardless of TFC link status.For timestamp synchronisation, dedicated finite-state machines (FSM) were implemented inVHDL. The master-side FSM periodically serialises the current timestamp into 32-bit words and– 3 –sends it to all downstream nodes and the endpoint FSM deserialises the timestamp and applies it tothe local counter. The endpoint FSM also captures the subcycle at which the timestamp has arrived,and the system clock is aligned accordingly to ensure phase determinism.5 Prototype evaluationIn order to evaluate the design, a test setup has been integrated at GSI1, Germany (Figure 4). EachBNL-712 has a slot for a timing mezzanine card (TMC) that adds an optical fibre interface to theboard. One of such TMC cards is used on each TFC Endpoint node for upstream communication,since the MiniPOD connections are reserved for GBT links to transport experimental data.Figure 3: Structure of Link Access Unit.Figure 4: Test setup for evaluation of theprototype design.As part of the test setup, one TFC Master and two Endpoints were mounted into server machinesusing PCIe. These machines were running Device Control Agent (DCA) software that can accessFPGA register file. High-level test control was implemented as a Python script that uses DCAbindings and interacts with the server machines over Ethernet. To provide a common time referencefor initial design evaluation, the server machines were synchronised with an off-the-shelf PrecisionTime Protocol (PTP) solution [5].During the test, local timestamp drift was measured on each node with reference to the commonPTP time. Local timestamps were recorded each second over a period of 104 s. The test itselfwas run twice: first with clock and timestamp synchronisation disabled and the second time withsynchronisation turned on. As a result, local time drifted off during the asynchronous run by nearly30 ms between the Endpoints and both Endpoints drifted from the Master node by over 100 ms(Figure 5a). With synchronisation on, no time drift has been observed and all three measured clocksremain synchronous with a constant phase offset (Figure 5b). Nevertheless, precision of the test islimited due to using PTP as a reference and uncertainty of timestamp readout latency.In addition to the lab test, the prototype has been integrated into the experimental setup with8 CRI Endpoints included. The setup has successfully passed the beam test and average collisionrate of 1 MHz has been achieved.6 Conclusion and outlookAs a result of the presented work, a prototype of the Timing and Fast Control system for the CBMexperiment has been developed with focus on synchronisation functionality. It features system-wide clock distribution through cascaded clock recovery and periodic broadcast of the reference1GSI Helmholtz Centre for Heavy Ion Research– 4 –(a) Asynchronous run. (b) Synchronous run.Figure 5: Time drift between TFC nodes in asynchronous and synchronous modes.timestamp. To support future data throttling functionality, the prototype provides bidirectionaloptical links from Master to all Endpoints.Clock distribution has been evaluated on a minimal lab setup as well as in a beam test and ithas been demonstrated that there is no significant clock drift between network nodes present. Asthe further steps for evaluation, long-term stability and precision clock drift tests are planned.Nonetheless, accuracy of time synchronisation in the current prototype design strongly dependson deterministic latency of downstream communication. Since data buffers are used in transceiverdatapaths to handle clock domain crossing, this is a bottleneck that will be addresses in future work.References[1] A. Rost, J. Adamczewski-Musch, T. Galatyuk, S. Linev, J. Pietraszko, M. Sapinski et al.,Performanceof the CVD Diamond Based Beam Quality Monitoring System in the HADES Experiment at GSI, inProceedings of International Particle Accelerator Conference, Melbourne, Australia, 2019doi:10.18429/JACoW-IPAC2019-WEPGW019[2] X. Gao, D. Emschermann, J. Lehnert, W.F.J. Müller, Throttling Studies for the CBM Self-triggeredReadout, in Proceedings of Topical Workshop on Electronics for Particle Physics, Santiago deCompostela, Spain, 2019 doi:10.22323/1.370.0085[3] X. Gao, D. Emschermann, J. Lehnert, W.F.J. Müller, Throttling strategies and optimization of thetrigger-less streaming DAQ system in the CBM experiment, Nuclear Instruments and Methods inPhysics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol.978, 2020, Art. 164442 doi:10.1016/j.nima.2020.164442[4] K. Chen, H. Chen, J. Huang, F. Lanni, S. Tang and W. Wu, A Generic High Bandwidth DataAcquisition Card for Physics Experiments, in IEEE Transactions on Instrumentation andMeasurement, vol. 69, no. 7, pp. 4569-4577, July 2020, doi:10.1109/TIM.2019.2947972[5] IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement andControl Systems, in IEEE Std 1588-2019 (Revision ofIEEE Std 1588-2008) , vol., no., pp.1-499, 16June 2020, doi:10.1109/IEEESTD.2020.9120376– 5 –

Prototype design of a timing and fast control system in the CBM experiment

https://publikationen.bibliothek.kit.edu/1000148486/149009311

Prototype design of a timing and fast control system in the CBM experiment

Abstract

Similar works

Full text

Available Versions

KITopen