The DAQ and Timing Hub is an ATCA hub board designed for the Phase-2 upgrade of the CMS experiment. In addition to providing high-speed Ethernet connectivity to all back-end boards, it forms the bridge between the sub-detector electronics and the central DAQ, timing, and trigger control systems. One important requirement is the distribution of several high-precision, phasestable, and LHC-synchronous clock signals for use by the timing detectors. The current paper presents first measurements performed on the initial prototype, with a focus on clock quality. It is demonstrated that the current design provides adequate clock quality to satisfy the requirements of the Phase-2 CMS timing detectors

Behrens, U.

Blažek, R.

Branson, J.

Brummer, P.

Cittolin, S.

Darlea, G.-L.

Deldicque, C.

Dobson, M.

Doualot, N.

Fulcher, J. R.

Gigi, D.

Gladki, M.

Glege, F.

Gomez-Ceballos, G.

Hansen, M.

Hegeman, J.

Jevtić, D.

Li, W.

Mecionis, A.

Meijers, F.

Meschi, E.

Mommsen, R. K.

Mor, K.

Morovic, S.

Orsini, L.

O’Dell, V.

Papakrivopoulos, I.

Paus, C.

Petrucci, A.

Pieri, M.

Rabady, D.

Raychino, K.

Rodriguez-Garcia, A.

Rácz, A.

Sakulin, H.

Schwick, C.

Silva-Gomes, D. da

Soursos, P.

Stahl, A.

Stankevicius, M.

Suthakar, U.

Troska, J.

Vazquez-Velez, C.

Zahid, A.

Zejdl, P.

Šimelevičius, D.

English

The DAQ and Timing Hub is an ATCA hub board designed for the Phase-2 upgrade of the CMS experiment. In addition to providing high-speed Ethernet connectivity to all back-end boards, it forms the bridge between the sub-detector electronics and the central DAQ, timing, and trigger control systems. One important requirement is the distribution of several high-precision, phase-stable, and LHC synchronous clock signals for use by the timing detectors. The current paper presents first measurements performed on the initial prototype, with a focus on clock quality. It is demonstrated that the current design provides adequate clock quality to satisfy the requirements of the Phase-2 CMS timing detectors

Hegeman, Jeroen

Blažek, Radovan

Behrens, Ulf

Branson, James

Brummer, Philipp

Cittolin, Sergio

Silva-Gomes, Diego Da

Darle, Georgiana-Lavinia

Deldicque, Christian

Dobson, Marc

Doualot, Nicolas

Fulcher, Jonathan Richard

Gigi, Dominique

Gladki, Maciej

Glege, Frank

Gomez-Ceballos, Guillelmo

Hansen, Magnus

Jevtić, Dragana

Li, Wei

Mecionis, Audrius

Meijers, Frans

Meschi, Emilio

Mommsen, Remigius K

Mor, Keyshav

Morovic, Srecko

O'Dell, Vivian

Orsini, Luciano

Papakrivopoulos, Ioannis

Paus, Christoph

Petrucci, Andrea

Pieri, Marco

Rabady, Dinyar

Raychino, Kolyo

Racz, Attila

Rodriguez-Garcia, Alvaro

Sakulin, Hannes

Schwick, Christoph

ŠimelevičIus, Dainius

Soursos, Panagiotis

Stahl, Andre

Stankevicius, Mantas

Suthakar, Uthayanath

Troska, Jan

Vazquez-Velez, Cristina

Zahid, Awais

Zejdl, Petr

CERN Document Server

First measurements with the CMS DAQ and Timing Hub prototype-1

Blažek, Radovan,

Behrens, Ulf,

Branson, James,

Brummer, Philipp,

Cittolin, Sergio,

Da Silva-Gomes, Diego,

Darlea, Georgiana-Lavinia,

Deldicque, Christian,

Dobson, Marc,

Doualot, Nicolas,

Fulcher, Jonathan Richard,

Gigi, Dominique,

Gladki, Maciej,

Glege, Frank,

Gomez-Ceballos, Guillelmo,

Hansen, Magnus,

Hegeman, Jeroen,

Jevtič, Dragana,

Li, Wei,

Mečionis, Audrius,

Meijers, Frans,

Meschi, Emilio,

Mommsen, Remigius K.,

Mor, Keyshav,

Morovic, Srecko,

O’Dell, Vivian,

Orsini, Luciano,

Papakrivopoulos, Ioannis,

Paus, Christoph,

Petrucci, Andrea,

Pieri, Marco,

Rabady, Dinyar,

Raychino, Kolyo,

Rácz, Attila,

Rodriguez-Garcia, Alvaro,

Sakulin, Hannes,

Schwick, Christoph,

Šimelevičius, Dainius,

Soursos, Panagiotis,

Stahl, Andre,

Stankevičius, Mantas,

Suthakar, Uthayanath,

Troska, Jan,

Vazquez-Velez, Cristina,

Zahid, Awais,

Zejdl, Petr,

Vilnius University Institutional Repository

First measurements with the CMS DAQ and Timing Hub prototype-1 /

KITopen

PoS(TWEPP2019)111First measurements with the CMS DAQ and TimingHub prototype-1Radovan Blažek1, Ulf Behrens6, James Branson2, Philipp Brummer1,c,Sergio Cittolin2, Diego Da Silva-Gomes3,b, Georgiana-Lavinia Darlea4,Christian Deldicque1, Marc Dobson1, Nicolas Doualot3,b,Jonathan Richard Fulcher1, Dominique Gigi1, Maciej Gladki1, Frank Glege1,Guillelmo Gomez-Ceballos4, Magnus Hansen1, Jeroen Hegeman∗, Dragana Jevtić1,Wei Li6, Audrius Mecionis 3,a, Frans Meijers1, Emilio Meschi1,Remigius K. Mommsen3, Keyshav Mor1, Srecko Morovic2, Vivian O’Dell3,Luciano Orsini1, Ioannis Papakrivopoulos5,b, Christoph Paus4, Andrea Petrucci2,Marco Pieri2, Dinyar Rabady1, Kolyo Raychino1, Attila Rácz1,Alvaro Rodriguez-Garcia1, Hannes Sakulin1, Christoph Schwick1,Dainius Šimelevičius7,b, Panagiotis Soursos1, Andre Stahl6,Mantas Stankevicius 3,a, Uthayanath Suthakar1, Jan Troska1,Cristina Vazquez-Velez1, Awais Zahid1, Petr Zejdl3,b1 CERN, Geneva, Switzerland2 University of California, San Diego, San Diego, California, USA3 FNAL, Chicago, Illinois, USA4 Massachusetts Institute of Technology, Cambridge, Massachusetts, USA5 Technical University of Athens, Athens, Greece6 Rice University, Houston, Texas, USA7 Vilnius University, Vilnius, Lithuania8 American University of Beirut, Beirut, Libanona Also at Vilnius University, Vilnius, Lithuaniab Also at CERN, Geneva, Switzerlandc Also at Karlsruhe Institute of Technology, Karlsruhe, GermanyE-mail: jeroen.hegeman@cern.chThis work was supported in part by the DOE and NSF (USA).The DAQ and Timing Hub is an ATCA hub board designed for the Phase-2 upgrade of the CMSexperiment. In addition to providing high-speed Ethernet connectivity to all back-end boards, itforms the bridge between the sub-detector electronics and the central DAQ, timing, and triggercontrol systems. One important requirement is the distribution of several high-precision, phase-stable, and LHC-synchronous clock signals for use by the timing detectors.The current paper presents first measurements performed on the initial prototype, with a focus onclock quality. It is demonstrated that the current design provides adequate clock quality to satisfythe requirements of the Phase-2 CMS timing detectors.Topical Workshop on Electronics for Particle Physics TWEPP20192-6 September 2019Santiago de Compostela - Spain∗Speaker.c© Copyright owned by the author(s) under the terms of the Creative CommonsAttribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). https://pos.sissa.it/PoS(TWEPP2019)111First measurements with the CMS DAQ and Timing Hub Jeroen Hegeman1. IntroductionThe CMS detector will undergo a major upgrade for Phase-2 of the LHC program: the HiLumiLHC, starting around 2026. The upgraded CMS detector will be read out at an unprecedented datarate of up to 50 Tb/s with an event rate of 750 kHz, selected by the level-1 hardware trigger, andan average event size of 7.5 MB. The Phase-2 CMS back-end electronics will be based on theATCA standard, with node boards receiving the detector data from the front-ends via optical links.An ATCA hub board, the DAQ and Timing Hub (DTH), will provide the interface between theback-end nodes and the central Trigger, Timing, and DAQ systems. This paper presents the firstmeasurements performed on the initial prototype production, with a focus on clock quality. Moredetail about the Phase-2 CMS Trigger-DAQ design can be found in [1, 2].The CMS DAQ and Timing Hub will connect to all back-end node boards. It is responsiblefor the distribution of clock, trigger, and fast-control data from the central trigger control systemto all back-end electronics and the handling of throttling signals in the other direction. The DAQinterface for the event data uses point-to-point optical links connected to back-end node boardsand runs a custom lossless ‘SlinkRocket’ protocol. The event data are aggregated in the DTH andtransmitted via standard commercial network (with links of 100 Gb/s or higher) and protocol (suchas TCP/IP) to event building computer nodes at the surface.The introduction of timing detectors for Phase-2 CMS, aiming for a 30 ps precision on individ-ual detector hits, strongly tightens the requirements on the clock and timing information distributedthroughout the experiment, with different sub-detectors introducing different requirements.2. The DTH development programAt TWEPP 2018 the design of the first DTH prototype (P1) was presented, together with someof the design challenges encountered [3]. Other contributions summarised the phase noise studiesperformed as part of the component selection for the DTH.The final DTH needs to fulfill three main tasks: the distribution of clock and timing signals toand from the back-end electronics, the aggregation and transmission of event data from the back-ends to the commercial data-to-surface network, and the management of the 1 Gbit/s Ethernetconnectivity required for each of the node slots. The P1 is designed to demonstrate the DAQand timing functions, whereas an independent prototype has been designed to develop a managedEthernet switch. The different development lines will later be merged into the final DTH design.The DTH P1 board features an FPGA (KU15P) with ancillary components for clock recoveryand jitter-cleaning, and serial links to the back-plane implementing the clock, timing, trigger andthrottling functions. A second FPGA (KU15P) with mid-board optics (FireFly) connecting to theback-end boards, and QFSP+ cages connecting to the standard commercial network, implementsthe DAQ event flow functions.On the DAQ side, proof-of-principle has been shown of the sharing of multiple TCP streamsthrough a single 100 GbE link. Optimisation in order to achieve the required throughput is currentlyunder way, as well as studies with high bandwidth in-FPGA memory.1PoS(TWEPP2019)111First measurements with the CMS DAQ and Timing Hub Jeroen Hegeman3. The role of the DTH in the clock distributionThe baseline clock distribution scheme for the Phase-2 CMS upgrade foresees the encodingof the LHC bunch clock in a high-speed serial stream (at ≈ 10Gbit/s) from the upgraded TriggerControl and Distribution System (TCDS) to the DAQ and Timing Hub board present in each sub-system back-end crate. The DTH recovers and cleans this embedded clock, and then distributestwo phase-locked clocks over the backplane to all node slots: the 40 MHz LHC bunch clock forbeam-synchronous logic, and a precision 320 MHz clock for use as high-speed transceiver (MGT)reference. Timing and synchronization signals are extracted from the incoming serial stream anddistributed on dedicated backplane lines to all node slots. The sub-detector back-end electronics inturn distribute the sampling clock embedded in an lpGBT or GBT serial stream to the front-ends.This scheme requires the DTH be capable of 1. recovering an LHC bunch clock and a higher-speedMGT reference clock of sufficient quality from the incoming serial stream from the TCDS master,and 2. reliably reproducing a fixed phase relationship between these clocks and the TCDS master.Figure 1 shows a simplified overview of the DTH clock paths. The ≈ 10Gbit/s serial streamfrom the TCDS master is received by an FPGA MGT via a standard SFP. The recovered 320 MHzclock is passed into the FPGA fabric, where it is divided down to recreate the 40 MHz LHC bunchclock. This bunch clock is output through the global clock network and used as input to the masterPLL/jitter attenuator, from where all on-board and backplane clocks are generated. In addition therecovered 320 MHz clock can be output straight from the MGT and, after a pass through a jittercleaner, used as MGT reference for all backplane TCDS links.Figure 1: Overview of the clock distribution paths on the DTH P1. In the baseline scheme the DTH receivesa high-speed serial stream from the TCDS master, containing both clock and synchronization information.The clock is recovered from the serial stream, divided down to the LHC bunch crossing frequency, andcleaned by the on-board master PLL. For test and evaluation purposes the P1 DTH prototype is also equippedwith a direct clock input to the master PLL, and several jitter attenuators have been equipped with differentcrystal and/or TCXO references.4. Backplane clock qualityFor the requirements of the Phase-2 CMS timing detectors the clock quality is characterized2PoS(TWEPP2019)111First measurements with the CMS DAQ and Timing Hub Jeroen Hegemanin terms of phase noise and the corresponding RMS jitter. High-frequency phase noise can becleaned using appropriate jitter attenuators, but low-frequency wander cannot. Therefore it is ofutmost importance that the design and implementation of the clock distribution do not introduceany significant low-frequency noise.1 10 100 1 k 10 k 100 k 1 M 10 M 100 Mfrequency offset [Hz]160140120100806040phase noise [dBc/Hz]Slot 3Slot 4Slot 5Slot 6Slot 7Slot 8Slot 9Slot 10Slot 11Slot 12Slot 13Slot 141 10 100 1 k 10 k 100 k 1 M 10 M 100 Mfrequency offset [Hz]160140120100806040phase noise [dBc/Hz]Slot 3Slot 4Slot 5Slot 6Slot 7Slot 8Slot 9Slot 10Slot 11Slot 12Slot 13Slot 14Figure 2: Phase noise curves for the 40 MHz (left) and 320 MHz (right) backplane clocks in all node slotsof the shelf. The former show minor slot-to-slot variations due to backplane attenuation affecting the signalslew rate. The latter clearly form two groups, corresponding to the two jitter cleaners disciplined by a TCXO(slots 3-8) and a crystal (slots 9-14).As part of the performance characterization of the DTH P1, a clean input clock was sourcedto the auxiliary input of the master PLL, and the backplane clock signals were inspected with thehelp of a commercial break-out board. Figure 2 shows the phase noise spectra of the 40 MHz and320 MHz backplane clocks. The high-frequency plateaus of the 40 MHz clock spectra show smallslot-to-slot variations that can be traced down to frequency-dependent attenuation on the backplaneaffecting the signal slew rate as a function of distance. The results from the 320 MHz clocks clearlyseparate into two distinct groups corresponding to two jitter cleaners on the board: one disciplinedby a TCXO and one disciplined by a plain crystal. (See also Fig. 1.) A comparison between thesetwo groups shows two design/implementation features. The additional spurs between 200 Hz and2 kHz in the TCXO-driven case seem to be related to imperfections in the power and PCB designof the DTH. The next iteration of the design aims to address these imperfections. The phase noiseplateau from 10 kHz to 1 MHz is a feature of the chosen TCXO. The behavior at low frequencies isin all cases determined by the phase noise contents of the input clock. The worst case jitter for the40 MHz (320 MHz) clocks shows an RMS of 4.9 ps (3.7 ps) when integrated over [1Hz,20MHz].Given the relatively poor performance of the TCXO-based part of the design, combined withthe good results obtained with the crystal-driven cleaner, the final DTH design will most likely relyon crystals to discipline all jitter attenuators.5. Clock distribution proof-of-principleA first two-DTH setup was used to demonstrate the board-to-board clock distribution. A‘sender’ DTH generates a high-speed PRBS stream from an auxiliary clock, and a ‘receiver’ DTHrecovers 40 MHz and 320 MHz clocks from this PRBS stream. Figure 3 shows a phase-noise3PoS(TWEPP2019)111First measurements with the CMS DAQ and Timing Hub Jeroen Hegemancomparison between the bunch clock at the output of the master PLL on the ‘sender’, and the sameclock signal on the ‘receiver’, as well as the spectra of two intermediate 320 MHz clocks. Thismeasurement demonstrates that 1. the DTH P1 does not introduce any significant low-frequencyphase noise, and that 2. the quality of the LHC bunch clock is almost unaffected by the encodingand recovery procedure.1 10 100 1 k 10 k 100 k 1 M 10 M 100 Mfrequency offset [Hz]16014012010080604020phase noise [dBc/Hz]sender, 40 MHz master clock (std = 2.1 ps)receiver, 320 MHz recovered clock (std = 3.6 ps)receiver, 320 MHz cleaned clock (std = 4.1 ps)receiver, 40 Mhz master clock (std = 3.4 ps)Figure 3: Results of the first dual-DTH measurements. One DTH was configured as ‘sender’, using anauxiliary 40 MHz clock to distribute a 10.24 MHz PRBS31 stream to a second ‘receiver’ DTH. This secondDTH recovered the 40 MHz and 320 MHz clocks from the incoming serial stream. The numbers in thelegend represent the RMS jitter as integrated over [1Hz,20MHz]. These results show that the clock qualityis not significantly affected by the serial encoding/decoding.6. Conclusion and outlookThe development plan of the Phase-2 CMS DAQ and Timing Hub is well under way. Allmeasurements and verifications performed so far with the first prototype clearly indicate that thehardware design meets the requirements for high-quality clock distribution imposed by the CMStiming detectors. The next step is to demonstrate that this still holds true when using the TCDS2protocol, and that in addition a fixed phase of all distributed clocks can be guaranteed even acrossresets and power-cycles.References[1] C. Collaboration, The Phase-2 Upgrade of the CMS DAQ Interim Technical Design Report, Tech. Rep.CERN-LHCC-2017-014. CMS-TDR-018, CERN, Geneva, Sep, 2017.[2] C. Collaboration, The Phase-2 Upgrade of the CMS L1 Trigger Interim Technical Design Report, Tech.Rep. CERN-LHCC-2017-013. CMS-TDR-017, CERN, Geneva, Sep, 2017.[3] J.-m. O. Andre, U. Behrens, A. Bocci, J. Branson, S. Cittolin, D. Da Silva Gomes et al., Design anddevelopment of the DAQ and Timing Hub for CMS Phase-2, Tech. Rep. CMS-CR-2018-307, CERN,Geneva, Oct, 2018.4

First measurements with the CMS DAQ and timing hub prototype-1

Crossref

https://publikationen.bibliothek.kit.edu/1000121405/83200175

First measurements with the CMS DAQ and timing hub prototype-1

Abstract

Similar works

Full text

Available Versions

CERN Document Server

Vilnius University Institutional Repository

KITopen

Crossref