Baby MIND is a new downstream muon range detector for the WGASCI experiment. This article discusses the distributed readout system and its timing requirements. The paper presents the design of the synchronization subsystem and the results of its test

Antonova, M.

Asfandiyarov, R.

Bayes, R.

Benoit, P.

Blondel, A.

Bogomilov, M.

Bross, A.

Cadoux, F.

Cervera, A.

Chikuma, N.

Dudarev, A.

Ekelof, T.

Favre, Y.

Fedotov, S.

Hallsjö, S.-P.

Ichikawa, A. K.

Izmaylov, A.

Karadzhov, Y.

Khabibullin, M.

Khotyantsev, A.

Kleymenova, A.

Koga, T.

Kostin, A.

Kudenko, Y.

Likhacheva, V.

Martinez, B.

Matev, R.

Medvedeva, M.

Mefodiev, A.

Minamino, A.

Mineev, O.

Mitev, G.

Nessi, M.

Nicola, L.

Noah, E.

Ovsiannikova, T.

Pais Da Silva, H.

Parsa, S.

Rayner, M.

Rolando, G.

Shaykhiev, A.

Simion, P.

Soler, P.

Suvorov, S.

Ten Kate, H.

Tsenov, R.

Vankova-Kirilova, G.

Yershov, N.

English

Enlighten

      Antonova, M. et al. (2017) Synchronization of the Distributed Readout Frontend Electronics of the Baby MIND Detector. In: 2017 XXVI International Scientific Conference Electronics (ET), Sozopol, Bulgaria, 13-15 Sep 2017, ISBN 9781538617533.     There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it.    http://eprints.gla.ac.uk/159441/              Deposited on: 21 March 2018                       Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Proc. XXVI International Scientific Conference Electronics - ET2017, September 13 - 15, 2017, Sozopol, Bulgaria    Synchronization of the distributed readout frontend electronics of the Baby MIND detector  M. Antonovaa, R. Asfandiyarovb, R. Bayesc, P. Benoitd, A. Blondelb, M. Bogomilove, A. Brossf, F. Cadouxb, A. Cerverag, N. Chikumah, A. Dudarevd, T. Ekelöfi, Y. Favreb, S. Fedotova, S-P. Hallsjöc, A.K. Ichikawaj, A. Izmaylova, Y. Karadzhovb, M. Khabibullina, A. Khotyantseva, A. Kleymenovaa, T. Kogah, A. Kostina, Y. Kudenkoa, V. Likhachevaa, B. Martinezb, R. Mateve, M. Medvedevaa, A. Mefodieva, A. Minaminok, O. Mineeva, G. Mitevl, M. Nessid, L. Nicolab, E. Noahb, T. Ovsiannikovaa, H. Pais Da Silvad, S. Parsab, M. Raynerb, G. Rolandod, A. Shaykhieva, P. Simioni, P. Solerc, S. Suvorova, R. Tsenove, H. Ten Kated, G. Vankova-Kirilovae and N. Yershova. aInstitute for Nuclear Research, Russian Academy of Sciences, Moscow, Russia bUniversity of Geneva, Section de Physique, DPNC, Geneva, Switzerland cUniversity of Glasgow, School of Physics and Astronomy, Glasgow, UK; dEuropean Organization for Nuclear Research, CERN, Geneva, Switzerland; eUniversity of Sofia, Department of Physics, Sofia, Bulgaria; fFermi National Accelerator Laboratory, Batavia, Illinois, USA; gIFIC (CSIC & University of Valencia), Valencia, Spain; hUniversity of Tokyo, Tokyo, Japan; iUppsala University, Uppsala, Sweden; jKyoto University, Kyoto, Japan; kYokohama National University, Yokohama, Japan; lInstitute for Nuclear Research and Nuclear Energy, Bulgarian Academy of Sciences, Sofia, Bulgaria e-mail: Yannick.Favre@unige.ch e-mail: georgi.mitev@cern.ch  Abstract – Baby MIND is a new downstream muon range detector for the WGASCI experiment. This article discusses the distributed readout system and its timing requirements. The paper presents the design of the synchronization subsystem and the results of its test. Keywords – WAGASCI, Baby MIND, neutrino, distributed readout, synchronization I. INTRODUCTION A. Detector overview  Baby MIND is a 70 ton modular magnetized iron detector developed and tested at CERN. It will be used as a downstream muon range detector in the WAGASCI experiment on the T2K neutrino beam line in Japan [1].  Fig. 1. Baby MIND detector at T9 beam-line at CERN.  It is composed of multiple magnet and scintillator modules [2]. The modular design allows for easy reconfiguration of the geometry of the detector and flexibility for transport and installation. B. Readout electronics overview  Figure 1 shows the detector under test at CERN beam-line, with the readout electronics installed in 8 mini-crates on top of the detector. Each mini-crate can accommodate up to 7 Front-End Boards (FEB) (fig. 2). Each FEB can read-out up to 96 detector channels. The data is transferred to the Data Acquisition System (DAQ) via USB 3 interface. The FEBs can work either in standalone mode, in which every FEB is connected to the readout computer via a USB3 connection, or in time division multiplexing (TDM) mode in which all FEBs of a mini-crate are chained and the data is passed to a single USB3 master FEB which sends it to the DAQ. This allows sharing the available data bandwidth and reducing the amount of required cables. There also is a possibility, while in TDM mode, to assign the full USB bandwidth to any selected FEB in the chain.  Fig. 2. Frontend electronics mini-crate. C. Synchronization requirements  In order to achieve a good timing resolution the FEBs run on a 2.5 ns internal clock, synchronized to a common 100 MHz clock. That puts some very tight requirements to the clock and synchronization distribution subsystem. Delays of the global clock delivered to the separate FEBs have to be kept within few hundred picoseconds accounting for all possible delay sources like the difference in the link lengths, chip-to-chip delay difference and signal jitter.  Beam Line & ext synchroSpillControl roomPCDAQPC #0PC #1PC #2Reset *Clock *Detector10GEthernetSwitchFEBFEBFEB 96-chUSB3 ...#1#0#5#2#3 USB3 ports#6 FEB per mini-crate#3PCs  & 1 Master clock board#3 mini-crates* optional signalsSYNCHRO : 1 RJ45/ 4 pairsCLOCK (50/100MHz)GTRIG (100KHz)SPILL (ST/END, NB#)RESETFS (10KHz-100KHz frame sync)USB : READOUT (EP1IN, 2Gb/s)CMD (EP2OUT, 1Mb/s)ANS (EP2IN, 1Mb/s)CHAIN +SYNCHRO : on backplaneREADOUT (1Gb/s on FS)CMD (1Mb/s) + ANS (1Mb/s)CLOCK (50/100MHz)GTRIG (100KHz)SPILL (ST/END, NB#)RESETFS (10KHz-100KHz frame sync)5m max.Master Clock(7 RJ45) Back-planeMini-crate #0Mini-crate #2FEBFEBFEB 96-ch...#1#0#5#2Mini-crate #0Mini-crate #2FEBFEBFEB 96-ch...#1#0#5#2Mini-crate #0#3 mini-crates#1 mini-crate#3 USB3 ports#1 USB3 portBack-planeBack-planeTOTAL = 576 Vert. CH + 3456 Hor. CH = 4032 CH42 FEB 96-ch7 MINI-CRATES3 PC1 Master Clock board Fig. 3. Baby MIND readout block diagram.  D. Synchronization subsystem overview  The block diagram of the Baby MIND readout system is shown on fig. 3. The synchronization subsystem (shown in green lines) takes input signals from the beam-line and combines them in a digital synchronization signal (SYNC). It also produces the global detector clock (CLK) and eventually synchronizes it to an external experiment clock. Both signals are then distributed to the readout FEBs.  The distribution path has a two-level fan-out structure. The first level contains the master clock board (MCB) which distributes the signals to the FEB mini-crates over 5m long CAT6 shielded Ethernet cables. The second level consists of the mini-crate backplane which distributes the CLK and SYNC to the FEB slots. It also provides a couple of gigabit links from one slot to the next which pass the data in the TDM mode readout chain. II. SYNCHRONIZATION SUBSYSTEM IMPLEMENTATION  At the time of writing only the backplane has been developed and tested. The master clock board is still under development. In order to test and qualify the detector a FEB emulates the signals of a single output of the MCB. A separate fan-out board was developed to distribute the signals to the mini-crates. A. Mini-crate backplane block diagram  Fig. 4. Baby MIND mini-crate backplane block diagram.  The mini-crate backplane receives differential CLK and SYNC signals from the MCB (fig. 4). A second spare data SYNC Fan-outAUX Fan-out CLK Fan-out Slot 0Slot 1 Slot 6 Slot 5 … Power monitorEnvironment monitorControllerPower gatings-LinkCLKRJ45 MCB InSYNCAUX24V 3.3V Power CLKSYNCRJ45 MCB OutAddress SMA Global trigger input2xGb link 2xGb link 2xGb linkline AUX running from the MCB through the fan-out to the FEBs has been added to facilitate flexibility and future upgrades. The 3 differential signals are distributed to all 7 FEB slots. The last remaining pair of the CAT6 cable is dedicated to a serial link for the back-plane controller, used to monitor slow-control parameters on the back-plane and control individual slot power gating.  Between every two adjacent slots there is a pair of serial links running at 1 Gb/s (with a possible future upgrade to 2 or 3 Gb/s), which allow to use the mini-crate FEBs in TDM mode.  Slot 0 is connected to SMA connectors on the back-plane, which allow the FEB in that slot to read external trigger and reset signals. The slot is also connected to a separate RJ45 connector that allows the FEB to output MCB emulation CLK and SYNC signals. B. Test setup and test fan-out PCBs.  Fig. 5. Baby MIND qualification fan-out board being tested.  Due to the unavailability of the master clock board for the qualification tests at CERN a stop-gap solution was developed. It uses an additional mini-crate with a FEB running in MCB emulation mode in slot 0. The dedicated MCB emulation output is fed back into the MCB input of the backplane. Two 4-output fan-out PCBs are connected to the back-plane and deliver the synchronization signals to the read-out mini-crates installed on Baby MIND.  While this solution works well, it has a few disadvantages to the real MCB. Due to the chips used it has worse pulse rise and fall times, jitter and output clock skew. It has no means to compensate a difference in the delays of the synchronization channels in contrast to the MCB. It is also harder to interface to the beamline signals. III. TEST RESULTS  This paper presents the timing parameters achieved by the test setup used to qualify the Baby MIND detector. The final setup to be used in Japan is expected to outperform it.  The measurements have been taken using an SRS CG635 precision clock generator (1ps random jitter, 80ps rise/fall time) and a 20 GHz LeCroy WaveMaster 820Zi oscilloscope. A. Backplane parameters and oscillograms  Figure 6 shows the difference in delays of a 100MHz signal passing through the back-plane. Rise and fall times around 60ps were observed.  The scope was found to show a difference of 42ps between the probes on channel 1 and two when connected to the same signal source (solder-in tip). This value was used to correct the measured clock delays. The values presented are obtained after statistical processing of 1000 samples. An example of the delay distribution can be seen on the lower plot on fig. 6.  Fig. 6. Backplane clock 100MHz signal skew.  Table 1 lists the minimum and maximum values of the clock delay variation between slot 0 and the other slots. It lists the median of the distribution which is taken as the delay value and also the difference between the minimum and maximum delay values.  TABLE 1. SLOT-TO-SLOT CLOCK DELAY Slot delay min (ps) max (ps) median (ps) max – min (ps) delay (ps) CLK_0-to-1 -3.4 3.8 -0.374 7.2 -0.374 CLK_0-to-2 4.5 10.4 7.45 5.9 7.45 CLK_0-to-3 10.2 16 12.96 5.8 12.96 CLK_0-to-4 6.4 12.4 9.494 6 9.494 CLK_0-to-5 2.1 7.9 4.647 5.8 4.647 CLK_0-to-6 -5.2 1.8 -1.979 7 -1.979   Fig. 7. Input CLK to SYNC delay after in-scope compensation.  To measure the delay between the CLK and SYNC signals we split the signal from the signal generator. Due to the difference in the used cable length an input delay of approximately 2 ns was observed. It was compensated in the scope and figure 7 shows the remaining measured input delay difference of 20 ps. This value was used to correct the measurements presented in table 2.    Figure 8 and table 2 show the CLK-to-SYNC delay for the slots of the back-plane. The average value of the delay is around 740ps. It is mostly due to the different driver chips (and delays) used for the fan-out of the CLK and SYNC/AUX, a choice driven by input level compatibility requirements. The SYNC/AUX signals are driven on the rising edge of the clock and sampled on the falling edge. To read the SYNC data properly the delay should not exceed Tclk/2 or 5 ns. The measured SYNC setup delay value is well within that margin.   Fig. 8. CLK to SYNC delay on the backplane.  TABLE 2. CLK-TO-SYNC DELAY FOR EVERY SLOT Slot delay min (ps) max (ps) median (ps) max – min (ps) delay (ps) CLK-to-SYNC_Input -1 41 20.66 42 20.66 CLK_0-to-SYNC_0 709.2 738.7 725.5 29.5 746.2 CLK_0-to-SYNC_1 715.2 737.2 725.3 22 745.9 CLK_0-to-SYNC_2 700 722.5 710.6 22.5 731.2 CLK_0-to-SYNC_3 709.4 733.4 720.2 24 740.9 CLK_0-to-SYNC_4 708.4 732.9 720.3 24.5 740.9 CLK_0-to-SYNC_5 699.2 724.2 711.2 25 731.9 CLK_0-to-SYNC_6 717.9 744.4 730.3 26.5 750.9 B. Test fan-out setup parameters TABLE 3. TEST FAN-OUT OUTPUT-TO-OUTPUT CLOCK DELAY Output delay min (ps) max (ps) median (ps) max - min (ps) delay (ps) CLK 0-to-0 -59.1 -17.1 -36.85 42 0 CLK 0-to-1 -98 -52 -77.13 46 -40.28 CLK 0-to-2 -70 -19 -44.71 51 -7.86 CLK 0-to-3 -89 -30 -58.63 59 -21.78 CLK 0-to-4 -83 -23 -48.66 60 -11.81 CLK 0-to-5 -105 -47 -70.58 58 -33.73 CLK 0-to-6 -70 -19 -42.91 51 -6.06 CLK 0-to-7 -87 -27 -58.3 60 -21.45  The MCB fan-out emulation setup delays were measured in a way very similar to the back-plane delay measurement (see fig. 5). As previously mentioned the setup consists of a backplane and a pair of 4-outut active fan-out boards. Table 3 lists the aggregate setup clock delays that were measured on the outputs. A maximum delay difference of 41ps has been observed among the outputs. Similar delays have been measured for the SYNC. C. Beam-line synchronization chain delay parameters  The beamline delay parameters are a superposition of MCB fan-out, cabling and back-plane delay parameters. Special measures have been taken to ensure equal cable lengths. The total FEB-to-FEB clock delay difference can be calculated as the sum of the maximum delay differences of the fan-out and the back-plane and equals: ΔTdclk = (7.2ps + 40.3ps) ≈ 50ps  (1)   Similarly for the SYNC delay difference: ΔTdsync = (29.5ps + 40.3ps) ≈ 70ps (2) IV. CONCLUSION  The test setup used to qualify the Baby MIND detector fulfills the synchronization timing requirements.  The resulting FEB-to-FEB clock delay difference is around 50ps.  The resulting FEB-to-FEB SYNC delay difference is around 70ps.  On the negative side the test setup isn’t well suited to interface to beam-line signals. Also it is unable to compensate for cable delay difference, for example in the case when a cable has to be replaced and the length of the new cable doesn’t match exactly the old one. These issues will be addressed with the design and development of the master clock board.  This study doesn’t take into account FEB onboard delay differences - signal traces and FPGA internal signal distribution. When the MCB development is complete the whole synchronization subsystem should be re-evaluated.  The synchronization beam-line test setup fulfilled its role to allow test and qualification of the Baby Mind detector at CERN in a correct and timely manner. ACKNOWLEDGEMENTS This project has received funding from the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 654168. REFERENCES [1] S.Parsa et al. Baby MIND Experiment Construction Status, arXiv:1704.08917 [physics.ins-det]. [2] A. Mefodiev et al. Baby MIND: A magnetized segmented neutrino detector for the WAGASCI experiment, arXiv:1705.10406 [physics.ins-det].  

Synchronization of the Distributed Readout Frontend Electronics of the Baby MIND Detector

Antonova, M

Asfandiyarov, R

Bayes, R

Benoit, P

Blondel, A

Bogomilov, M

Bross, A

Cadoux, F

Cervera, A

Chikuma, N

Dudarev, A

Ekelöf, T

Favre, Y

Fedotov, S

Hallsjö, S-P

Ichikawa, A K

Izmaylov, A

Karadzhov, Y

Khabibullin, M

Khotyantsev, A

Kleymenova, A

Koga, T

Kostin, A

Kudenko, Y

Likhacheva, V

Martinez, B

Matev, R

Medvedeva, M

Mefodiev, A

Minamino, A

Mineev, O

Mitev, G

Nessi, M

Nicola, L

Noah, E

Ovsiannikova, T

Pais Da Silva, H

Parsa, S

Rayner, M

Rolando, G

Shaykhiev, A

Simion, P

Soler, P

Suvorov, S

Tsenov, R

Ten Kate, H

Vankova-Kirilova, G

Yershov, N

CERN Document Server

Synchronization of the distributed readout frontend electronics of the Baby MIND detector

Synchronization of the Distributed Readout Frontend Electronics of the Baby MIND Detector

Abstract

Similar works

Full text

Available Versions

Enlighten

CERN Document Server

Crossref