Caselle, M.

Chilingaryan, S.

Dritschler, T.

Kopmann, A.

Rota, L.

Stevanovic, U.

Weber, M.

KITopen

Matter and  Technologies High frequency technologies for high-data rate DAQ systems M. Caselle, S. Cilingaryan, T. Dritschler,  A. Kopmann, L. Rota, U. Stevanovic, M. Weber Matter and Technologies Kickoff Meeting. 24-26 February 2015  DESY Hamburg Michele Caselle High frequency technologies for high-data rate DAQ systems ARD DTS Pico / Femto second  Beam diagnostic, detectors and DAQ system   q  Introduction to ANKA terahertz Coherent Synchrotron Radiation (CSR) q  Cryogenic terahertz detectors technologies q  High-bandwidth front-end  q  Picosecond pulse sampling “KAPTURE” system and beam test results q  New DMA-PCIe Readout compatible with GPU Direct access q  Conclusions Outline Michele Caselle High frequency technologies for high-data rate DAQ systems Terahertz Coherent Synchrotron Radiation at ANKA v  Circumference:  110.4 m v  RF-system:  500 MHz v  Harmonic number: 184 Normal mode Low alpha mode Reference: A.-S. Müller, et al. Observation of Coherent THz Radiation from the ANKA and MLS Storage Rings with a Hot Electron Bolometer. (TU5RFP027), 2009. 23rd Particle Accelerator Conference PAC09 Vancouver, Canada. Storage ring Beamlines ANKA is the Synchrotron Radiation Facility at the Karlsruhe Institute of Technology (KIT) buckets Terahertz 19th IEEE- Real Time Conference, 26-30 May 2014. Nara - Japan . M. Caselle Michele Caselle High frequency technologies for high-data rate DAQ systems Q u a s i - O p t i c a l Broadband Detector (Schottky diodes) H o t - E l e c t r o n -Bolometer (NbN) YBCO-Detector • Response time < 165 ps • Liquid He cooling • Developed at DLR • Response time < 15 ps • Liquid N2 cooling • Developed at KIT-IMS • Response time < 200 ps • No cooling required • Commercially available  (ACST, VDI) [1] A.D. Semenov, et al., IEEE Transactions on Microwave Theory and Techniques 55 (2007) 239  [2] P. Thoma, J. Raasch, et al., IEEE Trans. Appl. Supercond., Vol. 23, No 3, pp2400206, June 2013 [3] A. Semenov, et al., IEEE Electron Device Letters 31, (674) 2010 [1] [2] [3] Ultra-fast THz Detectors To detect and study of the emission characteristics of CSR in the THz range à  high time accuracy detector, spectrum of hundred GHz -> Terahertz Michele Caselle High frequency technologies for high-data rate DAQ systems Picosecond pulse sampling requirements YBCO	  Det.	  LNA	  Wideband	  	  Low	  Noise	  Ampliﬁer	  	  Coherent THz radiation Terahertz synchrotron  radiation 1.   DAQ for THz detectors: YBCO, Schottky Diodes, HEB, ..    2.   Pulse repetition rate: 500 MHz 3.   Continuous acquisition for long observation time: seconds, minutes…   4.   Wideband circuitries, bandwidth: DC-60GHz Requirements: Pulse measured with a real-time oscilloscope (bandwidth 60GHz) Pulse ~ Michele Caselle High frequency technologies for high-data rate DAQ systems Sampling points YBCO	  Det.	  LNA	  Wideband	  	  Low	  Noise	  Ampliﬁer	  	  Coherent THz radiation 1.    Sampling: each pulse sampled with 4 samples by KAPTURE system , minimum sampling time of 3 ps. (1) (2)  pulse	  sampling	  KAPTURE Terahertz synchrotron  radiation KAPTURE system (1) Pulse with repetition rate 500 MHz S1 S2 S3 S4 Picosecond pulse sampling requirements Michele Caselle High frequency technologies for high-data rate DAQ systems Sampling points 1.    Sampling: each pulse sampled with 4 samples by KAPTURE system , minimum sampling time of 3 ps. 2.    Data transfer: digital samples transferred to high-end GPU (Graphics Processing Units) by a PCIe-DMA architecture (1) (2)  (4) High throughput readout electronics pulse	  sampling	   FPGA	  PCIe DMA/PCIe KAPTURE DDR3	  32 Gb/s Continuously data streaming  GPU-DAQ Terahertz synchrotron  radiation (2) (1) Pulse with repetition rate 500 MHz S1 S2 S3 S4 KAPTURE system YBCO	  Det.	  LNA	  Wideband	  	  Low	  Noise	  Ampliﬁer	  	  Coherent THz radiation Picosecond pulse sampling requirements Michele Caselle High frequency technologies for high-data rate DAQ systems Sampling points YBCO	  Det.	  LNA	  Wideband	  	  Low	  Noise	  Ampliﬁer	  	  Coherent THz radiation 1.    Sampling: each pulse sampled with 4 samples by KAPTURE system , minimum sampling time of 3 ps. 2.    Data transfer: digital samples transferred to high-end GPU (Graphics Processing Units) by a PCIe-DMA architecture  3.    Real-time GPU data elaboration: pulses reconstruct, amplitude and peaking time respectively with “mV” “picosecond” accuracy are evaluated (1) (2)  (4) High throughput readout electronics pulse	  sampling	   FPGA	  PCIe DMA/PCIe KAPTURE DDR3	  32 Gb/s Continuously data streaming  GPU-DAQ Terahertz synchrotron  radiation (3) (2) (1) Pulse with repetition rate 500 MHz S1 S2 S3 S4 KAPTURE system Picosecond pulse sampling requirements Michele Caselle High frequency technologies for high-data rate DAQ systems KAPTURE - system LNA Wideband power divider KAPTURE board High throughput readout board PCIe link Pulse input DC-DC power supply Michele Caselle High frequency technologies for high-data rate DAQ systems Low Noise Amplifier (LNA) IN OUT IN OUT Gain control V Bias RF IN RF OUT High-freq. V-connectors Wideband Low Noise Amplifier q  MCM-D technology with new PCB materials for Microwave/RF design q  MMIC based on GaAs technology  q  Bandwidth DC- 55 GHz and Flat gain : 12 dB up to 48 GHz  q  Under development à new LNA with bandwidth: 0.5 – 65 GHz,  gain = 10 dB Michele Caselle High frequency technologies for high-data rate DAQ systems Wide-Band power divider Direct transfer (S21) Reflection (S11) Post-layout  simulation KIT power splitter 1:4 (S parameters) OUT1 OUT2 OUT3 OUT4 IN Power divider 1:4 outputs -­‐2,00E-­‐03	  2,00E-­‐03	  6,00E-­‐03	  1,00E-­‐02	  -­‐1,50E-­‐10	   -­‐5,00E-­‐11	   5,00E-­‐11	   1,50E-­‐10	   2,50E-­‐10	  Output ch. 1 Output ch. 2 FWHM 45 ps (Average)  Input pulse  FWHM ~ 42 ps (average) Outputs channels (time) dB Power divider 1:2 outputs IN OUT1 OUT2 Components not available on the market V Time (s) Michele Caselle High frequency technologies for high-data rate DAQ systems RF filters Track & hold Picosecond delay chip Fast ADC  Sampling stage Analog RF input Shielding via  Wideband CPW trans. line with via fences  100 GHz KAPTURE sampling board KAPTURE  sampling board Sampling channel High throughput readout  Board ü  Minimum sampling time: 3 psec à >300GS/s (equivalent sampling mode) ü  12 bit ADC resolution ü  Configurable for the readout of up to 4 ultra-fast detectors in parallel Michele Caselle High frequency technologies for high-data rate DAQ systems First beam test setup / May 2014 Terahertz beam line YBCO detector Cryogenic & detector  control Wideband power splitter LNA KAPTURE board and readout KAPTURE Graphic User Interface Time characterization (à next slide) Michele Caselle High frequency technologies for high-data rate DAQ systems -­‐5,00E-­‐03	  5,00E-­‐03	  1,50E-­‐02	  2,50E-­‐02	  3,50E-­‐02	  4,50E-­‐02	  5,50E-­‐02	  -­‐1,00E-­‐10	   -­‐5,00E-­‐11	   0,00E+00	   5,00E-­‐11	   1,00E-­‐10	   1,50E-­‐10	  Ampliture	  (V)	  .me	  (sec)	  42 ps Time characterization with YBCO detector pulse YBCO detector pulse acquired using equivalent sampling method by KAPTURE: minimum sampling time 3 ps, pulse repetition rate 500 MHz Pu lse measured by rea l - t ime oscilloscope (bandwidth 60GHz) “Average” Michele Caselle High frequency technologies for high-data rate DAQ systems -­‐5,00E-­‐03	  5,00E-­‐03	  1,50E-­‐02	  2,50E-­‐02	  3,50E-­‐02	  4,50E-­‐02	  5,50E-­‐02	  -­‐1,00E-­‐10	   -­‐5,00E-­‐11	   0,00E+00	   5,00E-­‐11	   1,00E-­‐10	   1,50E-­‐10	  Ampliture	  (V)	  .me	  (sec)	  42 ps step 150 ps 42 ps Time characterization with YBCO detector pulse YBCO detector pulse acquired using equivalent sampling method by KAPTURE: sampling time 3 ps, pulse repetition rate 500 MHz Fine delay  (step 3 ps) The pulse width (FWHM) measured by KAPTURE is 42 ps in agreement with the measurement by fast real-time oscilloscope Results: Pu lse measured by rea l - t ime oscilloscope (bandwidth 60GHz) Pulse measured by KAPTURE,  operating in sampling timing  “Average” Michele Caselle High frequency technologies for high-data rate DAQ systems Turn Sampled bucket Peak pulse amplitude 16 2 ns 184 bunches revolution time 368ns Train 1 and 2 Single filled bucket ANKA Possible filling scheme at ANKA Real-time pulse sampling v  Each pulse acquired by 4 samples v  Pulse repetition rate of 500 MHz  v  Pulse reconstruction (GPU) and measurements of: v  Pulse amplitude  (mV) v  Peaking time (ps) v  Pulse width (ps) v  Fast Fourier Transform (GPU) Performance real-time mode: Beam test with YBCO detector and KAPTURE Michele Caselle High frequency technologies for high-data rate DAQ systems An Ultra-fast Picosecond Digitizer for Coherent Synchrotron Radiation KAPTURE system ü  Simultaneous monitor of CSR fluctuation of all buckets. (multi-bunch mode) ü  Continuous turn-by-turn acquisition ü  Studies of CSR fluctuation at different bunch current regimes Frequency behaviour of CSR @ different bunch current CSR fluctuation (time domain) With YBCO detector Acquired with Schottky diode Opens up new diagnostic possibilities such as  instantaneous measurement  of bursting threshold and longitudinal particle dynamics Michele Caselle High frequency technologies for high-data rate DAQ systems High-speed DAQ architecture and real-time elaboration Real-time feedback to FPGA Driver GPU/CPU algorithms Driver GPU/CPU algorithms Data evaluation Data source From Detectors GPUs/CPUs infrastructure GPU Card Real time  analysis  PCIe GEN 2/3  X8 or x16 lanes Up to 0.25 GB/s Optical /electrical Data Storage FPGA Memory FPGA Input stage Memory FPGA Connection DDR - Memory PCIe Back-end ReadOut card Point – to – point DAQ architecture  Ø  Very-fast data link à to move data from data-source to real-time elaboration Ø  Real-time elaboration à by Graphics Processing Unit (GPU)  q  High flexibility readout card à based on FPGA q  Fast data throughput à based on last generation of “commercial” data link PCIe/InfiniBand q  High-flexibility real-time elaboration à based on GPU On-line display Michele Caselle High frequency technologies for high-data rate DAQ systems High-speed DAQ architecture and real-time elaboration Real-time feedback to FPGA Driver GPU/CPU algorithms Driver GPU/CPU algorithms Data evaluation Data source From Detectors GPUs/CPUs infrastructure GPU Card Real time  analysis  PCIe GEN 2/3  X8 or x16 lanes Up to 0.25 GB/s Optical /electrical Data Storage FPGA Memory FPGA Input stage Memory FPGA Connection DDR - Memory PCIe Back-end ReadOut card Point – to – point DAQ architecture  Ø  Very-fast data link à to move data from data-source to real-time elaboration Ø  Real-time elaboration à by Graphics Processing Unit (GPU)  Ø  GPUDirect data access à data write into GDDR5 Memory and not in the system memory q  High flexibility readout card à based on FPGA q  Fast data throughput à based on last generation of “commercial” data link PCIe/InfiniBand q  High-flexibility real-time elaboration à based on GPU On-line display Michele Caselle High frequency technologies for high-data rate DAQ systems Novel concept of DMA CPU 1. 2. 3. 4. 5. Block Length Mem Base Addr 6. REQ ACK 1.  Driver write the command and Descriptors in the FPGA and memory blocks allocation (Initialization) 2.  DMA – Start by REQuests to CPU and waiting the ACKnowledge signal   3.  DMA load the descriptor from the FIFO and fetch the DATA 4.  Initiated the Memory Block and 5. Data transfer from I/O à to Block Memory 6.  Update the Status à number of blocks written, address, status ..  Initialization DMA Controller DMA Command Register Device Base Register Block Length Register Mem Base Register Temporary Data Register Status WR/RD update FPGA Descriptors FIFO FPGA DMA engine System memory (Driver) Memory Block DMA Status Memory Block .. Memory Block Ring buffer organization GPU Operations: Michele Caselle High frequency technologies for high-data rate DAQ systems Comparison with commercial PCIe-DMA architecture Virtex 6 –> XC6VLX240–2 FF1759 PCIe and GTX HW-core  DMA engine NW DMA engines Same logic functions Michele Caselle High frequency technologies for high-data rate DAQ systems ü Average data throughput of 3.5 GB/s (DMA max data transfer @ 4 GB/s). ü No bit errors observed (tested up to several TB of data exchanged) PCIe-DMA architecture - performance GEN 1 GEN 2 Memory Write Performance (Average) Dash lines theoretical limit L. Rota & M. Caselle “High-throughput PCIe DMA architecture for Gigabyte Data Transmission”. IEEE-Transactions on Nuclear Science. Real Time Conference 2014  Michele Caselle High frequency technologies for high-data rate DAQ systems PCIe-DMA for generation 3 Preliminary results of a DMA – PCIe based on GEN 3 x8 lanes Virtex 7 -> XC7VX690T -2  Xilinx PCIe Core KIT - DMA No optimized firmware and driver  Low occupancy NO errors à detected during the data transferred > 6 GB/s Michele Caselle High frequency technologies for high-data rate DAQ systems KIT-DMA for High Energy Physics DAQ ALICE C-RORC à  RobinNP Alice DAQ  ATLAS DAQ  ATLAS – FELIX (FE Link eXchange) Are interested in KIT-DMA for HL-LHC upgrade  PCIe x8 GEN 2 PCIe x8 GEN 3 Michele Caselle High frequency technologies for high-data rate DAQ systems Conclusions q  Kapture àproper instrumentation for picosecond diagnostic beam and CSR studies. q Thanks to Kapture à published more than 20 scientific papers on CSR studies for future accelerators q Synchrotron community is interested in Kapture q  New DMA-PCIe Readout compatible with GPU Direct access q HEB community interested at KIT-DMA logic q  Future Kapture à based on SiGe or different monolithic integrated circuits Thank you for you attention 

High frequency technologies for high-data rate DAQ systems

https://publikationen.bibliothek.kit.edu/230101433/3816856

High frequency technologies for high-data rate DAQ systems

Abstract

Similar works

Full text

Available Versions

KITopen