Balzer, M.

Caselle, M.

Chilingaryan, S.

Dritschler, T.

Farago, T.

Herth, A.

Kopmann, A.

Rota, L.

Stevanovic, U.

Vogelgesang, M.

Weber, M.

English

KITopen

KIT – University of the State of Baden-Wuerttemberg and  National Research Center of the Helmholtz Association Institute for Data Processing and Electronics - IPE www.kit.edu The UFO DAQ-Framework for High-performance Data Acquisition in Synchrotron Applications A. Kopmann, M. Balzer, M. Caselle, S. Chilingaryan, T. Dritschler, T. Farago, A. Herth, L. Rota, U.Stevanovic, M. Vogelgesang, M. Weber  Institute for Data Processing and Electronics 2 Goals !   Micro X-ray imaging offers a broad field of applications: !   Medical science !   Non-destructive testing !   Material and life science research etc… !   Are time-resolved measurements possible?  03.06.14 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Institute for Data Processing and Electronics 3 Application: 4D X-ray Cine-Tomography Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 dos Santos Rolo T et al. PNAS 2014;111:3921-3926 Institute for Data Processing and Electronics 4 Goals !   Micro X-ray imaging offers a broad field of usage: !   Medical science !   Non-destructive testing !   Material and life science research etc… !   Are time-resolved measurements possible? !   2D: radiography with > 1000 fps  !   3D: tomography and laminography > 10 fps !   2D projections > 5000 fps  !   DAQ Requirements: !   Large (continuous) data rates !   On-line data analysis + monitoring !   Trigger + automatic control 03.06.14 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Institute for Data Processing and Electronics 5 Motivation: Smart scientific cameras  !   Detectors for high speed imaging !   What is commercially available? Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Internal Memory e.g.: 36GB ! 5s  Readout Camera Link (250 / 850 MB/s) USB3 (new: 500 MB/s) GigE Vision (125 MB/s)   CMOS sensor     pco.dimax 4MP@1300fps (7500MB/s) Data rates limit camera performance No online data analysis Custom embedded logic is missing Scientific sensor developments? Institute for Data Processing and Electronics 6 Motivation: Smart scientific cameras  !   Goal: Modular DAQ Framework Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 3. Full streaming readout e.g. using PCIe, IB 1. Modular  sensor interface     e.g. for Helmholtz Detector and Systems Platform (DTS) 2. Embedded logic & Control e.g. Smart Phase Contrast (w HZG) 4. Real-time data processing e.g. with GPUs Institute for Data Processing and Electronics 7 Motivation: Smart scientific cameras  !   Components: Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 3. Readout •  DMA engine •  Linux drivers  1. Modular  sensor interface     2. Embedded data processing •  Fast control system 4. GPU Computing •  Camera abstraction •  Programming environment •  Optimized algorithms Institute for Data Processing and Electronics 8 DAQ framework Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Uros Stevanovic – 19th Real Time Conference, M y 26-30, Nara, Japan Institute for Data Processing and Electronics 9 DAQ framework Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 "  Smart camera platform "  Image based trigger "  PCIe DMA engine for fast data streaming Institute for Data Processing and Electronics 10 Smart camera platform – architecture Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Camera readout FPGA internal architecture CPU/%GPU%memory%memory%memory%Chipset%root%port%PCIe%2%DMA%%(KIT_IPCore)%User%bank%register%FPGA%control%FSM%Master%control%FIFO%FIFO%DDR3%Memory%CMOS%image2sensor%On2line%parallel%data%processing%CMOS%Control%Interleaving%FSM%SerDes%%%input%stage%%(KIT_IPCore)%Frame req. Ext. exposure time Chip configuration Optical/Electrical X4/8 lanes @ 5Gb/s PCI Express cable DDR3%interface%(KIT_IPCore)%... 4/8 4/8 Lentgh up to 100m FIFO%Institute for Data Processing and Electronics 11 Daughter card CMOSIS sensor & optical lens Mother readout and data processing card Virtex 6 FPGA  DDR Memory EndPoint PCIExpress link PCIe backplane and  power supply •  Low noise daughter PCB design •  Cooling system (silicon sensor) controlled by FPGA. •  Remote camera control  (i.e. external exp. time, frame req, enable readout, busy, etc.) First high smart camera with CMOS  sensor 03.06.14 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Institute for Data Processing and Electronics 12 Smart camera platform – front-end Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 CMOS%image2sensor%CMOS%Control%Interleaving%FSM%SerDes%%%input%stage%%(KIT_IPCore)%Frame req. Ext. exposure time Chip configuration ... FIFO%Front-end: •  FMC daughter board •  CMOS image sensors •  SerDes input stage     Institute for Data Processing and Electronics 13 Smart camera platform – sensors Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 CMOS sensors: "  4 MPixels and 2.2 MPixels "  Monochromatic and color "  Pixel pitch 5.5 µm "  Cooling     Institute for Data Processing and Electronics 14 Smart camera platform – sensors Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 In progress: High frame rate CMOS sensor   "  Vacuum environment "  Peltier cell: -5 ºC with 30 % of the power 5k fps @ 1MP CMOS sensors: "  4 MPixels and 2.2 MPixels "  Monochromatic and color "  Pixel pitch 5.5 µm "  Cooling     Institute for Data Processing and Electronics 15 A smart non-camera application Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Ultrafast digitizer for beam monitoring: •  Programmable picosecond sampling •  Data rate 4x500MS/s ! 3 GB/s •  Pulse reconstruction Picture Terahertz detector…  Detector response / a. u. Time / a.u. ADC1 ADC2 ADC3 ADC4 on GPU  (Graphic Processing Unit) / FPGA  fit Tunable sampling 12 - 100 ps Institute for Data Processing and Electronics 16 Smart camera platform – streaming readout Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 CPU/GPU high performance processing stage PCI%Express%+%DMA%%(KIT_IPCore)%User%bank%register%High-speed PCIe-DMA engine Gen2 x8 – 3.4 GB streaming DDR3%Memory%DDR3%interface%(KIT_IPCore)%UFO camera THz detector Bolometer Institute for Data Processing and Electronics 17 Smart camera platform – streaming readout Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 CPU/GPU high performance processing stage PCI%Express%+%DMA%%(KIT_IPCore)%User%bank%register%High-speed PCIe-DMA engine DDR3%Memory%DDR3%interface%(KIT_IPCore)%UFO camera THz detector Bolometer Institute for Data Processing and Electronics 18 Smart camera platform – embedded processing Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 On#line'parallel'data'embedded'processing'Possible applications: •  (Simple) image processing •  Camera internal control: •  Trigger •  Selective readout •  Exposure/gain control •  Control of sample environment •  Automation of complex procedures •  Feedback control  Institute for Data Processing and Electronics 19 Example: Image based self-trigger Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Reference frame ... Only 2% of rows  have changed Most of the time no changes Rare fast event @ 10 KHz Event N Event N+1 Only small and a-priori unknown changes ! Need for adaptive frame rate  Application: Bubbles merging in gelatinous agar … Frame N Frame N+1 … Solution:  Interleaving readout and event-trigger Institute for Data Processing and Electronics 20 Fast reject efficiency 03.06.14 !   Frame rate depends on the used parameters and the event size !   Effective frame rate can be more than five times higher than nominal !   Concert can automatically set initial parameter values with the desired maximum speed, object size detection, and minimum noise influence -- area 100, row thr 35 -- area 60, row thr 40 -- area 60, row thr 35 + actual frame rate Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Institute for Data Processing and Electronics 21 Programmable hardware !   The DAQ framework is fully programmable !   Based on FPGAs !   Three IP cores !   Custom “smart” logic UFO camera FPGA utilization: !   Yellow:         PCIe DMA !   Dark Blue:    DDR3 interface IP !   Red:             Input stage !   Bright blue:  Control and                 processing logic Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Xilinx XC6VLX365TFF1759-2 Institute for Data Processing and Electronics 22 UFO camera at the IMAGE beamline Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Vacuum case for cooling Readout board Virtex 6 FPGA PCIe x8 connection F-mount Institute for Data Processing and Electronics 24 Smart phase contrast camera !   Goals:  !   Automatic grating control !   Reconstruction (with GPU) Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Institute for Data Processing and Electronics 25 03.06.14  "  Optimized for high throughput "  Configurable register model "  Comfortable usage   Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework DAQ framework Institute for Data Processing and Electronics 26 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework ALPS Advanced Linux PCI Services Components !   PCI driver !   Register Model    A reusable components for custom PCI electronics Arbitrary PCI  Device XML  Register  Model Scripting DMA in  User  Space Motivation !   Synchronization Software and    Hardware development !   Easy hardware debugging !   Keeping drivers up to date with latest    Linux kernels !   Multiple common components !   DMA Engine  !   Custom Event Plugins !   Web API (planned) 03.06.14 Institute for Data Processing and Electronics 27 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework PCI + PCIe interface in user-space Driver PCI Bar mapping DMA Memory Management IRQ Handling Interlocking Thin Linux Module about 2000 lines only PCILIB User-space Layer pcitool Command-line tool GUI User Interface in Python/GTK Web Service Remote Programming Interface Scripting Bash, Perl, Python LabVIEW Control System Integration PCI / PCI Express Board (Variety of FPGA Boards: IPE Camera, etc) VFIO + UIO 03.06.14 Institute for Data Processing and Electronics 28 03.06.14  "  Linux (64bit) "  Supported cameras:  "  pco  "  photon focus "  UFO camera "  Tango driver  Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework DAQ framework Institute for Data Processing and Electronics 29 03.06.14 "  Seamless device integration "  High performance computing "  Image-based feedback control Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework DAQ framework Institute for Data Processing and Electronics 30 “Concert” = Fast control system for IMAGE Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Beam line Readout Computing Storage UFO Framework Filters + Algorithms                      “Concert” = Fast control for IMAGE Beamline instrumentation Sample env. + optics Computing infrastructure Slow devices  (via Tango) Fast  devices 03.06.14 Institute for Data Processing and Electronics 31 Concert details Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 !   Python based !   Underlying hardware-specific details are abstracted: !   e.g. Camera class provides grab method to acquire a frame: !   e.g. setting the exposure time:  !   Asynchronous device access: !   Parameters access and device methods wrapped in future objects !   Methods to query state, final result, callback frame = cam.grab() cam.exposure_time= 2*q.milliseconds   # Req. units  # Synchronous access motor.position = 1 * q.mm # Asynchronous access returns a future future = motor.set_position(1 * q.mm) future.wait() class Motor(Device):     @async      def move(self, delta):          self.position += delta  motor.move(-2 * q.mm).wait() Institute for Data Processing and Electronics 32 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Filtered back projection performance GPU: 4 x GTX590 , 8 cores CPU: 2 x Xeon X5650, 12 cores (both from 2011) FPD Computing throughput    == Readout rate  ✔ 03.06.14 ≈10s / tomogram  Institute for Data Processing and Electronics 33 UFO parallel computing framework How to support code development for GPUs? Requirements: !   Processes data streams (usually 1 to 4 dimensional floating point data) !   Detect and use all hardware resources Developer: !   Hides parallelization and concurrency details !   Management of memory transfers !   Multiple implementations (e.g. for CPU + GPU) !   Automatic scheduling User: !   Simple end-user interface !   GUI + Scripting !   Modular algorithm design Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Institute for Data Processing and Electronics 34 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Realization •  Implemented in OpenCL •  Define algorithms as self-contained tasks  •  Specify data flow as edges in a graph 03.06.14 !  Matthias Vogelgesang gives          an introduction tomorrow  Institute for Data Processing and Electronics 35 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework Summary •  The UFO DAQ framework is modular •  Front-end / embedded processing / fast readout / GPU computing •  Various sensors exist •  Applications:  •  high-throughput camera,  •  intelligent trigger,  •  phase contrast •  Non camera use: beam monitoring with ps resolution •  UFO parallel computing framework supports  •  Development and management of optimized code  •  Supported architectures GPU, CPU, Xeon Phi, <every OpenCl device> •  FBP is now faster than DAQ  •  throughput ~1 GB/s           03.06.14 Institute for Data Processing and Electronics 36 Morphological dynamics and kinematics analysis of a moving screw joint Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Institute for Data Processing and Electronics 37 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework What’s next? Scaling up to clusters Camera SSD Cache up to 8 GB/s 4GB/s <2 GB/s 0,25 GB/s Transfer rates Master Node Task Scheduler (lots of memory) Distributed Storage OpenCL Node OpenCL Node Readout PC CUDA + GPUDirect Storage Node Storage Node Infiniband (Optical) IB router LSDF Large Scale Data Facility 03.06.14 Institute for Data Processing and Electronics 38 Science 3D  2.6.-4.6.2014,   UFO DAQ-Framework 03.06.14 Thank you! http://ufo.kit.edu https://github.com/ufo-kit/concert 

The UFO DAQ-framework for high-performance data acquisition in synchrotron applications

https://publikationen.bibliothek.kit.edu/230098798/3816761

The UFO DAQ-framework for high-performance data acquisition in synchrotron applications

Abstract

Similar works

Full text

Available Versions

KITopen