Caselle, M.

Chilingaryan, S.

dos Santos Rolo, T.

Farago, T.

Kopmann, A.

Mirone, A.

Stevanovic, U.

Vogelgesang, M.

English

Santos Rolo, T. dos

KITopen

small (4GB) standard (11GB) Big (120 GB) Huge (300 GB)020040060080010001200Memory SSDMB/sCPU GPU11010010001000017,931016,26MB/sA GPU-based Architecture for Real-TimeData Assessment at Synchrotron ExperimentsS. Chilingaryan1, M. Caselle1, T. Farago1, A. Kopmann1, A. Mirone2, T. dos Santos Rolo1, U. Stevanovic1, M. Vogelgesang1 High Speed X-Ray ImagingTypical SetupX-ray tomography has been proven to be a valuable tool for understanding internal, otherwise invisible, mechanisms in biology, materials research and other fields. Detectors employed at modern synchrotrons are able to deliver images with high resolution and at high frame rates  generating up to several gigabytes per second. The ability to process this information in real-time and present to the users without long processing delays is extremely important for synchrotron operation. It will increase experiment throughput and enable image-based control of dynamical processes under study. We have developed a GPU-based platform for high speed tomography optimized for continuous operation with streamed data. Our system consists of dedicated hardware platform, a camera  abstraction layer, a pipelined parallel programming framework, and a high-speed implementation of tomographic reconstruction. Using only a single GPU server we are able to handle the full throughput of CameraLink interface with 850 MB/s.θ=180θ=0Tomography at Synchrotron Light SourcesThe sample, evenly rotating  in the front of a pixel detector, is penetrated by X-rays produced in the synchrotronThe pixel detector registers series of parallel 2D projections of the sample density at different angles.3D Image ReconstructionFiltered back-projection is used to reconstruct 3D images from a manifold of 2D projections. The projection values are smeared back over 2D cross sections and integrated over all projection angles. To reduce blurring effect the projections are filtered in the Fourier space before being back projected.1 Projection 2 Projections 3 Projections 20 Projections 1000 ProjectionsSample: Plastic holder with porose polyethylene grainsSource data: 24GB (2000 projections,  3 Mpix, 32 bits)3D Image: 11GB (3 Gpix, 32 bits)Complexity: 53 Tflop back-projection + 0.6 Tflop filteringGoal: Reconstruct 3D image  in 1 minuteHeads of a newt larva showing bone formation and muscle insertions (top) and a stick insect (bottom), acquisition time 2s.1  Karlsruhe Institute of Technology, Karlsruhe, Germany    2 European Synchrotron Radiation Facility, Grenoble, France∑p=1PI p  x⋅cos pa − y⋅sin  pa , z According to the Back Projection algorithm, the pixel at position (x,y,z)  is computed by                                                , where P is the number of projections α is the angle between projections, and Ip is the image of p-th projection.Storage RingBending MagnetDMM MonochromatorAttenuatorSlitsBe-windowSlitsExperimentSampleXRYFilmDetectorCCDOpen Source Parallel Processing FrameworkCamera abstraction layer working in synchronous and asynchronous modes and supporting CameraLink, Firewire, and PCI-express camerasEasy exchange of parallel algorithms written in OpenCL language,  automated management of OpenCL buffersSupport multiple algorithms implementation optimized to different architectures and hardware-aware task schedulingPipelined processing, i.e. readout, CPU-based preprocessing, GPU-based reconstruction, and storage are executed in parallel.  Glib/GObject, scripting language support with introspection. Multiple python scripts for standard tasks.acquisitionflat fieldcorrection... ...noise reductionPreprocessingExecuted on CPUssinogramgenerationFFT... ...filterReconstructionExecuted on GPUsiFFT backprojectionStorage......manual post-processingLibUCA: Camera AbstractionModerate size data-sets are stored in memory and huge data-sets are cached on SSD RaidFiltered Back ProjectionGT200Base version Uses texture engineFermiHigh computation power, but low speed of texture unit Reduce load on texture engine: use shared memory to cache the fetched data and, then, perform linear interpolationusing computation units.KeplerLow bandwidth of integer inst-ructions, but high register countUses texture engine, but processes 16 projections at once and 16 points per thread to enhance cache hit rate GCNHigh performance of texture engine and computation nodesBalance usage of texture engine and computation nodes to get highest performanceVLIWExecutes 5 independent operations per threadComputes 16 points per thread in order to provide sufficient flow of independent instructions to VLIW engine+100%+530% +95%+75%Tunning Back Projection for Hardware ArchitecturesPerformanceFour stage pipeline is used. Data is prefetched from SSDs, preprocessed on CPUs, reconstructed using all GPUs, and stored to the real-time storage.All 4 stages are executed in parallel.Pinned (unswappable) memory buffers are used to speed-up data transfers. The slice is split in blocks and the transfer of next block is interleaved with reconstruction of the current one.Batched mode of Fourier library is used for better performance. Two real convolutions are computed using a single complex FFT transformAn-architecture specific variations of back projection algorithm is to better utilize GPU resourcesContact: http://ufo.kit.eduSuren Chilingaryan <Suren.Chilingaryan@kit.edu>Andreas Kopmann <Andreas.Kopmann@kit.edu>CPU: 2 x Xeon X5650 (total 12 cores at 2.66 Ghz)GPUs: 2 x GTX 580 + 4 x GTX580 ExternalImage LoaderPool of SinogramsCPU taskPool of Vertical Slices TextureData StorageWHGPU thread1st Stage2nd StageDouble bufferingDouble bufferingFilteringPCIe Dat a Tran sfe rPCIe  Dat aTran sfe rHost memory

A GPU-based architecture for real-time data assessment at synchrotron experiments

https://publikationen.bibliothek.kit.edu/220091846/3816306

A GPU-based architecture for real-time data assessment at synchrotron experiments

Abstract

Similar works

Full text

Available Versions

KITopen