In this contribution, we present preliminary results of using graphic card with hardware support for ray tracing for physics simulation of a positron emission tomography (PET) scanner. On our simplistic setup, we notice an impressive (about 350 times) speedup compared to Geant4 code running on modern multicore CPU. We expect this speedup to come down but remain substantial also for other more complicated scenarios

Białas, Piotr

Nowakowski, Kamil

Sagało, Przemysław

Acta Physica Polonica B

English

Jagiellonian Univeristy Repository

Vol. 51 (2020) Acta Physica Polonica B No 1SIMULATING THE J-PET DETECTOR ON NVidia RAYTRACING HARDWARE∗P. Bialas, P. Sagało, K. NowakowskiFaculty of Physics, Astronomy and Applied Computer ScienceJagiellonian University, Kraków, Poland(Received November 25, 2019)In this contribution, we present preliminary results of using graphic cardwith hardware support for ray tracing for physics simulation of a positronemission tomography (PET) scanner. On our simplistic setup, we notice animpressive (about 350 times) speedup compared to Geant4 code running onmodern multicore CPU. We expect this speedup to come down but remainsubstantial also for other more complicated scenarios.DOI:10.5506/APhysPolB.51.1911. IntroductionMonte-Carlo methods (MC) are by now extensively used for designinghardware and software for medical imaging. The software of choice for thiskind of simulations are Geant4 [1] and its extension GATE [2]. While Geant4is a powerful simulation toolkit it can be slow as it currently does not takeadvantage of the vector hardware as found in modern processors and GPUs.Efforts to change this state are underway, notably in the framework of theGeantV project [3]. GeantV is a major undertaking aimed at vectorizing mostof the Geant4 code and intended to be used for simulations in the nextgeneration high-energy experiments. We have opted instead for an approachcustomized for a particular application: simulating our prototype positronemission tomography scanner J-PET [4–7]. This is suited for our limitedresources and permitted us to experiment with the newest hardware thatwould not necessarily scale up to bigger projects. Restricting our focus toPET has the advantage that the energy range is also very limited, so wedecided to simulate only the Compton scattering as it is dominant processin the considered energy range.∗ Presented at the 3rd Jagiellonian Symposium on Fundamental and Applied SubatomicPhysics, Kraków, Poland, June 23–28, 2019.(191)192 P. Bialas, P. Sagało, K. Nowakowski2. Ray tracingThe problems with efficient vectorization are, of course, not unique toparticle physics simulations. In fact, those simulations are similar to a ren-dering technique for global illumination known as ray tracing [8]. In thisapproach, rays of light are traced usually from the camera to the objecton scene and then reflected, refracted or scattered according to laws of op-tics. There were many attempts to implement efficient ray tracing on GPUs(e.g. [9, 10]). One big effort came from NVidia in the form of the Op-tiX engine [11]. This provided a relatively easy to use framework for GPUaccelerated ray tracing and it was already used for particle detector simula-tions [12]. Meanwhile, NVidia has released GPUs with hardware support forray tracing. This contribution presents results obtained using OptiX enginewith NVidia RTX 2080 Ti graphics card.3. ImplementationPrograming the RTX pipeline amounts to specifying programs that willbe executed automatically on the graphic hardware. Two main programsare: ray generation program that will generate initial rays, and the closesthit program that will be called when ray intersection with the triangle closestto its origin is detected. This program can in turn spawn other rays.Physics process is defined by the mean free path and scattering angledistribution. We store the energy-dependent mean free path as 1D layeredtexture, one layer for each material. For sampling scattering angle prob-ability distribution, we use the inverse cumulant method. We discretizethe whole energy range and for each energy value, we calculate the inversecumulant. The resulting array is stored as a 2D texture. The additionaladvantage of using textures is optimized memory access on GPU and auto-matic interpolation.When the closest hit program is invoked, it checks if the particle inter-acted before hitting the border by comparing the interaction length to thedistance traveled. If so, it calculates the energy change and scattering angleand recasts the ray in appropriate direction from the interaction point. Ifnot, ray is recast in the same direction. Obviously, this is possible only forneutral particles or in the absence of magnetic field which is the case forPET scanners.A common bottleneck of GPUs is the data transfer between CPU andGPU. We generate all of the rays directly on GPU so the transfer to GPU isnot a problem. However, we are using quite a lot of memory on the GPU. Thisis due to the fact that we allocate each event its own memory slot capable ofstoring a fixed number of scatterings. For a detector like the J-PET, mostof those slots are unoccupied. For this reason, before transferring the databack to CPU we use stream compaction to eliminate empty slots.Simulating the J-PET Detector on NVidia Ray Tracing Hardware 1934. ResultsWe have tested our implementation on the three layers J-PET proto-type [6, 13]. This detector contains 192 half meter long scintillators in total.We have compared our code to a multithreaded Geant4 implementation withall the physics processes except Compton scattering switched off. Geant4code was executed on a six core Intel i9 CPU. The results are presented inTable I. One event corresponded to casting two rays (γ quanta) in oppositedirections. The rays origins were selected randomly from a 4 mm3 voxel,with one corner at the center of the detector.TABLE ITiming (in seconds) for simulating three layers of the J-PET detector. Only Comp-ton scattering was considered.Geant4/i91 RTX 2080Ti2 GTX 10803HW4 SW4 HW4 SW4No. of threads5 12 8704 21760 12800 12800No. of events (×106) Speedup1 6.2 0.02 0.03 0.07 0.08 3102 12.9 0.04 0.07 0.14 0.15 3224 26.1 0.08 0.13 0.27 0.29 3308 57.7 0.16 0.25 0.53 0.57 3651 6 cores, 32GB RAM.2 4352 CUDA cores, 11.75 GFLOPS, 11GB GRAM.3 2560 CUDA cores, 8.20 GFLOPS, 8GB GRAM.4 HW—Hardware triangle-ray intersection, SW— Software triangle-ray intersection.5 The number of threads was chosen as to give the best performance for 8 millionsevents.The obtained results clearly show the advantage of the hardware accel-eration for ray tracing. The GTX 2080Ti is nominally only 1.4 times fasterwhen considering GFLOPS than GTX 1080 but over three times faster onour simulation. Also switching off the hardware acceleration for ray triangleintersection test increases the time by 50%.5. DiscussionWe have shown that new graphics hardware is capable of running physicalsimulations over 350 times faster compared to standard simulation softwareon modern CPU. However this represents probably only the upper limitof what can be achieved. Scaling the implementation to handle multiplephysical processes will most probably result in decreased performance asthe thread divergence will increase. Also the processes with very short step194 P. Bialas, P. Sagało, K. Nowakowskilengths such as ionization will not benefit as much. Support for new shapeshas also to be added as far we are using the built-in hardware acceleratedtriangle meshes. This enables to efficiently use simple shapes as boxes, butwould be very wasteful for other shapes like cylinders or cones. Those shapescan be accommodated using custom written intersection programs.We have discovered two issues with presented approach. Spawning newrays is done in recursive fashion and the recursion depth is limited currentlyto 31 by hardware. This was not a problem in the case of the simple modelwe simulated, however, in general, the recursion would have to be managedexplicitly by maintaining a stack in global memory.The second issue relates to numerical accuracy. Geant4 uses double pre-cision and GPUs use single precision. Because of this, our physical precisionwas of the order of ten microns. This is adequate in our simulations butscaling up to bigger, in terms of size, detectors would entail proportionalloss in precision.We find those preliminary results very encouraging. It is obvious thatrealistic simulations still require a significant extension of our code whichwill undoubtedly result in loss of the performance but even tenfold drop inperformance would still result in significant speedup. In our opinion, the raytracing acceleration hardware seems to be suitable for simulations of PETscanners and maybe other medical devices. This, because those applicationsdo not require simulating magnetic field, energy range of particles is limitedand the detectors are generally small.REFERENCES[1] S. Agostinelli et al., Nucl. Instrum. Methods Phys. Res. A 506(3), 250(2003).[2] S. Jan et al., Phys. Med. Biol. 49, 4543 (2004).[3] G. Amadio et al., J. Phys.: Conf. Ser. 664, 072006 (2015).[4] P. Moskal et al., Phys. Med. Biol. 64, 055017 (2019).[5] P. Kowalski et al., Phys. Med. Biol. 63, 165008 (2018).[6] S. Niedzwiecki et al., Acta Phys. Pol. B 48, 1567 (2017).[7] P. Moskal et al., Phys. Med. Biol. 61, 2025 (2016).[8] T. Whitted, Commun. ACM 23, 343 (1980).[9] J. Popov, J. Günther, H.-P. Seidel, Comp. Graph. Forum 26, 415 (2007).[10] J. Gunther, et al., in :Proceedings of IEEE Symposium on Interactive RayTracing, September 10–12, 2007.[11] S.G. Parker et al., ACM Trans. Graph. 29, 66 (2010).[12] S. Blyth, J. Phys.: Conf. Ser. 898, 042001 (2017).[13] G. Korcyl et al., IEEE Trans. Med. Imag. 37, 2526 (2018).

Simulating the J-PET detector on NVidia ray tracing hardware

https://ruj.uj.edu.pl/xmlui/bitstream/handle/item/153439/bialas_sagalo_nowakowski_simulating_the_j-pet_detector_on_nvidia_ray_tracing_hardware_2020.pdf?sequence=1&isAllowed=y

Simulating the J-PET detector on NVidia ray tracing hardware

Abstract

Similar works

Full text

Available Versions

Jagiellonian Univeristy Repository