A weakness in the wall of a cerebral artery causing a dilation or ballooning
of the blood vessel is known as a cerebral aneurysm. Optimal treatment requires
fast and accurate diagnosis of the aneurysm. HemeLB is a fluid dynamics solver
for complex geometries developed to provide neurosurgeons with information
related to the flow of blood in and around aneurysms. On a cost efficient
platform, HemeLB could be employed in hospitals to provide surgeons with the
simulation results in real-time. In this work, we developed an improved version
of HemeLB for GPU implementation and result visualization. A visualization
platform for smooth interaction with end users is also presented. Finally, a
comprehensive evaluation of this implementation is reported. The results
demonstrate that the proposed implementation achieves a maximum performance of
15,168,964 site updates per second, and is capable of speeding up HemeLB for
deployment in hospitals and clinical investigations

AbiNahed, Julien

Amira, Abbes

Bensaali, Faycal

Chen, Minsi

Coveney, Peter V

Dakua, Sarada

Esfahani, Sahar Soheilian

Richardson, Robin A

Younes, Georges

Zhai, Xiaojun

arXiv

University of Essex Research Repository

HEMELB ACCELERATION AND VISUALIZATION FOR CEREBRALANEURYSMSSahar Soheilian Esfahani?, Xiaojun Zhai†, Minsi Chen‡, Abbes Amira?, Faycal Bensaali?Julien AbiNahed§, Sarada Dakua§, Georges Younes§, Robin A. Richardson‖, Peter V. Coveney‖? College of Engineering, Qatar University, Doha, Qatar† School of CS and EE, University of Essex, Colchester, UK‡ Department of Computer Science, University of Huddersfield, Huddersfield, UK§ Department of Surgery, Hamad Medical Corporation, Doha, Qatar‖ Centre for Computational Science, University College London, London, UKAbstract—A weakness in the wall of a cerebral arterycausing a dilation or ballooning of the blood vessel is knownas a cerebral aneurysm. Optimal treatment requires fastand accurate diagnosis of the aneurysm. HemeLB is a fluiddynamics solver for complex geometries developed to provideneurosurgeons with information related to the flow of blood inand around aneurysms. On a cost efficient platform, HemeLBcould be employed in hospitals to provide surgeons with thesimulation results in real-time. In this work, we developedan improved version of HemeLB for GPU implementationand result visualization. A visualization platform for smoothinteraction with end users is also presented. Finally, a com-prehensive evaluation of this implementation is reported. Theresults demonstrate that the proposed implementation achievesa maximum performance of 15,168,964 site updates per second,and is capable of speeding up HemeLB for deployment inhospitals and clinical investigations.Keywords-Cerebral aneurysm, HemeLB, Visualization, GPUI. INTRODUCTIONA cerebral aneurysm is an abnormal focal dilation of abrain artery caused by a weakness in the blood vessel wall[1]. The number of patients who suffer from cerebrovas-cular disorders like cerebral aneurysms is growing in non-developed countries [2]. A quick review of the statisticsshows that people who already have or will develop brainaneurysm are between 1.5% to 5% of the general population[3]. Another study shows that about 5% of patients withgrowing cerebral aneurysm and 0.5-1.1% of those with non-growing brain aneurysms will suffer from rupturing [4].An effective treatment of aneurysms are endovascularapproaches which reduce the operative risks, pains and costof hospitalization [5]. These methods, which take advan-tage of intra-aneurysmal coils, may fail due to a lack ofinformation about the aneurysm status. In recent years, theThis paper was made possible by National Priorities Research Program(NPRP) grant No. 5-792-2-328 from the Qatar National Research Fund (amember of Qatar Foundation). The statements made herein are solely theresponsibility of the authors.combination of coils with stents has been widely used as anefficient treatment which reorganizes the blood flow in andaround the aneurysm [6]. In order to get the best treatment,the interventional radiologist has to identify the vasculargeometry and estimate cerebral blood flow behavior such aspressure and velocity from 2D and/or 3D images. Presently,there are few approaches to measure these flows and relateddata intraoperatively. Consequently, diagnosis and treatmentof aneurysms is highly dependent on the experience and skillof the radiologist.Modeling and simulation of fluid flow and hemodynamicscan provide clinicians with more precise analysis and thusdiagnosis of aneurysm effects. Simulation of fluid flowin large scale and complex geometries requires a physi-cal model and substantial computing resources as well asefficient software. Recently, the lattice-Boltzmann method(LBM) is widely used for simulation of fluid flows. TheLBM parallelizes efficiently making it useful for time-dependent simulation of large systems[7]. In [8], the ef-ficient hardware architecture of the LBM is introduced.HemeLB (Hemodynamic lattice-Boltzmann) is a massivelyparallel LBM fluid solver developed to simulate fluid flowsin sparse and complex systems on large supercomputingresources [7], [9], [10]. The aim of designing HemeLB wasto provide neurosurgeons with timely and clinically relevantassistance [11]. HemeLB blood flow simulations have beendeployed on High Performance Computing (HPC) platformsincluding HECToR, Blue Waters, SuperMUC and ARCHER[9], [10], [11]. It is required to optimize HemeLB ondedicated computational infrastructures in order to employthe software in clinical environments. Multicore platforms,General Purpose Graphical Processing Units (GPGPUs) andField Programmable Gate Arrays (FPGAs) are probablytodays most powerful computational hardware found invarious applications such as machine learning and artificialintelligence [12], [13], [14].The objective of this paper is to develop a version ofHemeLB optimized for GPU visualization and to evaluatearXiv:1906.11925v1  [physics.comp-ph]  27 Jun 2019it on a workstation with CUDA (Compute Unified DeviceArchitecture) capable GPUs. To achieve this, additionalfunctions and modifications have been made to the originalHemeLB software to include a real-time GPU based ren-dering engine for running HemeLB in local environments.Additionally, a visualization platform intended to ease accessto the simulation environment was also designed. Thisvisualization platform will help clinicians to simply launchand use the HemeLB simulation software in hospitals. Inorder to evaluate the proposed implementation, a set ofcomprehensive tests using real patient data has been carriedout on an Exxact Tensor workstation with four NVIDIAGPUs.The rest of the paper is organized as follows. In Section2, the lattice-Boltzmann model used in HemeLB is brieflydescribed. Section 3 presents the visualization in HemeLB.The experimental results and their evaluations are reportedin Section 4 followed by conclusions in Section 5.II. HEMELB MODELThe LBM is a fast and efficient technique for the sim-ulation of large and complex fluid systems, particularlyvia parallel implementations [7]. HemeLB applies a highlyparallelized implementation of the LBM, with performanceoptimizations for the sparse geometries common in vascularsystems. The LBM represents the geometry as a lattice offluid sites, each equipped with a single-particle distribu-tion describing the propagation to neighbouring (or nearby)sites along a finite number of discrete lattice vectors. Bymodelling the evolution of these particle distributions overconsecutive propagation and collision steps, we may recoverthe hydrodynamic behaviour expected for a continuum fluid.In the proposed system, we use a three dimensional latticewith 19 discrete velocities (D3Q19). Figure 1 presents anillustration of such a D3Q19 lattice.123456789 10111213 14015161817Figure 1. Lattice node of D3Q19 model [7].HemeLB is implemented with different Boundary Con-ditions (BC) [7], such as Ladd iolets for velocity inletBC [15] and Bouzidi-Firdaouss-Lallemand (BFL) for theinterpolated wall collision BC [16]. By the use of topology-aware two-level domain decomposition, HemeLB providesa good workload distribution for parallel implementation. Inaddition, the improvements made in HemeLB decrease re-dundant operations, improve pattern regularity and enhanceintra-machine communications [9].Shared MemoryPeripherals InterfaceGPU1MemoryGPU2MemoryGPU3MemoryGPU4MemoryCPUFigure 2. Simplified architecture of Exxact Tensor TWS-289059-DPNworkstation.III. HEMELB IMPLEMENTATIONThe benefits of GPU-oriented solutions such as processingspeed and analysis of large datasets are achieved by the useof thousands of processor cores on a single chip [12]. Inthe following section, the real-time visualization of HemeLBon an Exxact Tensor workstation with four CUDA capableGPUs is discussed.A. Architecture of Exxact Tensor workstationThe proposed work is implemented on an Exxact TensorTWS-289059-DPN workstation which is known as a deeplearning NVIDIA GPU solution [17]. The platform consistsof one Intel core i7-5960X processor, 4 TB HDD, 1 TB SSDas well as four GeForce GTX 1080 Ti GPUs. Each NVIDIAGPU has 11 GB GDDR5X random access memory with 484GB/s memory bandwidth and 3584 CUDA cores. The GPUbase clock speed is 1481 MHz and the memory is runningat 1376 MHz. Figure 2 shows the simplified architecture ofthe workstation with four NVIDIA GPUs.B. HemeLB configurationThe original version of HemeLB code is availablein [18]. The openmpi package and CUDA platformare required for the implementation of the proposedsystem. Once the package is built, compilation flags areneeded to be set as boundary conditions. The boundarysettings used in this experiment were INLET BOUNDARY,WALL INLET BOUNDARY, WALL BOUNDARY, andUSE VELOCITY WEIGHTS FILE which were set toLADDIOLET, LADDIOLETBFL, BFL, and ON respectively.Table I represents the simulation, geometry and inletssettings used in the HemeLB.Table IHEMELB SIMULATION, GEOMETRY AND INLETS SETTINGS.HemeLB settings ValuesStep length 2× 10−5 sVoxel size 100× 10−6Inlet velocity file (see Figure 8)C. HemeLB visualization1) HemeLB interface: The original HemeLB code ex-poses a low-level technical interface useful for advancedusers and accessed via the commandline. In order to en-able clinicians to run HemeLB conveniently, a visualizationplatform has been developed. It provides end users with asmooth interaction environment for running the software. Itallows clinicians and neurosurgeons to steer the visualizationand manage the simulation operations. Figure 3 illustratesthis visualization platform designed with the Qt cross-platform framework.Figure 3. Visualization platform designed for real-time HemeLB resultsvisualization.The visualization platform helps clinicians and neurosur-geons to launch HemeLB with a simple click. First, the usershould upload the configuration file of the subject underexamination, then set some input settings such as simulationmode and rendering number. This platform gives the controlof the simulation mode to the clinicians. The user can choosea single core or multicore simulation mode before runningthe software. The number of cores in multicore mode canalso be selected by the end user. Another option is thenumber of rendered frames which can be chosen from alist or entered by the clinician. Finally, the simulate andinterrupt buttons give the control of simulation operation tothe user. A flowchart of the designed platform is depictedin Figure 4.Launch the visualization platformLoad the config fileSelect the simulation modeDetermine the number of renderingStart the simulationView the resultsSteer the visualization parametersEndStartInterrupt the simulationFigure 4. Flowchart of the HemeLB visualization platform.2) Real time visualization: Visualization of HemeLB isdone on devoted CUDA capable GPUs. The LBM nodesand the visualization client communicate via the existingmessage passing interface in HemeLB. Figure 5 shows thearchitecture of the proposed framework. In Figure 5, Latticeproperties are computed and cached on each CPU nodethen communicated to Node0. This node schedules latticedata transfer from the compute nodes and handles the viewsteering. The incoming lattice data is shared with GPU nodesand rendering is performed at the same time.The three-tier architecture of the visualization client ispresented in Figure 6. The first tier is the OpenGL appli-cation and the middle one is the host layer which storesthe application data such as steering parameters and cachedlattice properties. Eventually, all the voxels for direct volumerendering are stored in the rendering layer which is depictedas the top tier. The CUDA cores in the rendering layerperform the volume rendering kernel based on the raymarching algorithm [19].Rendeing Engine (GPU)LB Calculation Engine (CPU)InterconnectionsMessage Passing InterfaceCPU Node (0)LBMCPU Node (1)LBMCPU Node (n-1)LBMCPU Node (n)LBM?.?.Figure 5. Architecture of proposed visualization framework.CUDA CoresVolume Render KernelGPU MemoryView ParamsTransfer Function Texture3D Texture UnitCPU CoresLBM RoutinesHost MemoryViz Steering ParamsCached Lattice PropertiesOpenGL Visualisation FrontendCPUGPUFigure 6. The three-tier internal architecture of visualization client.It is well known that the number of memory transfers iscritical in real-time applications. In HemeLB visualization,the memory transfers include the transmission of latticedata to the visualization volume and a number of steeringsimulation. In order to optimize the memory transfers in theproposed framework, the following two-level memory accesssolution is deployed. In the first place, the visualizationvolume is stored in a 3D texture unit to enable the fastsampling of voxel values. Secondly, in order to store theviewing parameters and the look-up table, the GPU constantmemory buffers are used. This allows Direct Memory Ac-cess (DMA) whose performance is comparable to readingfrom registers. The steerable parameters introduced in thevisualized HemeLB include zooming, model rotation andadjustment of scaling and offset of the transfer functions.IV. EXPERIMENTAL RESULTS AND DISCUSSIONFive subjects of 3D Rotational Angiography (3DRA)selected by Hamad Medical Cooperation (HMC) clinicianswere used to evaluate the proposed visualization solution.Figure 7 represents two STereoLithograph (STL) files usedto evaluate HemeLB visualization. In Table II the numberof lattice sites and blocks for each subject are presented.HemeLB was employed using a D3Q19 lattice model withsimple bounce-back boundary condition and a fixed phys-ical viscosity of 0.004 Pa.s. The inlet velocity applied forsimulation with each STL input is illustrated in Figure 8.(a) Subject 16 (b) Subject 23Figure 7. STL files used for evaluation of visualized HemeLB.Table IIOVERVIEW OF SIMULATION DOMAINS.Name Number of lattice sites Number of blocksSubject 16 296,814 15,000Subject 23 132,910 5,320Table III shows the performance for two subjects sim-ulated on the Exxact Tensor workstation. The maximumperformance of 15,168,964 Site Updates Per Second (SUPS)is achieved in this evaluation. As it is depicted in TableIII, the larger geometry has the highest SUPS, although thetiming performance for different subjects is similar. Thesimulation results of two geometries obtained using theproposed version of HemeLB are illustrated in Figure 9.Figure 8. The inlet blood velocity applied in each patient test case.Table IIISIMULATION TIMING DATA.Name Total time (s) Number of steps SUPSSubject 16 906 46,302 15,168,964Subject 23 297 30,558 13,674,962(a) Subject 16 (b) Subject 23Figure 9. Results of HemeLB visualization framework.V. CONCLUSIONIn this paper, a solution for designing and implementing areal-time visualization version of HemeLB on a workstationis presented. The proposed implementation enables HemeLBto be exploited locally in hospitals rather than in a distributedenvironment. In addition, a simple visualization platform forthe HemeLB environment is also introduced in this work.This platform allows clinicians to launch the simulation, andto view and steer the visualized results in a user friendlyand smooth manner. The reported results in this experienceare obtained from tests performed on real patient data.The results achieve a maximum performance of 15,168,964site updates per second and demonstrate that the proposedimplementation is capable of supporting HemeLB in clinicalenvironments. A working system which can be used inhospitals for training and practicing the cerebral surgeriescan be developed as a future work.REFERENCES[1] R.T. Higashida, “What you should know about cerebralaneurysms,” Pamphlet. American Heart Association Cardio-vascular Council, 2003.[2] N. Benkirane, A. Bennis, S. Bellakhdar, R. Habbal, S. Aidi,M. El Alaoui Faris, H. El Otmani, B. El Moutawakil, M.A.Rafai, and I. Slassi, “Stroke risk factors in a moroccanpopulation: a multicentric prospective study,” Journal of theNeurological Sciences, vol. 357, pp. e367, October 2015.[3] T. Mashiko, K. Otani, R. Kawano, T. Konno, N. Kaneko,Y. Ito, and E. Watanabe, “Development of three-dimensionalhollow elastic model for cerebral aneurysm clipping sim-ulation enabling rapid and low cost prototyping,” WorldNeurosurgery, vol. 83, pp. 351–361, March 2015.[4] Verywell health, “Brain aneurysm prognosis,”https://www.verywellhealth.com/brain-aneurysm-prognosis-3146347, 2018, [Online; accessed 04 Jan. 2019].[5] A. Molyneux, R. Kerr, International Subarachnoid AneurysmTrial (ISAT) Collaborative Group, et al., “Internationalsubarachnoid aneurysm trial (isat) of neurosurgical clippingversus endovascular coiling in 2143 patients with rupturedintracranial aneurysms: a randomised trial,” Journal of strokeand cerebrovascular diseases, vol. 11, no. 6, pp. 304–314,October 2002.[6] B.M. Kim, D.J. Kim, and D.I. Kim, “Stent application forthe treatment of cerebral aneurysms,” Neurointervention, vol.6, pp. 53–70, August 2011.[7] M.D. Mazzeo and P.V. Coveney, “HemeLB: A high perfor-mance parallel lattice-Boltzmann code for large scale fluidflow in complex geometries,” Computer Physics Communi-cations, vol. 178, pp. 894914, June 2008.[8] X. Zhai, A. Amira, F. Bensaali, A. Al-Shibani, A. Al-Nassr,A. El-Sayed, M. Eslami, S. Dakua, and J. Abinahed, “ZynqSoC based acceleration of the lattice-Boltzmann method,”Concurrency and Computation: Practice and Experience, p.e5184, 2019.[9] D. Groen, J. Hetherington, H.B. Carver, R.W. Nash, M.O.Bernabeu, and P.V. Coveney, “Analysing and modelling theperformance of the HemeLB lattice-Boltzmann simulationenvironment,” Journal of Computational Science, vol. 4, pp.412–422, September 2013.[10] A. Patronis, R.A. Richardson, S. Schmieschek, B.J.N Wylie,R.W. Nash, and P.V. Coveney, “Modelling patient-specificmagnetic drug targeting within the intracranial vasculature,”Frontiers in physiology, vol. 9, pp. 331, April 2018.[11] D. Groen, R.A. Richardson, R. Coy, U.D. Schiller, H. Chan-drashekar, F. Robertson, and P.V. Coveney, “Validation ofpatient-specific cerebral blood flow simulation using transcra-nial doppler measurements,” Frontiers in Physiology, vol. 9,pp. 721, June 2018.[12] M.A. Suchard, Q. Wang, C. Chan, J. Frelinger, A. Cron, andM. West, “Understanding GPU programming for statisticalcomputation: Studies in massively parallel massive mixtures,”Journal of computational and graphical statistics, vol. 19, no.2, pp. 419–438, January 2010.[13] X. Zhai, M. Eslami, E.S. Hussein, M.S. Filali, S.T. Shalaby,A. Amira, F. Bensaali, S. Dakua, J. Abinahed, A. Al-Ansari,et al., “Real-time automated image segmentation techniquefor cerebral aneurysm on reconfigurable system-on-chip,”Journal of Computational Science, vol. 27, pp. 35–45, July2018.[14] Hamza Djelouat, Xiaojun Zhai, Mohamed Al Disi, AbbesAmira, and Faycal Bensaali, “System-on-chip solution forpatients biometric: A compressive sensing-based approach,”IEEE Sensors Journal, vol. 18, no. 23, pp. 9629–9639, 2018.[15] A.J.C. Ladd, “Numerical simulations of particulate suspen-sions via a discretized Boltzmann equation. part 1. theoreticalfoundation,” Journal of fluid mechanics, vol. 271, pp. 285–309, July 1994.[16] M. Bouzidi, M. Firdaouss, and P. Lallemand, “Momentumtransfer of a lattice-Boltzmann fluid with boundaries,” Physicsof fluids, vol. 13, pp. 3452–3459, November 2001.[17] Exxact Corporation, “Exxact Tensor TWS-289059-DPN spec-ification,” https://www.exxactcorp.com/Exxact-TWS-289059-DPN-E289059, 2018, [Online; accessed 25 Dec. 2018].[18] UCL, “HemeLB Github repository,”https://github.com/UCL/HemeLB, 2018, [Online; accessed10 Oct. 2018].[19] K. Zhou, Z. Ren, S. Lin, H. Bao, B. Guo, and H.Y. Shum,“Real-time smoke rendering using compensated ray march-ing,” in ACM Transactions on Graphics (TOG). ACM, August2008, vol. 27, p. 36.

HEMELB Acceleration and Visualization for Cerebral Aneurysms

Crossref

Hemelb Acceleration and Visualization for Cerebral Aneurysms

A weakness in the wall of a cerebral artery causing a dilation or ballooning of the blood vessel is known as a cerebral aneurysm. Optimal treatment requires fast and accurate diagnosis of the aneurysm. HemeLB is a fluid dynamics solver for complex geometries developed to provide neurosurgeons with information related to the flow of blood in and around aneurysms. On a cost efficient platform, HemeLB could be employed in hospitals to provide surgeons with the simulation results in real-time. In this work, we developed an improved version of HemeLB for GPU implementation and result visualization. A visualization platform for smooth interaction with end users is also presented. Finally, a comprehensive evaluation of this implementation is reported. The results demonstrate that the proposed implementation achieves a maximum performance of 15,168,964 site updates per second, and is capable of speeding up HemeLB for deployment in hospitals and clinical investigations.</p

Soheilian Esfahani, Sahar

Zhao, Xiaojun

Richardson, Robin A.

Coveney, Peter V.

Huddersfield Research Portal

HEMELB ACCELERATION AND VISUALIZATION FOR CEREBRAL ANEURYSMSahar Soheilian Esfahani? Xiaojun Zhai† Minsi Chen‡ Abbes Amira? Faycal Bensaali?Julien AbiNahed§ Sarada Dakua§ Georges Younes§ Robin Richardson‖ Peter V. Coveney‖? College of Engineering, Qatar University, Doha, Qatar† School of CS and EE, University of Essex, Colchester, UK‡ Department of Computer Science, University of Huddersfield, Huddersfield, UK§ Department of Surgery, Hamad Medical Corporation, Doha, Qatar‖ Centre for Computational Science, University College London, London, UKABSTRACTA weakness in the wall of a cerebral artery will cause a di-lation or blood vessel ballooning which is called a cerebralaneurysm. The best treatment for the patients needs fast andaccurate diagnosis of the aneurysm. HemeLB is a fluid flowsimulation environment in complex geometries developed tohelp neurosurgeons with the aneurysm blood flow related in-formation, therefore, a cost efficient platform for HemeLBimplementation can be employed in hospitals to provide sur-geons with the simulation results in real-time. In this work,an improved version of HemeLB for GPU implementationand result visualization is developed. A graphical user inter-face for smooth interaction with end users is also presented.Finally, a comprehensive evaluation of this implementationis reported. The obtained results demonstrate that the pro-posed implementation achieves a maximum performance of15,168,964 site updates per second and is capable to speed upHemeLB for deployment in hospitals and clinical investiga-tions.Index Terms— Cerebral aneurysm, HemeLB, Visualiza-tion, GPU1. INTRODUCTIONCerebral aneurysm is an abnormal focal dilation of a brainartery caused by a weakness in blood vessel wall [1]. Thenumber of patients who suffer from cerebrovascular disorderslike cerebral aneurysm is growing in non-developed countries[2]. A quick review of the statistics shows that people who al-ready have or will develop brain aneurysm are between 1.5%to 5% of the general population [3]. Another study shows thatabout 5% of patients with growing cerebral aneurysm and 0.5-1.1% of those with non-growing brain aneurysms will sufferfrom rupturing [4].This paper was made possible by National Priorities Research Program(NPRP) grant No. 5-792-2-328 from the Qatar National Research Fund (amember of Qatar Foundation). The statements made herein are solely theresponsibility of the authors.An effective treatment of aneurysms is endovascular ap-proaches which reduce the operative risks, pains and cost ofhospitalization [5]. These methods which take advantages ofintra-aneurysmal coils, may fail due to lack of informationabout the aneurysm status. In recent years, combination ofcoils with stents has been widely used as an efficient treat-ment which reorganizes the blood flow into and around theaneurysm [6]. In order to get the best treatment, the interven-tional radiologist has to identify the vascular geometry andestimate cerebral blood flow behavior such as pressure andvelocity from 2D and/or 3D images. Presently, there are fewapproaches to measure these flows and related data intraoper-atively. Consequently, diagnosis and treatment of aneurysmsare highly dependent on the experience and skill of the radi-ologist.Modeling and simulation of fluid flow and hemodynam-ics can provide clinicians with more precise analysis and thusdiagnosis of aneurysm. Simulation of fluid flow in large scaleand complex geometries requires physical model, substantialcomputing resources as well as efficient software. Recently,Lattice-Boltzmann (LB) method is widely used for simulationof fluid flows. LB was developed for time-dependent simula-tion of large systems by using a parallel implementation [7].In [8], the efficient hardware architecture of LB method isintroduced. HemeLB (Hemodynamic lattice-Boltzmann) is amassively parallel LB fluid solver developed to simulate fluidflows in sparse and complex systems on large supercomput-ing resources [7, 9, 10]. The aim of designing HemeLB wasto provide neurosurgeons with timely and clinically relevantassistance [11]. HemeLB blood flow simulation and its per-formance have been deployed on High Performance Comput-ing (HPC) platforms including HECToR, Blue Waters, Su-perMUC and ARCHER [9, 10, 11]. It is required to optimizeHemeLB on dedicated computational infrastructures in orderto employ the software in clinical environments. MulticoreGeneral Purpose Graphical Processing Units (GPGPUs) andField Programmable Gate Arrays (FPGAs) are probably to-days most powerful computational hardware in various ap-plications such as machine learning and artificial intelligence[12, 13].The objective of this paper is to develop and evaluate avisualized version of HemeLB on a workstation with CUDA(Compute Unified Device Architecture) capable GPUs. Toachieve this, an optimization of the original HemeLB for GPUimplementation is introduced. In order to evaluate the pro-posed implementation, a set of comprehensive tests using realpatients data has been carried out on Exxact Tensor worksta-tion with four NVIDIA GPUs.The rest of the paper is organized as follows. In Section2, HemeLB lattice-Boltzmann model is described. Section3 presents HemeLB visualization. The experimental resultsand their evaluations are reported in Section 4 followed bythe conclusions in Section 5.2. HEMELB MODELLB is a fast technique for simulation of large and com-plex fluid systems by means of parallel implementations [7].HemeLB code applies parallelized LB method as a fluidsolver, which represents fluid with particles. In this methodit is assumed that the propagation and collision processesof particles are done over a discrete lattice mesh. In theproposed system, three dimensional lattice with 19 discretevelocity directions (D3Q19) is used. Figure 1 represents theD3Q19 model.123456789 10111213 14015161817Fig. 1: Lattice node of D3Q19 model [?].HemeLB is implemented with different Boundary Condi-tions (BC) [7], such as Ladd iolets for velocity inlet BC [14]and Bouzidi-Firdaouss-Lallemand (BFL) for the interpolatedwall collision BC [15]. By the use of topology-aware two-level domain decomposition, HemeLB provides a good work-load distribution for parallel implementation. In addition, theimprovements made in HemeLB decrease redundant opera-tions, improve pattern regularity and enhance intra-machinecommunications [9].3. HEMELB IMPLEMENTATIONThe benefits of GPU-oriented solutions such as processingspeed and large datasets analyzing are achieved by the useof thousands of processor cores in a single chip [12]. In thefollowing section, the real-time visualization of HemeLB onShared MemoryPeripherals InterfaceGPU1MemoryGPU2MemoryGPU3MemoryGPU4MemoryCPUFig. 2: Simplified architecture of Exxact Tensor TWS-289059-DPN workstation.Exxact Tensor workstation with four CUDA capable GPUs isdiscussed.3.1. Architecture of Exxact Tensor workstationThe proposed work is implemented on Exxact Tensor TWS-289059-DPN workstation which is known as deep learningNVIDIA GPU solution [16]. The platform consists of oneIntel core i7-5960X processor, 4 TB HDD, 1 TB SSD as wellas four GeForce GTX 1080 Ti GPUs. Each NVIDIA GPUbenefits of 11 GB GDDR5X random access memory with 484GB/s memory bandwidth and 3584 CUDA cores. The GPUbase clock speed is 1481 MHz and the memory is running at1376 MHz. Figure 2 shows the simplified architecture of theworkstation with four NVIDIA GPUs.3.2. HemeLB configurationThe original version of HemeLB code is available in [17]. Theopenmpi package and CUDA platform are required for theimplementation of the proposed system. Once the packageis built, compilation flags given in Table 1 are needed to beset as boundary conditions. Table 2 represents the simulation,geometry and inlets/outlets settings used in the HemeLB.Table 1: HemeLB boundary settings.cmake settings ValuesHEMELB INLET BOUNDARY LADDIOLETHEMELB WALL INLET BOUNDARY LADDIOLETBFLHEMELB USE VELOCITY WEIGHTS FILE ONHEMELB WALL BOUNDARY BFL3.3. HemeLB visualization3.3.1. HemeLB interfaceThe original HemeLB was designed for command line pro-fessional users. In order to enable clinicians to run HemeLBTable 2: HemeLB simulation, geometry and inlets/outletssettings.HemeLB settings ValuesStep length 2× 10−5 sVoxel size 100× 10−6Inlet velocity fileInlet radius 1.9× 10−3Outlet amplitude 0.0 mmHgOutlet phase 0.0 radOutlet period 1.0 sconveniently, a Graphical User Interface (GUI) has been de-veloped. It provides end users with a smooth interaction en-vironment for running the software. It allows clinicians andneurosurgeons to steer the visualization and manage the sim-ulation operations. Figure 3 illustrates the main GUI windowdesigned with Qt cross-platform framework.Fig. 3: Main GUI window designed for HemeLB visualiza-tion.The GUI helps clinicians and neurosurgeons to launchHemeLB by a simple click. First, the user should upload theconfiguration file of the subject under examination, then setsome input settings such as simulation mode and renderingnumber. This GUI gives the control of the simulation modeto the clinicians. The user can choose single core or multicoresimulation mode before running the software. The number ofcores in multicore mode can also be selected by the end user.Another option is the number of rendering which can be cho-sen from a list or entered by the clinician. Finally, the simulateand interrupt buttons give the control of simulation operationto the user. The flowchart of the designed GUI is depicted inFigure 4.3.3.2. Real time visualizationVisualization of HemeLB is done on devoted CUDA capableGPUs. The LB nodes and the visualization client communi-cate by the existing message passing interface in HemeLB.Figure 5 shows the architecture of the proposed framework.In Figure 5, Lattice properties are computed and cashed ineach compute node then transferred to RANK0 node. Thisnode schedules lattice data transfer from the compute nodesLaunch the GUILoad the config fileSelect the simulation modeDetermine the number of renderingStart the simulationView the resultsSteer the visualization parametersEndStartInterrupt the simulationFig. 4: Flowchart of HemeLB GUI.and handles the view steering. The incoming lattice data isshared with GPU nodes and rendering is performed at thesame time.The three-tier architecture of the visualization client ispresented in Figure 6. The first tier is OpenGL applicationand the middle one is the host layer which stores the applica-tion data such as steering parameters and cached lattice prop-erties. Eventually, all the voxels for direct volume renderingare stored in the rendering layer which is depicted as the toptier. The CUDA cores in the rendering layer perform the vol-ume rendering kernel based on the ray marching algorithm[18].GPUsCPUsInterconnectionsMessage Passing InterfaceCompute NodeLBMCompute NodeLBMCompute NodeLBMCompute NodeLBM?.?.Fig. 5: Architecture of proposed visualization framework.CUDA CoresVolume Render KernelGPU MemoryView ParamsTransfer Function Texture3D Texture UnitCPU CoresLBM RoutinesHost MemoryViz Steering ParamsCached Lattice PropertiesOpenGL Visualisation FrontendFig. 6: The three-tier internal architecture of visualizationclient.It is well known that the number of memory transfers iscritical in real-time applications. In HemeLB visualization,the memory transfers include the transmission of lattice datato the visualization volume and a number of steering simu-lation. In order to optimize the memory transfers in the pro-posed framework, the following two-level memory access so-lution is deployed. In the first place, the visualization volumeis stored in a 3D texture unit to enable the fast voxel valuessampling. Secondly, in order to store the viewing parame-ters and the look-up table, the GPU constant memory buffersare used. This allows Direct Memory Access (DMA) whichperformance is comparable to reading from registers.The steerable parameters introduced in the visualizedHemeLB include zooming, model rotation and adjustment ofscaling and offset of the transfer functions.4. EXPERIMENTAL RESULTS AND DISCUSSIONFive subjects of 3D Rotational Angiography (3DRA) selectedby Hamad Medical Cooperation (HMC) clinicians were usedto evaluate the proposed solution. Figure 7 represents twoSTereoLithograph (STL) files used to evaluate HemeLB vi-sualization. In Table 3 the number of lattice sites and blocksfor each subject are presented. HemeLB was employed usingD3Q19 lattice node model with simple bounce-back bound-ary condition and a fixed physical viscosity of 0.004 Pa.s. Theinlet velocities applied to all STLs in the simulation is illus-trated in Figure 8.(a) Subject 16 (b) Subject 23Fig. 7: STL files used for evaluation of visualized HemeLB.Table 3: Overview of simulation domains.Name Number of lattice sites Number of blocksSubject 16 296,814 15,000Subject 23 132,910 5,320Table 4 shows the timing performance of simulating twosubjects on the Exxact Tensor workstation. The maximumperformance of 15,168,964 Site Updates Per Second (SUPS)is achieved in this evaluation. As it is depicted in Table 4, thelarger geometry has the highest SUPS, although the timingperformance for different subjects is similar. This insures thatthe SUPS per core is independent of other factors. The sim-ulation results of two geometries obtained from the proposedversion of HemeLB are illustrated in Figure 9.Fig. 8: The inlet blood velocity applied to STLs.Table 4: Simulation timing data.Name Total time (s) Number of steps SUPSSubject 16 906 46,302 15,168,964Subject 23 297 30,558 13,674,9625. CONCLUSIONIn this paper, a solution for designing and implementing areal-time visualization version of HemeLB on a workstationis presented. The proposed implementation enables HemeLBto be exploited locally in hospitals rather than a distributedenvironment. In addition, a GUI for HemeLB environmentis also introduced in this work. The GUI allows cliniciansto launch the HemeLB, view and steer the visualized re-sults in a user friendly and smooth manner. The reportedresults in this experience, obtained from comprehensive testsperformed on real patient data. The results achieve a maxi-mum performance of 15,168,964 site updates per second anddemonstrate that the proposed implementation is capable tosupport HemeLB in clinical environments. A working systemwhich can be used in hospitals for training and practicing thecerebral surgeries can be developed as a future work.(a) Subject 16 (b) Subject 23Fig. 9: Results of HemeLB visualization framework.6. REFERENCES[1] R.T. Higashida, “What you should know about cere-bral aneurysms,” Pamphlet. American Heart Associa-tion Cardiovascular Council, 2003.[2] N. Benkirane, A. Bennis, S. Bellakhdar, R. Habbal,S. Aidi, M. El Alaoui Faris, H. El Otmani, B. ElMoutawakil, M.A. Rafai, and I. Slassi, “Stroke risk fac-tors in a moroccan population: a multicentric prospec-tive study,” Journal of the Neurological Sciences, vol.357, pp. e367, October 2015.[3] T. Mashiko, K. Otani, R. Kawano, T. Konno, N. Kaneko,Y. Ito, and E. Watanabe, “Development of three-dimensional hollow elastic model for cerebral aneurysmclipping simulation enabling rapid and low cost proto-typing,” World Neurosurgery, vol. 83, pp. 351–361,March 2015.[4] Verywell health, “Brain aneurysm prognosis,”https://www.verywellhealth.com/brain-aneurysm-prognosis-3146347, 2018, [Online; accessed 04 Jan.2019].[5] A. Molyneux, R. Kerr, International SubarachnoidAneurysm Trial (ISAT) Collaborative Group, et al., “In-ternational subarachnoid aneurysm trial (isat) of neuro-surgical clipping versus endovascular coiling in 2143patients with ruptured intracranial aneurysms: a ran-domised trial,” Journal of stroke and cerebrovasculardiseases, vol. 11, no. 6, pp. 304–314, October 2002.[6] B.M. Kim, D.J. Kim, and D.I. Kim, “Stent applicationfor the treatment of cerebral aneurysms,” Neurointer-vention, vol. 6, pp. 53–70, August 2011.[7] M.D. Mazzeo and P.V. Coveney, “HemeLB: A high per-formance parallel lattice-Boltzmann code for large scalefluid flow in complex geometries,” Computer PhysicsCommunications, vol. 178, pp. 894914, June 2008.[8] X. Zhai, A. Amira, F. Bensaali, A. Al-Shibani, A. Al-Nassr, A. El-Sayed, M. Eslami, S. Dakua, and J. Abi-nahed, “Zynq SoC based acceleration of the lattice-Boltzmann method,” Concurrency and Computation:Practice and Experience, in press.[9] D. Groen, J. Hetherington, H.B. Carver, R.W. Nash,M.O. Bernabeu, and P.V. Coveney, “Analysing andmodelling the performance of the HemeLB lattice-Boltzmann simulation environment,” Journal of Com-putational Science, vol. 4, pp. 412–422, September2013.[10] A. Patronis, R.A. Richardson, S. Schmieschek, B.J.NWylie, R.W. Nash, and P.V. Coveney, “Modellingpatient-specific magnetic drug targeting within the in-tracranial vasculature,” Frontiers in physiology, vol. 9,pp. 331, April 2018.[11] D. Groen, R.A. Richardson, R. Coy, U.D. Schiller,H. Chandrashekar, F. Robertson, and P.V. Coveney,“Validation of patient-specific cerebral blood flow simu-lation using transcranial doppler measurements,” Fron-tiers in Physiology, vol. 9, pp. 721, June 2018.[12] M.A. Suchard, Q. Wang, C. Chan, J. Frelinger, A. Cron,and M. West, “Understanding GPU programmingfor statistical computation: Studies in massively paral-lel massive mixtures,” Journal of computational andgraphical statistics, vol. 19, no. 2, pp. 419–438, January2010.[13] X. Zhai, M. Eslami, E.S. Hussein, M.S. Filali, S.T. Sha-laby, A. Amira, F. Bensaali, S. Dakua, J. Abinahed,A. Al-Ansari, et al., “Real-time automated image seg-mentation technique for cerebral aneurysm on reconfig-urable system-on-chip,” Journal of Computational Sci-ence, vol. 27, pp. 35–45, July 2018.[14] A.J.C. Ladd, “Numerical simulations of particulate sus-pensions via a discretized Boltzmann equation. part 1.theoretical foundation,” Journal of fluid mechanics, vol.271, pp. 285–309, July 1994.[15] M. Bouzidi, M. Firdaouss, and P. Lallemand, “Momen-tum transfer of a lattice-Boltzmann fluid with bound-aries,” Physics of fluids, vol. 13, pp. 3452–3459,November 2001.[16] Exxact Corporation, “Exxact Ten-sor TWS-289059-DPN specification,”https://www.exxactcorp.com/Exxact-TWS-289059-DPN-E289059, 2018, [Online; accessed 25 Dec.2018].[17] UCL, “HemeLB Github repository,”https://github.com/UCL/HemeLB, 2018, [Online;accessed 10 Oct. 2018].[18] K. Zhou, Z. Ren, S. Lin, H. Bao, B. Guo, and H.Y.Shum, “Real-time smoke rendering using compen-sated ray marching,” in ACM Transactions on Graphics(TOG). ACM, August 2008, vol. 27, p. 36.

English

A weakness in the wall of a cerebral artery causing a dilation or ballooning of the blood vessel is known as a cerebral aneurysm. Optimal treatment requires fast and accurate diagnosis of the aneurysm. HemeLB is a fluid dynamics solver for complex geometries developed to provide neurosurgeons with information related to the flow of blood in and around aneurysms. On a cost efficient platform, HemeLB could be employed in hospitals to provide surgeons with the simulation results in real-time. In this work, we developed an improved version of HemeLB for GPU implementation and result visualization. A visualization platform for smooth interaction with end users is also presented. Finally, a comprehensive evaluation of this implementation is reported. The results demonstrate that the proposed implementation achieves a maximum performance of 15,168,964 site updates per second, and is capable of speeding up HemeLB for deployment in hospitals and clinical investigations. - 2019 IEEE.This paper was made possible by National Priorities Research Program (NPRP) grant No. 5-792-2-328 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors.Scopu

Esfahani S.S.

Zhai X.

Chen M.

Amira A.

Bensaali F.

Abinahed J.

Dakua S.

Younes G.

Richardson R.A.

Coveney P.V.

Qatar University Institutional Repository

A weakness in the wall of a cerebral artery causing a dilation or ballooning of the blood vessel is known as a cerebral aneurysm. Optimal treatment requires fast and accurate diagnosis of the aneurysm. HemeLB is a fluid dynamics solver for complex geometries developed to provide neurosurgeons with information related to the flow of blood in and around aneurysms. On a cost efficient platform, HemeLB could be employed in hospitals to provide surgeons with the simulation results in real-time. In this work, we developed an improved version of HemeLB for GPU implementation and result visualization. A visualization platform for smooth interaction with end users is also presented. Finally, a comprehensive evaluation of this implementation is reported. The results demonstrate that the proposed implementation achieves a maximum performance of 15,168,964 site updates per second, and is capable of speeding up HemeLB for deployment in hospitals and clinical investigations

HEMELB Acceleration and Visualization for Cerebral Aneurysms

Abstract

Similar works

Full text

Available Versions

University of Essex Research Repository

Crossref

Huddersfield Research Portal

Qatar University Institutional Repository

UCL Discovery