The task consisting on estimating the translation, rotation and scaling of an image with respect to another take of the same scene obtained at different times, viewpoints and/or lightning conditions is known as image registration. Applications like environmental disasters management or rescue operations depend on real-time hyperspectral images registration, but most of the current FFT-based techniques ignore such performance needs. Ordóñez et al. proposed HYFMGPU [1], a single-GPU algorithm whose performance makes it suitable for real-time use cases. As hyperspectral sensors improve, both the size of images and the wavelength ranges covered are expected to increase, so that a multi-GPU implementation is proposed to satisfy such growing needs.Universidad de Valladolid (Consejería de Educación of Junta de Castilla y León, Ministerio de Economía, Industria y Competitividad of Spain, and European Regional Development Fund (ERDF) program: Project PCAS (TIN2017-88614-R), Project PROPHET (VA082P17) and CAPAP-H6 network (TIN2016-81840-REDT)

Blanco Heras, Dora

Fernández Fabeiro, Jorge

González Escribano, Arturo

Ordóñez, Álvaro

Spanish

Repositorio Documental de la Universidad de Valladolid

Proceedings of the 18th International Conferenceon Computational and Mathematical Methodsin Science and Engineering, CMMSE 2018July 9–14, 2018.Towards a Multi-device Version of the HYFMGPUAlgorithm for Hyperspectral Scenes RegistrationJorge Fernández-Fabeiro1, Álvaro Ordóñez2, Arturo González-Escribano1and Dora B. Heras21 Departamento de Informática, University of Valladolid, Spain2 Centro Singular de Investigación en Tecnolox́ıas da Información (CiTIUS),Universidade de Santiago de Compostela, Spainemails: jorge@infor.uva.es, alvaro.ordonez@usc.es, arturo@infor.uva.es,dora.blanco@usc.esAbstractHyperspectral image registration is a relevant task for real-time applications likeenvironmental disasters management or search and rescue scenarios. Traditional algo-rithms were not devoted to real-time performance, the HYFMGPU algorithm havingarisen as a solution to such a lack. Sensors are expected to evolve and thus generateimages with finer resolutions and wider wavelength ranges, so a multi-GPU implementa-tion seems to be necessary in a near future. This work presents a first approach to sucha multi-device version, identifying some stages of the pipeline as the most suitable torun in parallel in several GPUs. An MPI+CUDA variation of the original HYFMGPUalgorithm is implemented, achieving speedups of 1.83× in 2 GPUs and 3.08× in 4GPUs for the stages of the pipeline distributed among several devices. Different issuesrelated to communications-derived time overloads and to some CUDA-based librariesparticularities, as long as some optimization possibilities out of the currently distributedstages, were also detected. We plan to tackle them in further development stages of thismulti-GPU implementation.Key words: Hyperspectral imaging, image registration, Fourier transforms, multi-GPU, CUDA, OpenMP, remote sensing.c©CMMSE ISBN: 978-84-697-7861-6Towards a Multi-GPU Version of the HYFMGPU Algorithm1 IntroductionThe task consisting on estimating the translation, rotation and scaling parameters of a givenimage with respect to a second take of the same scene obtained at different times, view-points and/or lightning conditions is known as image registration. During the last years,a whole family of FFT-based image registration techniques appeared, but most of themignored time performance. However, many real-time applications such as the managementof natural disasters or surveillance operations depend on hyperspectral images being pro-cessed in real-time. GPUs were used to boost tasks like classification, target detection orsegmentation of this kind of images, but few efforts were made to achieve a real-time im-plementation of a hyperspectral registration algorithm. In [1], Ordóñez et al. introducedHYFM, a Fourier-Mellin algorithm for hyperspectral images registration, and implementeda sequential CPU version of it. That work was followed by HYFMGPU [2], a single-GPUversion whose performance makes it suitable to be used in real-time environments. Never-theless, as both the image resolution and the wavelength range of the sensors that capturehyperspectral images improve, both the size of images and the number of bands to be pro-cessed is expected to increase. In this work we present the first steps taken to achieve amulti-GPU implementation able to satisfy these needs.The rest of this extended abstract is organized as follows: Section 2 summarises theHYFMGPU algorithm, describing then in Section 3 the first approach to a multi-GPUversion of it. The preliminary results obtained by this approach are introduced in Section 4,and finally in Section 5 we present the conclusions and some brief future research lines.2 The HYFMGPU algorithmThe HYFMGPU algorithm expects as input a pair of hyperspectral images (reference andtarget), the goal being to register the target image, this is, to compute how it is rotated,shifted and scaled with respect to the reference image. The algorithm comprises six mainstages, which are depicted in Figure 1.Figure 1: HYFM scheme for registration of two hyperspectral imagesc©CMMSE ISBN: 978-84-697-7861-6J. Fernández-Fabeiro, Á. Ordóñez, A. González-Escribano, D. B. HerasThe HYFMGPU algorithm projects these stages to a single GPU by means of a CUDAimplementation that relies on some specific libraries, namely cuBLAS, cuSOLVER, cuFFT,NVIDIA Performance Primitives (NPP), and Thrust. This implementation can be roughlydecomposed in the following steps:1. Initialization: Input images are loaded in global memory, and some arrays neededby further steps are also loaded and/or computed.2. Preprocessing: In Stage I, a single-kernel Blackman filter is applied first to bothinputs to remove higher frequencies, which might be detrimental for the precision ofthe registration. Those filtered images are centered, and then a principal componentanalysis (PCA) is applied to each of them in order to extract their most relevantfeatures by retaining a reduced number of principal components (i.e., transformedbands of the original images). Both cuBLAS and cuSOLVER operations are used toimplement this analysis. Finally, sizes of both reduced images are expanded to thenearest common upper power of 2, and data is transformed to complex values.3. Selected bands processing and composition: In this step, pairs composed of thesame PCA-extracted band from both inputs are processed by computing a high-passfilter (Stage II), a log-polar map (Stage III), and a phase correlation (Stage IV). Asthis three substeps are FFT-based, the underlying operations are implemented bymeans of the cuFFT library. Since this is a single-GPU implementation, the device iscommanded to iterate over all the band pairs in order to perform this stage. Finally,a single-kernel reduction (Stage V) is performed in the device in order to average thelog-polar correlated maps obtained for each pair of PCA-extracted bands.4. Peak processing: The log-polar peaks contained in the average map computed inthe previous step are descendently sorted in the device using the Thrust library, select-ing a given number and processing them one-by-one. This process starts by rotatingand scaling the first component of the target image several times using specific func-tions from the NPP library. Next, a phase correlation and a cuBLAS-based maximumsearch are performed on the cartesian grid to determine the correct angle and trans-lation parameters. Finally, the highest peak of all the cartesian grids is selected, as itscoordinates determine the shift parameters. In turn, their log-polar counterparts de-cide the scale factor and the rotation angle. All these values are the expected outputof the Stage VI of the pipeline.3 Proposal of multi-GPU parallelizationA quick glance at the HYFM pipeline depicted in Figure 1 reveals the workload comprisingStages II, III and IV as a very interesting candidate to a multi-device coarse-grain par-c©CMMSE ISBN: 978-84-697-7861-6Towards a Multi-GPU Version of the HYFMGPU Algorithmallelization. Notice how this workload, described in the step 3 of the single-GPU CUDAimplementation introduced in Section 2, is performed by a loop that iterates over each pairof bands extracted from reference and target images.Initial OpenMP+CUDA approachIn a very initial approach we opted to implement a OpenMP-threaded version of theHYFMGPU host program, keeping a master thread to control the single-GPU parts ofthe multi-device implementation (Steps 1, 2 and 4), and defining a parallel section to com-mand each GPU to run its part of the Step 3. So, once the master thread loads the images, itcommands its GPU to apply the Blackman filter, run the PCA, and distribute the extractedcomponents equally among the GPUs available. Once the program enters the parallel sec-tion, each thread commands its GPU to convert its subset of the PCA-extracted bands tocomplex type and perform over them in a loop Stages II, III and IV of the pipeline. After theprogram exits this parallel section, the master threads takes the control back, averaging thepartial log-polar maps computed by each GPU (Stage V) and processing the peaks (StageVI) in order to get the final shift, scale and angle outputs expected after this last stage.As Step 3 of the CUDA implementation description from Section 2 shows, the cuFFTlibrary is used to process a the PCA-extracted bands. Because of that, each GPUs needs itsown set of FFT execution plans, whose initialization must be invoked from the correspond-ing OpenMP thread. The first tests showed that despite having distributed the workloadamong several GPUs, the execution time of the Step 3 was multiplied by the number ofdevices, instead of being divided as expected. So, the way how CUDA and OpenMP in-teract when threads are trying to prepare their corresponding cuFFT plans makes theseinitialization operations to be eventually sequentialized. As Lončar et al. explain in [3],combining FFT operations and OpenMP-threaded parallelizations is not a trivial task.They also recommend to use FFTW3 [4] in such cases, but there is no support for this FFTlibrary in CUDA. To solve this cuFFT issue, the implementation we have just describedwas shifted from forking multiple OpenMP threads to run separated MPI processes. Thisway, the FFT plans needed by each GPU were prepared in different processes with theirown memory spaces, which made the sequentialization problem disappear.MPI+CUDA versionIn this MPI-based evolution, a number of processes equal to the number of GPUs islaunched. Since MPI follows a SPMD model, some noticeable variations were introduced inthe workflow of the host program. Now each process loads its own full copy of the images,and commands its GPU to apply the Blackman filter and run the PCA on them. Despitehaving the all the PCA-extracted components available, each GPU is forced to perform theStages II, III and IV of the pipeline just on its own bands subset. In this case, the interme-c©CMMSE ISBN: 978-84-697-7861-6J. Fernández-Fabeiro, Á. Ordóñez, A. González-Escribano, D. B. Herasdiate log-polar maps processed by each GPU are gathered by means of MPI synchronousmessages in the process controlling the GPU 0, which composes them in a final log-polargrid (Stage V) and explore the peaks (Stage VI) to compute the final result.4 Experimental resultsAll tests were run in a node equipped with a dual-socket host CPU composed of 2 Intel XeonE5-2609v3 (1.9 GHz, 6 cores each) with 64 GB of RAM, and 4 GPUs NVIDIA GeForceGTX TITAN Black (GK110B architecture, compute capability 3.5, 15 SMs with 192 CUDAcores each, up to 2880) controlled by the 384.59 driver. The CUDA code has been compiledunder Linux using nvcc from CUDA Toolkit 9.0, as well as the libraries used. Multi-threadedpreliminary tests were supported by linking libraries from OpenMP 3.1, whereas the MPIsupport was provided by mpich-3.2. The multi-GPU implementation was evaluated byregistering rotated and scaled variations of the Pavia University test case, a common singlereference hyperspectral image of 610×340 pixels and 103 bands already used in [2]. Thevalues of the parameters that control the registration algorithm were also the same as thosespecified in [2], this is: 8 bands to be extracted in the PCA, vector α = {1, 1/4, 1/16, 1/64}for the FFTs performed to obtain the log-polar maps, and 50 highest peaks to be examinedin the peak processing stage.VersionWall time in seconds (Speedup)Step 1 Step 2 Step 3 Step 4 Whole algorithmHYFMGPU 0.698 s 0.575 s 0.512 s 0.309 s 2.094 sMPI+2GPU 0.687 s 0.569 s 0.279 s (1.83×) 0.323 s 1.852 s (1.13×)MPI+4GPU 0.714 s 0.569 s 0.166 s (3.08×) 0.382 s 1.831 s (1.14×)Table 1: Wall times in seconds (and selected speedups) for each step and the whole HYFMalgorithm run for the Pavia University test case in NVIDIA GeForce Titan Black GPUsTable 1 shows the wall time consumed by the original single-GPU CUDA implemen-tation and the MPI+CUDA version when it is run in 2 and 4 GPUs to perform Steps 1to 4 from Section 2 individually, and to run the whole algorithm. Moreover, the speedupsachieved by the multi-GPU versions for both the Step 3 and the whole algorithm are shownin the corresponding entries of Step 3 and Whole algorithm columns. In relation to theFFTs performed in Step 3, it must be noticed that the cuFFT API allows the program-mer to disaggregate the creation of FFT execution plans [5] in three steps: (1) handlecreation, (2) estimation of sizes for temporary buffers, and (3) execution plan definition.We experimentally checked that the time consumed by handles creation and plans defini-tion was negligible with respect to the time needed to estimate the buffer sizes. Since thisestimation step does not depend on the number of PCA-extracted components processed,we moved it to the initialization part of the program and, hence, the FFT plans creationc©CMMSE ISBN: 978-84-697-7861-6Towards a Multi-GPU Version of the HYFMGPU Algorithmtime is ascribed to Step 1. Along the computation of high-pass filters, log-polar grids andphase correlations, times in Step 3 column also include the conversion to complex type ofthe extracted components and the calculation of some auxiliary buffers needed. It is alsoremarkable that the times shown in Step 4 column increases as the number of GPUs does,since the communication of the partial results back to the process acting as master is beingsummed up here.The speedups of 1.83× for 2 GPUs and 3.08× for 4 GPUs shown in Step 3 columnconfirm that the loop performing the MLFFT+LogPolar+Correlation stages was a suitablecandidate to be distributed among several GPUs, despite the time needed by additionalauxiliary buffer computations and data communications making them to distance from thetheoretical maximums of 2× and 4×, respectively. Furthermore, for the specific experimentsrun (Pavia University input images, GPUs with GK110B architecture and compute capa-bility 3.5), the wall time consumed by this step represented about a 25% of the wall timeof the whole algorithm run in a single GPU, so that Amdahl’s Law is limiting the overallspeedup achievable by distributing the workload of this step among 2 GPUs to 1.14×, andamong 4 GPUs up to 1.23×. Therefore, by now the multi-device version is achieving a 99%and a 93% of the theoretical maximum performance when run in 2 and 4 GPUs respectively.5 Conclusions and future workIn this work we have introduced an initial approach of a multi-device version of the originalCUDA implementation of the single-GPU HYFMGPU algorithm. The speedups of 1.83×for 2 GPUs and 3.08× for 4 GPUs obtained for the Step 3 from Section 2 confirmed it as asuitable candidate to be distributed among several devices.Furthermore, in the current experimentation conditions (Pavia University input im-ages, GPUs with GK110B architecture and compute capability 3.5), the wall time of thetasks currently distributed among several GPUs only represented about a 25% of the totalexecution time of the HYFM pipeline, so that finding more steps of the algorithm that couldbe parallelized in several GPUs is another interesting research line. For instance, once thepeaks are sorted at the beginning of Stage VI, they could be scattered among the GPUs.Then, each device would process a subset of the peaks locally and return its highest oneto the host program, which will compute the final result. Moreover, currently the PCAis performed by a single-GPU non-iterative procedure based on that one presented in [6].Namely, it would be quite interesting to implement a multi-GPU version of the PCA thatwas able not only to boost the component extraction part but also to make each subsetof bands to be directly stored in the global memory of the corresponding GPU. Also moreexperiments with larger images (regarding both space and bands) and in different devicesmust be run in order to evaluate how the experimentation conditions affect the weight thateach step has in the execution time of the algorithm.c©CMMSE ISBN: 978-84-697-7861-6J. Fernández-Fabeiro, Á. Ordóñez, A. González-Escribano, D. B. HerasFinally, although the usage of MPI instead of OpenMP helped to solve the cuFFTsequentialization issue, there is still a fixed time that must be consumed to estimate thesizes of the temporary buffers needed by the FFT execution plans. This inconvenience mightbe overcome by exploiting CUDA streamed execution, namely trying to hide the latency ofthese estimations among other CUDA and/or specific-purpose GPU libraries asynchronouscalls [5, 7, 8].AcknowledgementsThis work has been partially supported by:Universidad de Valladolid – Junta de Castilla y León, Ministry of Economy, Industry andCompetitiveness of Spain, and European Regional Development Fund (ERDF) program:Project PCAS (TIN2017-88614-R), Project PROPHET (VA082P17) and CAPAP-H6 net-work (TIN2016-81840-REDT).Universidade de Santiago de Compostela – Conselleŕıa de Cultura, Educación e OrdenaciónUniversitaria of Xunta de Galicia (grant numbers GRC2014/008 and ED431G/08) andMinistry of Economy, Industry and Competitiveness of Spain (grant number TIN2016-76373-P), all co–funded by the European Regional Development Fund (ERDF) program.References[1] Á. Ordóñez and F. Argüello and D. B. Heras, Fourier–Mellin registration oftwo hyperspectral images, International Journal of Remote Sensing, Volume 38, Number11 (2017), pp. 3253–3273.[2] Á. Ordóñez and F. Argüello and D. B. Heras, GPU Accelerated FFT-BasedRegistration of Hyperspectral Scenes, IEEE Journal of Selected Topics in Applied EarthObservations and Remote Sensing, Volume 10, Number 11 (2017), pp. 4869–4878.[3] V. Lončar and L. E. Young-S. and S. Škrbić and P. Muruganandam and S.K. Adhikari and A. Balaž, OpenMP, OpenMP/MPI, and CUDA/MPI C programsfor solving the time-dependent dipolar Gross–Pitaevskii equation, Computer PhysicsCommunications, Volume 209 (2016), pp. 190–196.[4] M. Frigo and S. G. Johnson, FFTW3 Library, Massachusetts Institute of Technol-ogy (MIT), available in http://www.fftw.org/.[5] NVIDIA Corporation, cuFFT Library User’s Guide, available in https://docs.nvidia.com/cuda/pdf/CUFFT_Library.pdfc©CMMSE ISBN: 978-84-697-7861-6Towards a Multi-GPU Version of the HYFMGPU Algorithm[6] A. S. Garea and D. B. Heras and F. Argüello, GPU classification of remote-sensing images using kernel ELM and extended morphological profiles, InternationalJournal of Remote Sensing, Volume 37, Number 24 (2016), pp. 5918–5935.[7] NVIDIA Corporation, cuBLAS Library User’s Guide, available in https://docs.nvidia.com/cuda/pdf/CUBLAS_Library.pdf[8] NVIDIA Corporation, cuSOLVER Library User’s Guide, available in https://docs.nvidia.com/cuda/pdf/CUSOLVER_Library.pdfc©CMMSE ISBN: 978-84-697-7861-6

CMMSE 2018

https://uvadoc.uva.es/bitstream/10324/32186/1/paper-01b-cmmse18-jorge-confpaper.pdf

CMMSE 2018

Abstract

Similar works

Full text

Available Versions

Repositorio Documental de la Universidad de Valladolid