Numerical solution of 3-D electromagnetic problems in exploration geophysics and its implementation on massively parallel computers

The growing significance, technical development and employment of electromagnetic (EM) methods in exploration geophysics have led to the increasing need for reliable and fast techniques of interpretation of 3-D EM data sets acquired in complex geological environments. The first and most important step to creating an inversion method is the development of a solver for the forward problem. In order to create an efficient, reliable and practical 3-D EM inversion, it is necessary to have a 3-D EM modelling code that is highly accurate, robust and very fast. This thesis focuses precisely on this crucial and very demanding step to building a 3-D EM interpretation method. The thesis presents as its main contribution a highly accurate, robust, very fast and extremely scalable numerical method for 3-D EM modelling in geophysics that is based on finite elements (FE) and designed to run on massively parallel computing platforms. Thanks to the fact that the FE approach supports completely unstructured tetrahedral meshes as well as local mesh refinements, the presented solver is able to represent complex geometries of subsurface structures very precisely and thus improve the solution accuracy and avoid misleading artefacts in images. Consequently, it can be successfully used in geological environments of arbitrary geometrical complexities. The parallel implementation of the method, which is based on the domain decomposition and a hybrid MPI-OpenMP scheme, has proved to be highly scalable - the achieved speed-up is close to the linear for more than a thousand processors. Thanks to this, the code is able to deal with extremely large problems, which may have hundreds of millions of degrees of freedom, in a very efficient way. The importance of having this forward-problem solver lies in the fact that it is now possible to create a 3-D EM inversion that can deal with data obtained in extremely complex geological environments in a way that is realistic for practical use in industry. So far, such imaging tool has not been proposed due to a lack of efficient, parallel FE solutions as well as the limitations of efficient solvers based on finite differences. In addition, the thesis discusses physical, mathematical and numerical aspects and challenges of 3-D EM modelling, which have been studied during my research in order to properly design the presented software for EM field simulations on 3-D areas of the Earth. Through this work, a physical problem formulation based on the secondary Coulomb-gauged EM potentials has been validated, proving that it can be successfully used with the standard nodal FE method to give highly accurate numerical solutions. Also, this work has shown that Krylov subspace iterative methods are the best solution for solving linear systems that arise after FE discretisation of the problem under consideration. More precisely, it has been discovered empirically that the best iterative method for this kind of problems is biconjugate gradient stabilised with an elaborate preconditioner. Since most commonly used preconditioners proved to be either unable to improve the convergence of the implemented solvers to the desired extent, or impractical in the parallel context, I have proposed a preconditioning technique for Krylov methods that is based on algebraic multigrid. Tests for various problems with different conductivity structures and characteristics have shown that the new preconditioner greatly improves the convergence of different Krylov subspace methods, which significantly reduces the total execution time of the program and improves the solution quality. Furthermore, the preconditioner is very practical for parallel implementation. Finally, it has been concluded that there are not any restrictions in employing classical parallel programming models, MPI and OpenMP, for parallelisation of the presented FE solver. Moreover, they have proved to be enough to provide an excellent scalability for it

Parallel 3-D marine controlled-source electromagnetic modelling using high-order tetrahedral Nédélec elements

We present a parallel and high-order Nédélec finite element solution for the marine controlled-source electromagnetic (CSEM) forward problem in 3-D media with isotropic conductivity. Our parallel Python code is implemented on unstructured tetrahedral meshes, which support multiple-scale structures and bathymetry for general marine 3-D CSEM modelling applications. Based on a primary/secondary field approach, we solve the diffusive form of Maxwell’s equations in the low-frequency domain. We investigate the accuracy and performance advantages of our new high-order algorithm against a low-order implementation proposed in our previous work. The numerical precision of our high-order method has been successfully verified by comparisons against previously published results that are relevant in terms of scale and geological properties. A convergence study confirms that high-order polynomials offer a better trade-off between accuracy and computation time. However, the optimum choice of the polynomial order depends on both the input model and the required accuracy as revealed by our tests. Also, we extend our adaptive-meshing strategy to high-order tetrahedral elements. Using adapted meshes to both physical parameters and high-order schemes, we are able to achieve a significant reduction in computational cost without sacrificing accuracy in the modelling. Furthermore, we demonstrate the excellent performance and quasi-linear scaling of our implementation in a state-of-the-art high-performance computing architecture.This project has received funding from the European Union's Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement No. 777778. Furthermore, the research leading to these results has received funding from the European Union's Horizon 2020 programme under the ChEESE Project (https://cheese-coe.eu/ ), grant agreement No. 823844. In addition, the authors would also like to thank the support of the Ministerio de Educación y Ciencia (Spain) under Projects TEC2016-80386-P and TIN2016-80957-P. The authors would like to thank the Editors-in-Chief and to both reviewers, Dr. Martin Cuma and Dr. Raphael Rochlitz, for their valuable comments and suggestions which helped to improve the quality of the manuscript. This work benefited from the valuable suggestions, comments, and proofreading of Dr. Otilio Rojas (BSC). Last but not least, Octavio Castillo-Reyes thanks Natalia Gutierrez (BSC) for her support in CSEM modeling with BSIT.Peer ReviewedPostprint (author's final draft

Multidisciplinary computational aerosciences

As the challenges of single disciplinary computational physics are met, such as computational fluid dynamics, computational structural mechanics, computational propulsion, computational aeroacoustics, computational electromagnetics, etc., scientists have begun investigating the combination of these single disciplines into what is being called multidisciplinary computational aerosciences (MCAS). The combination of several disciplines not only offers simulation realism but also formidable computational challenges. The solution of such problems will require computers orders of magnitude larger than those currently available. Such computer power can only be supplied by massively parallel machines because of the current speed-of-light limitation of conventional serial systems. Even with such machines, MCAS problems will require hundreds of hours for their solution. To efficiently utilize such a machine, research is required in three areas that include parallel architectures, systems software, and applications software. The main emphasis of this paper is the applications software element. Examples that demonstrate application software for multidisciplinary problems currently being solved at NASA Ames Research Center are presented. Pacing items for MCAS are discussed such as solution methodology, physical modeling, computer power, and multidisciplinary validation experiments

HPC geophysical electromagnetics: a synthetic VTI model with complex bathymetry

We introduce a new synthetic marine model for 3D controlled-source electromagnetic method (CSEM) surveys. The proposed model includes relevant features for the electromagnetic geophysical community such as large conductivity contrast with vertical transverse isotropy and a complex bathymetry profile. In this paper, we present the experimental setup and several 3D CSEM simulations in the presence of a resistivity unit denoting a hydrocarbon reservoir. We employ a parallel and high-order vector finite element routine to perform the CSEM simulations. By using tailored meshes, several scenarios are simulated to assess the influence of the reservoir unit presence on the electromagnetic responses. Our numerical assessment confirms that resistivity unit strongly influences the amplitude and phase of the electromagnetic measurements. We investigate the code performance for the solution of fundamental frequencies on high-performance computing architectures. Here, excellent performance ratios are obtained. Our benchmark model and its modeling results are developed under an open-source scheme that promotes easy access to data and reproducible solutions.The work of O.C-R., conducted in the frame of PIXIL project, has been 65% cofinanced by the European Regional Development Fund (ERDF) through the Interreg V-A SpainFrance-Andorra program (POCTEFA2014-2020). BSC authors have received funding from the European Union’s Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement N◦ 777778. Furthermore, the development of PETGEM has received funding from the European Union’s Horizon 2020 programme, grant agreement N◦ 828947, and from the Mexican Department of Energy, CONACYT-SENER Hidrocarburos grant agreement N◦ B-S-69926.Peer ReviewedPostprint (published version

Simulation of high temperature superconductors and experimental validation

In this work, we present a parallel, fully-distributed finite element numerical framework to simulate the low-frequency electromagnetic behaviour of superconducting devices, which efficiently exploits high performance computing platforms. We select the so-called H-formulation, which uses the magnetic field as a state variable. Nédélec elements (of arbitrary order) are required for an accurate approximation of the H-formulation for modelling electromagnetic fields along interfaces between regions with high contrast medium properties. An h-adaptive mesh refinement technique customized for Nédélec elements leads to a structured fine mesh in areas of interest whereas a smart coarsening is obtained in other regions. The composition of a tailored, robust, parallel nonlinear solver completes the exposition of the developed tools to tackle the problem. First, a comparison against experimental data is performed to show the availability of the finite element approximation to model the physical phenomena. Then, a selected state-of-the-art 3D benchmark is reproduced, focusing on the parallel performance of the algorithms.Peer ReviewedPostprint (author's final draft

Simulation of marine controlled source electromagnetic measurements using a parallel fourier hp-finite element method

We introduce a new numerical method to simulate geophysical marine controlled source electromagnetic (CSEM) measurements for the case of 2D structures and finite 3D sources of electromagnetic (EM) excitation. The method of solution is based on a spatial discretization that combines a 1D Fourier transform with a 2D self-adaptive, goal-oriented, hp-Finite element method. It enables fast and accurate simulations for a variety of important, challenging and practical cases of marine CSEM acquisition. Numerical results confirm the high accuracy of the method as well as some of the main physical properties of marine CSEM measurements such as high measurement sensitivity to oil-bearing layers in the subsurface. In our model, numerical results indicate that measurements could be affected by the finite oil-bearing layer by as much as 104% (relative difference). While the emphasis of this paper is on EM simulations, the method can be used to simulate different physical phenomena such as seismic measurements

Numerical Modeling and High Speed Parallel Computing: New Perspectives for Tomographic Microwave Imaging for Brain Stroke Detection and Monitoring

This paper deals with microwave tomography for brain stroke imaging using state-of-the-art numerical modeling and massively parallel computing. Microwave tomographic imaging requires the solution of an inverse problem based on a minimization algorithm (e.g. gradient based) with successive solutions of a direct problem such as the accurate modeling of a whole-microwave measurement system. Moreover, a sufficiently high number of unknowns is required to accurately represent the solution. As the system will be used for detecting the brain stroke (ischemic or hemorrhagic) as well as for monitoring during the treatment, running times for the reconstructions should be reasonable. The method used is based on high-order finite elements, parallel preconditioners from the Domain Decomposition method and Domain Specific Language with open source FreeFEM++ solver