Improving edge finite element assembly for geophysical electromagnetic modelling on shared-memory architectures

This work presents a set of node-level optimizations to perform the assembly of edge finite element matrices that arise in 3D geophysical electromagnetic modelling on shared-memory architectures. Firstly, we describe the traditional and sequential assembly approach. Secondly, we depict our vectorized and shared-memory strategy which does not require any low level instructions because it is based on an interpreted programming language, namely, Python. As a result, we obtained a simple parallel-vectorized algorithm whose runtime performance is considerably better than sequential version. The set of optimizations have been included to the work-flow of the Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM) which is developed as open-source at the Barcelona Supercomputing Center. Finally, we present numerical results for a set of tests in order to illustrate the performance of our strategy.This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 644202. The research leading to these results has received funding from the European Union's Horizon 2020 Programme (2014-2020) and from Brazilian Ministry of Science, Technology and Innovation through Rede Nacional de Pesquisa (RNP) under the HPC4E Project (www.hpc4e.eu), grant agreement No. 689772. Authors gratefully acknowledge the support from the Mexican National Council for Science and Technology (CONACYT). All numerical tests were performed on the MareNostrum supercomputer of the Barcelona Supercomputing Center - Centro Nacional de Supercomputación (www.bsc.es).Peer ReviewedPostprint (author's final draft

Edge-elements formulation of 3D CSEM in geophysics : a parallel approach

Electromagnetic methods (EM) are an invaluable research tool in geophysics whose relevance has increased rapidly in recent years due to its wide industrial adoption. In particular, the forward modelling of three-dimensional marine controlled-source electromagnetics (3D CSEM FM) has become an important technique for reducing ambiguities in the interpretation of geophysical datasets through mapping conductivity variations in the subsurface. As a consequence, the 3D CSEM FM has application in many areas such as hydrocarbon/mineral exploration, reservoir monitoring, CO2 storage characterization, geothermal reservoir imaging and many others due to there quantities often displaying conductivity contrasts with respect to their surrounding sediments. However, the 3D CSEM FM at real scale implies a numerical challenge that requires an important computational effort, often too high for modest multicore computing architectures, especially if it fuels an inversion process. On the other hand, although the HPC code development is dominated by compiled languages, the popularity of high-level languages for scientific computations has increased considerably. Among all of them, Python is probably the language that has shown more interest, mainly because of flexibility and its simple and clean syntax. However, its use for HPC geophysical applications is still limited, which suggests a path for research, development and improvement. Therefore, this thesis reports the attempts at designing and implementing a methodology that has not been systematically applied for solving 3D CSEM FM with an HPC application baked upon Python. The net contribution of this effort is the development and documentation of a new open-source modelling code for 3D CSEM FM in geophysics, namely, the Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM). The importance of having this modelling tools lies in the fact that they provide synthetic results that can be compared with real data which has a practical use both in the industry and academia. Still, available 3D CSEM FM codes are usually written in low-level languages whose implemented methods are often innaccessible to the scientific community since they are commercial. PETGEM is written mostly in Python and relies on mpi4py and petsc4py packages for parallel computations. Other scientific Python packages used include Numpy andScipy. This code is designed to cope with the main challenges encountered within the numerical simulation of the problem under consideration: tackle realistic problems with accuracy, efficiency and flexibility. It uses the Nédélec Edge Finite Element Method (EFEM) as discretisation technique because its divergence-free basis is very well suited for solving Maxwell¿s equations. Furthermore, it supports completely unstructured tetrahedral meshes which allows the representation of complex geometries and local refinement, positively impacting the accuracy of the solution. The parallel implementation of the code using shared/distributed-memory architectures is investigated and described throughout this document. In addition, the thesis deals with the numerical and physical challenges of the 3D CSEM FM problem. Through this work, frequency-domain Maxwell's equations have been discretised using EFEM and validated by comparison with analytical solutions and published data, proving that modelling results are highly accurate. Moreover, this work discusses an automatic mesh adaptation strategy and the convergence rate of the iterative solvers that are widely used in the literature for solving the EM problem is presented. In summary, this thesis shows that it is possible to integrate Python and HPC for the solution of 3D CSEM FM at large scale in an effective way. The new modelling tool is easy to use and the adopted algorithms are not only accurate and efficient but also have the possibility to easily add or remove components without having to rewrite large sections of the code.Los métodos electromagnéticos (EM) son una herramienta de investigación inestimable en geofísica, cuya relevancia ha aumentado rápidamente en los últimos años debido a su amplia adopción industrial. En particular, el modelado electromagnético de fuente controlada (3D CSEM FM) se ha convertido en una técnica importante para reducir las ambigüedades en la interpretación de datos geofísicos a través del mapeo de las variaciones de conductividad en el subsuelo. Como resultado, el 3D CSEM FM tiene aplicación en muchas áreas como la exploración de hidrocarburos/minerales, monitoreo de yacimientos, caracterización de almacenamiento de CO2, imágenes de yacimientos geotérmicos, entre otros, debido a que éstos muestran contrastes de conductividad con respecto a sus sedimentos circundantes. Sin embargo, el 3D CSEM FM a escala real implica un desafío numérico que requiere un esfuerzo computacional importante, a menudo demasiado exigente para arquitecturas multicore modestas, especialmente si éste forma parte de un proceso de inversión. Por otra parte, aunque el desarrollo aplicaciones HPC está dominado por lenguajes compilados, la popularidad de los lenguajes de alto nivel para cómputo científico ha aumentado considerablemente. Entre todos ellos, Python es probablemente el idioma que ha mostrado más interés, principalmente a su flexibilidad y sintaxis simple. Sin embargo, su uso para geocómputo con HPC sigue siendo limitado, lo que sugiere un camino para la investigación, el desarrollo y la mejora. Por lo tanto, esta tesis describe el diseño e implementación de una metodología que hasta ña fecha no se ha aplicado sistemáticamente para resolver el 3D CSEM FM con una aplicación HPC basada en Python. La contribución neta de este esfuerzo es el desarrollo y documentación de un nuevo código open-source para el modelado 3D CSEM FM en geofísica, es decir, Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM). La importancia del desarrollo de estas herramientas radica en el hecho de que proporcionan resultados sintéticos que pueden ser comparados con datos reales, lo cual tiene un uso práctico en la industria y el mundo académico. A pesar de ello, los códigos disponibles para 3D CSEM FM suelen estar escritos en lenguajes de bajo nivel, y en muchos casos sus métodos no son accesibles a la comunidad científica ya que son comerciales. PETGEM ha sido principalmente escrito en Python y se basa en paquetes mpi4py y petsc4py para cálculos paralelos. El código está diseñado para hacer frente a los principales desafíos que se encuentran en la simulación numérica del problema en cuestión: abordar problemas realistas con precisión, eficiencia y flexibilidad. Además, utiliza el Método de Elementos Finitos de Borde (EFEM) como técnica de discretización ya que sus bases son muy adecuadas para resolver las ecuaciones de Maxwell. Además, soporta mallas tetraédricas no estructuradas que permiten la representación de geometrías complejas y refinamiento local, impactando positivamente la precisión de la solución. A lo largo del documento se investiga la implementación paralela en arquitecturas de memoria compartida/distribuida. Además, la tesis revisa los desafíos numéricos y físicos del problema 3D CSEM FM. A través de este trabajo, las ecuaciones de Maxwell en el dominio de la frecuencia se han discretizado utilizando EFEM y validado contra soluciones analíticas y datos previamente publicados, lo que demuestra que los resultados del modelado son precisos. Por otra parte, este trabajo discute una estrategia de adaptación automática de malla y la tasa de convergencia de los solvers iterativos que se utilizan ampliamente en la literatura. En resumen, esta tesis muestra que es posible integrar Python y HPC para la solución de 3D CSEM FM a gran escala de una manera efectiva. La nueva herramienta de modelado es fácil de usar y los algoritmos adoptados no sólo son precisos y eficientes, sino también flexibles

Parallel 3-D marine controlled-source electromagnetic modelling using high-order tetrahedral Nédélec elements

We present a parallel and high-order Nédélec finite element solution for the marine controlled-source electromagnetic (CSEM) forward problem in 3-D media with isotropic conductivity. Our parallel Python code is implemented on unstructured tetrahedral meshes, which support multiple-scale structures and bathymetry for general marine 3-D CSEM modelling applications. Based on a primary/secondary field approach, we solve the diffusive form of Maxwell’s equations in the low-frequency domain. We investigate the accuracy and performance advantages of our new high-order algorithm against a low-order implementation proposed in our previous work. The numerical precision of our high-order method has been successfully verified by comparisons against previously published results that are relevant in terms of scale and geological properties. A convergence study confirms that high-order polynomials offer a better trade-off between accuracy and computation time. However, the optimum choice of the polynomial order depends on both the input model and the required accuracy as revealed by our tests. Also, we extend our adaptive-meshing strategy to high-order tetrahedral elements. Using adapted meshes to both physical parameters and high-order schemes, we are able to achieve a significant reduction in computational cost without sacrificing accuracy in the modelling. Furthermore, we demonstrate the excellent performance and quasi-linear scaling of our implementation in a state-of-the-art high-performance computing architecture.This project has received funding from the European Union's Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement No. 777778. Furthermore, the research leading to these results has received funding from the European Union's Horizon 2020 programme under the ChEESE Project (https://cheese-coe.eu/ ), grant agreement No. 823844. In addition, the authors would also like to thank the support of the Ministerio de Educación y Ciencia (Spain) under Projects TEC2016-80386-P and TIN2016-80957-P. The authors would like to thank the Editors-in-Chief and to both reviewers, Dr. Martin Cuma and Dr. Raphael Rochlitz, for their valuable comments and suggestions which helped to improve the quality of the manuscript. This work benefited from the valuable suggestions, comments, and proofreading of Dr. Otilio Rojas (BSC). Last but not least, Octavio Castillo-Reyes thanks Natalia Gutierrez (BSC) for her support in CSEM modeling with BSIT.Peer ReviewedPostprint (author's final draft

PETGEM: A parallel code for 3D CSEM forward modeling using edge finite elements

We present the capabilities and results of the Parallel Edge-based Tool for Geophysical Electromagnetic modeling (PETGEM), as well as the physical and numerical foundations upon which it has been developed. PETGEM is an open-source and distributed parallel Python code for fast and highly accurate modeling of 3D marine controlled-source electromagnetic (3D CSEM) problems. We employ the N\'ed\'elec Edge Finite Element Method (EFEM) which offers a good trade-off between accuracy and number of degrees of freedom, while naturally supporting unstructured tetrahedral meshes. We have particularised this new modeling tool to the 3D CSEM problem for infinitesimal point dipoles asumming arbitrarily isotropic media for low-frequencies approximations. In order to avoid source-singularities, PETGEM solves the frequency-domain Maxwell's equations of the secondary electric field, and the primary electric field is calculated analytically for homogeneous background media. We assess the PETGEM accuracy using classical tests with known analytical solutions as well as recent published data of real life geological scenarios. This assessment proves that this new modeling tool reproduces expected accurate solutions in the former tests, and its flexibility on realistic 3D electromagnetic problems. Furthermore, an automatic mesh adaptation strategy for a given frequency and specific source position is presented. We also include a scalability study based on fundamental metrics for high-performance computing (HPC) architectures.Comment: \c{opyright} 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/ This project has received funding from the EC-H2020 under the Marie Sklodowska-Curie grant agreement No. 644202, and from the EC-H2020 under the HPC4E Project, grant agreement No. 68977

Python for HPC geophysical electromagnetic applications: experiences and perspectives

Nowadays, the electromagnetic modelling are a fun-damental tool in geophysics due to their wide field of application: hydrocarbon and mineral exploration, reservoir monitoring, CO storage characterization, geothermal reservoir imaging and many others. In particular, the 3D CSEM forward modelling (FM) is an established tool in the oil & gas industry because of the hope that the application of such methods would eventually lead to the direct detection of hydrocarbons through their insulating properties. Although 3D CSEM FM is nowadays a well-known geophysical prospecting tool and his fundamental mathematical theory is well-established, the state-of-art shows a relative scarsity of robust, flexible, modular and open-source codes to simulate these problems on HPC platforms, which is crucial in the future goal of solving inverse problems. In this talk we describe our experience and perspectives in the development of an HPC python code for the 3D CSEM FM, namely, PETGEM. We focus on three points: 1) 3D CSEM FM theory from a practical point of view, 2) PETGEM features and Python potential for HPC applications, and 3) Modelling results of real-life 3D CSEM FM cases. These points depict that PETGEM could be an attractive and competitive HPC tool to simulate real-scale of 3D CSEM FM in geophysics

Electromagnetic imaging and deep learning for transition to renewable energies: a technology review

Electromagnetic imaging is a technique that has been employed and perfected to investigate the Earth subsurface over the past three decades. Besides the traditional geophysical surveys (e.g., hydrocarbon exploration, geological mapping), several new applications have appeared (e.g., characterization of geothermal energy reservoirs, capture and storage of carbon dioxide, water prospecting, and monitoring of hazardous-waste deposits). The development of new numerical schemes, algorithms, and easy access to supercomputers have supported innovation throughout the geo-electromagnetic community. In particular, deep learning solutions have taken electromagnetic imaging technology to a different level. These emerging deep learning tools have significantly contributed to data processing for enhanced electromagnetic imaging of the Earth. Herein, we review innovative electromagnetic imaging technologies and deep learning solutions and their role in better understanding useful resources for the energy transition path. To better understand this landscape, we describe the physics behind electromagnetic imaging, current trends in its numerical modeling, development of computational tools (traditional approaches and emerging deep learning schemes), and discuss some key applications for the energy transition. We focus on the need to explore all the alternatives of technologies and expertise transfer to propel the energy landscape forward. We hope this review may be useful for the entire geo-electromagnetic community and inspire and drive the further development of innovative electromagnetic imaging technologies to power a safer future based on energy sources.This work was supported by the European Union’s Horizon 2020 research and innovation programme under grant agreements No. 955606 (DEEP-SEA) and No. 777778 (MATHROCKS). Furthermore, the research leading of this study has received funding from the Ministerio de Educación y Ciencia (Spain) under Project TED2021-131882B-C42.Peer ReviewedPostprint (published version

Meshing strategies for 3d geo-electromagnetic modeling in the presence of metallic infrastructure

In 3D geo-electromagnetic modeling, an adequate discretisation of the modeling domain is crucial to obtain accurate forward responses and reliable inversion results while reducing the computational cost. This paper investigates the mesh design for subsurface models, including steel-cased wells, which is relevant for many exploration settings but still remains a numerically challenging task. Applying a goal-oriented mesh refinement technique and subsequent calculations with the high-order edge finite element method, simulations of 3D controlled-source electromagnetic models in the presence of metallic infrastructure are performed. Two test models are considered, each needing a distinct version of approximation methods to incorporate the conductive steel casings of the included wells. The influence of mesh quality, goal-oriented meshing, and high-order approximations on problem sizes, computational cost, and accuracy of electromagnetic responses is investigated. The main insights of our work are: (a) the applied numerical schemes can mitigate the computational burden of geo-electromagnetic modeling in the presence of steel artifacts; (b) investigating the processes driving the meshing of models with embedded metallic infrastructures can lead to adequate strategies to deal with the inversion of such electromagnetic data sets. Based on the modeling results and analyses conducted, general recommendations for modeling strategies are proposed when performing simulations for challenging steel infrastructure scenarios.Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. The work of O.C-R. has received funding from the Ministerio de Educación y Ciencia (Spain) under Project TED2021-131882B-C42.The code development of P.R. has been financed by the Smart Exploration project. Smart Exploration has received funding from the European Union’s Horizon 2020 Framework Programme under grant agreement N∘ 775971. The computations were enabled by resources provided by the Swedish National Infrastructure for Computing (SNIC) at UPPMAX partially funded by the Swedish Research Council through grant agreement N∘ SNIC 2021/22-883.Peer ReviewedPostprint (published version

HPC geophysical electromagnetics: a synthetic VTI model with complex bathymetry

We introduce a new synthetic marine model for 3D controlled-source electromagnetic method (CSEM) surveys. The proposed model includes relevant features for the electromagnetic geophysical community such as large conductivity contrast with vertical transverse isotropy and a complex bathymetry profile. In this paper, we present the experimental setup and several 3D CSEM simulations in the presence of a resistivity unit denoting a hydrocarbon reservoir. We employ a parallel and high-order vector finite element routine to perform the CSEM simulations. By using tailored meshes, several scenarios are simulated to assess the influence of the reservoir unit presence on the electromagnetic responses. Our numerical assessment confirms that resistivity unit strongly influences the amplitude and phase of the electromagnetic measurements. We investigate the code performance for the solution of fundamental frequencies on high-performance computing architectures. Here, excellent performance ratios are obtained. Our benchmark model and its modeling results are developed under an open-source scheme that promotes easy access to data and reproducible solutions.The work of O.C-R., conducted in the frame of PIXIL project, has been 65% cofinanced by the European Regional Development Fund (ERDF) through the Interreg V-A SpainFrance-Andorra program (POCTEFA2014-2020). BSC authors have received funding from the European Union’s Horizon 2020 programme under the Marie Sklodowska-Curie grant agreement N◦ 777778. Furthermore, the development of PETGEM has received funding from the European Union’s Horizon 2020 programme, grant agreement N◦ 828947, and from the Mexican Department of Energy, CONACYT-SENER Hidrocarburos grant agreement N◦ B-S-69926.Peer ReviewedPostprint (published version

Land CSEM Simulations and Experimental Test Using Metallic Casing in a Geothermal Exploration Context: Vallès Basin (NE Spain) Case Study

Controlled-source electromagnetic (CSEM) measurements are complementary data for magnetotelluric (MT) characterization although its methodology on land is not sufficiently developed and tested as in marine environments. Acquiring expertise in CSEM is crucial for surveys in places where MT cannot be performed due to high levels of cultural noise. To acquire that expertise, we perform CSEM experiments in the Vallès fault [Northeast (NE), Spain], where MT results have been satisfactory and allow us to verify the CSEM results. The Vallès basin is relevant for potential heat generation because of the presence of several geothermal anomalies and its nearby location in urban areas. In this article, we present the experimental setup for that region, a 2-D joint MT+CSEM inverse model, several 3-D CSEM simulations in the presence of metallic casing, and its comparison with real data measurements. We employ a parallel and high-order vector finite element algorithm to discretize the governing equations. By using an adapted meshing strategy, different scenarios are simulated to study the influence of the source position/direction and the conductivity model in a metallic casing presence. An excellent agreement between the simulated data and analytical/real field data demonstrates the feasibility of study metallic structures in realistic configurations. Our numerical results confirm that metallic casing strongly influences electromagnetic (EM) responses, making surface measurements more sensitive to resistivity variations near the metallic structure. It could be beneficial getting higher signal-to-noise ratios and sensitivity to deep targets. However, such a casing effect depends on the input model (e.g., conductivity contrasts, frequency, and geometry)