Massively-parallel electrical-conductivity imaging of hydrocarbonsusing the Blue Gene/L supercomputer

Large-scale controlled source electromagnetic (CSEM)three-dimensional (3D) geophysical imaging is now receiving considerableattention for electrical conductivity mapping of potential offshore oiland gas reservoirs. To cope with the typically large computationalrequirements of the 3D CSEM imaging problem, our strategies exploitcomputational parallelism and optimized finite-difference meshing. Wereport on an imaging experiment, utilizing 32,768 tasks/processors on theIBM Watson Research Blue Gene/L (BG/L) supercomputer. Over a 24-hourperiod, we were able to image a large scale marine CSEM field data setthat previously required over four months of computing time ondistributed clusters utilizing 1024 tasks on an Infiniband fabric. Thetotal initial data misfit could be decreased by 67 percent within 72completed inversion iterations, indicating an electrically resistiveregion in the southern survey area below a depth of 1500 m below theseafloor. The major part of the residual misfit stems from transmitterparallel receiver components that have an offset from the transmittersail line (broadside configuration). Modeling confirms that improvedbroadside data fits can be achieved by considering anisotropic electricalconductivities. While delivering a satisfactory gross scale image for thedepths of interest, the experiment provides important evidence for thenecessity of discriminating between horizontal and verticalconductivities for maximally consistent 3D CSEM inversions

Massively parallel electrical conductivity imaging of the subsurface: Applications to hydrocarbon exploration

Three-dimensional (3D) geophysical imaging is now receiving considerable attention for electrical conductivity mapping of potential offshore oil and gas reservoirs. The imaging technology employs controlled source electromagnetic (CSEM) and magnetotelluric (MT) fields and treats geological media exhibiting transverse anisotropy. Moreover when combined with established seismic methods, direct imaging of reservoir fluids is possible. Because of the size of the 3D conductivity imaging problem, strategies are required exploiting computational parallelism and optimal meshing. The algorithm thus developed has been shown to scale to tens of thousands of processors. In one imaging experiment, 32,768 tasks/processors on the IBM Watson Research Blue Gene/L supercomputer were successfully utilized. Over a 24 hour period we were able to image a large scale field data set that previously required over four months of processing time on distributed clusters based on Intel or AMD processors utilizing 1024 tasks on an InfiniBand fabric. Electrical conductivity imaging using massively parallel computational resources produces results that cannot be obtained otherwise and are consistent with timeframes required for practical exploration problems

Numerical solution of 3-D electromagnetic problems in exploration geophysics and its implementation on massively parallel computers

The growing significance, technical development and employment of electromagnetic (EM) methods in exploration geophysics have led to the increasing need for reliable and fast techniques of interpretation of 3-D EM data sets acquired in complex geological environments. The first and most important step to creating an inversion method is the development of a solver for the forward problem. In order to create an efficient, reliable and practical 3-D EM inversion, it is necessary to have a 3-D EM modelling code that is highly accurate, robust and very fast. This thesis focuses precisely on this crucial and very demanding step to building a 3-D EM interpretation method. The thesis presents as its main contribution a highly accurate, robust, very fast and extremely scalable numerical method for 3-D EM modelling in geophysics that is based on finite elements (FE) and designed to run on massively parallel computing platforms. Thanks to the fact that the FE approach supports completely unstructured tetrahedral meshes as well as local mesh refinements, the presented solver is able to represent complex geometries of subsurface structures very precisely and thus improve the solution accuracy and avoid misleading artefacts in images. Consequently, it can be successfully used in geological environments of arbitrary geometrical complexities. The parallel implementation of the method, which is based on the domain decomposition and a hybrid MPI-OpenMP scheme, has proved to be highly scalable - the achieved speed-up is close to the linear for more than a thousand processors. Thanks to this, the code is able to deal with extremely large problems, which may have hundreds of millions of degrees of freedom, in a very efficient way. The importance of having this forward-problem solver lies in the fact that it is now possible to create a 3-D EM inversion that can deal with data obtained in extremely complex geological environments in a way that is realistic for practical use in industry. So far, such imaging tool has not been proposed due to a lack of efficient, parallel FE solutions as well as the limitations of efficient solvers based on finite differences. In addition, the thesis discusses physical, mathematical and numerical aspects and challenges of 3-D EM modelling, which have been studied during my research in order to properly design the presented software for EM field simulations on 3-D areas of the Earth. Through this work, a physical problem formulation based on the secondary Coulomb-gauged EM potentials has been validated, proving that it can be successfully used with the standard nodal FE method to give highly accurate numerical solutions. Also, this work has shown that Krylov subspace iterative methods are the best solution for solving linear systems that arise after FE discretisation of the problem under consideration. More precisely, it has been discovered empirically that the best iterative method for this kind of problems is biconjugate gradient stabilised with an elaborate preconditioner. Since most commonly used preconditioners proved to be either unable to improve the convergence of the implemented solvers to the desired extent, or impractical in the parallel context, I have proposed a preconditioning technique for Krylov methods that is based on algebraic multigrid. Tests for various problems with different conductivity structures and characteristics have shown that the new preconditioner greatly improves the convergence of different Krylov subspace methods, which significantly reduces the total execution time of the program and improves the solution quality. Furthermore, the preconditioner is very practical for parallel implementation. Finally, it has been concluded that there are not any restrictions in employing classical parallel programming models, MPI and OpenMP, for parallelisation of the presented FE solver. Moreover, they have proved to be enough to provide an excellent scalability for it

Laboratory technology research: Abstracts of FY 1998 projects

The Laboratory Technology Research (LTR) program supports high-risk, multidisciplinary research partnerships to investigate challenging scientific problems whose solutions have promising commercial potential. These partnerships capitalize on two great strengths of the country: the world-class basic research capability of the DOE Office of Science (SC) national laboratories and the unparalleled entrepreneurial spirit of American industry. Projects supported by the LTR program in FY 1998 explore the applications of basic research advances relevant to DOE`s mission over a full range of scientific disciplines. The program presently emphasizes three critical areas of mission-related research: advanced materials, intelligent processing and manufacturing research, and environmental and biomedical research. Abstracts for 85 projects are contained in this report

Laboratory technology research - abstracts of FY 1997 projects

The Laboratory Technology Research (LTR) program supports high-risk, multidisciplinary research partnerships to investigate challenging scientific problems whose solutions have promising commercial potential. These partnerships capitalize on two great strengths of this country: the world-class basic research capability of the DOE Energy Research (ER) multi-program national laboratories and the unparalleled entrepreneurial spirit of American industry. A distinguishing feature of the ER multi-program national laboratories is their ability to integrate broad areas of science and engineering in support of national research and development goals. The LTR program leverages this strength for the Nation`s benefit by fostering partnerships with US industry. The partners jointly bring technology research to a point where industry or the Department`s technology development programs can pursue final development and commercialization. Projects supported by the LTR program are conducted by the five ER multi-program laboratories. These projects explore the applications of basic research advances relevant to DOE`s mission over a full range of scientific disciplines. The program presently emphasizes three critical areas of mission-related research: advanced materials; intelligent processing/manufacturing research; and sustainable environments

Multidimensional computation and visualisation for marine controlled source electromagnetic methods

The controlled source electromagnetic method is improving the search for oil and gas in marine settings and is becoming an integral component of many exploration toolkits. While the level of detail and benefit obtained from recorded electromagnetic data sets is limited to the tools available, interpretation is fundamentally restricted by non-unique and equivalent solutions. I create the tools necessary to rapidly compute and visualise multi-dimensional electromagnetic fields generated for a variety of controlled source electromagnetic surveys. This thesis is divided into two parts: the creation of an electromagnetic software framework and the electromagnetic research applications.The creation of a new electromagnetic software framework is covered in Part I. Steps to create and test a modern electromagnetic data structure, three-dimensional visualisation and interactive graphical user interface from the ground up are presented. Bringing together several computer science disciplines ranging from parallel computing, networking and computer human interaction to three-dimensional visualisation, a package specifically tailored to marine controlled source electromagnetic compuation is formed. The electromagnetic framework is comprised of approximately 100,000 lines of new Java code and several third party libraries, which provides low-level graphical, network and execution cross-platform functionality. The software provides a generic framework to integrate most computational engines and algorithms into the coherent global electromagnetic package enabling the interactive forward modelling, inversion and visualisation of electromagnetic data.Part II is comprised of several research applications utilising the developed electromagnetic software framework. Cloud computing and streamline visualisation are covered. These topics are covered to solve several problems in modern controlled source electromagnetic methods. Large 3D electromagnetic modelling and inversion may require days or even weeks to be performed on a single-threaded personal computers. A massively parallelised electromagnetic forward modelling and inversion methods can dramatically was created to improve computational time. The developed ’macro’ parallelisation method facilitated the reduction in computational time by several orders of magnitude with relatively little additional effort and without modification of the internal electromagnetic algorithm. The air wave is a significant component of marine controlled source electromagnetic surveys however there is controversy and confusion over its defintion. The airwave has been described as a reflected, refracted, direct or diffusing wave, which has lead to confusion over its physical reality

Synthetic modelling study of marine controlled-source electromagnetic data for hydrocarbon exploration

The marine controlled-source electromagnetic method (CSEM) is a geophysical technique for mapping subsurface electrical resistivity structure in the offshore environment. It has gained ground in recent years as a tool for remote detection and mapping of hydrocarbon reservoirs as it serves as an independent yet complementary method to seismic acquisition. While CSEM data contains useful information about the subsurface, modelling and inversion are required to convert data into interpretable resistivity images. Improvement of modelling tools will assist in closing the gap between acquisition and interpretation of CSEM data. The primary focus of this study was to explore the limits of our present modelling capabilities in the context of marine electromagnetic scenarios. Software based on the three-dimensional CSEM finite-element forward code CSEM3DFWD (Ansari and Farquharson, 2014; Ansari et al., 2015) was employed in this study. While testing of this software had been expanded to models of relevance to mineral exploration, its performance for models which are representative of marine geologic environments, in particular those which are encountered in offshore oil and gas exploration, had not yet been investigated. In this study, marine models of increasing complexity were built and tested, with the ultimate goal of synthesizing marine CSEM data for three-dimensional earth models which were complete in their description of the subsurface. Computed responses were compared to results existing in the literature, when available. To investigate the capability of the code in modelling realistic scenarios, forward solutions were computed for a marine reservoir model based on the real-life North Amethyst oil field, located in the Jeanne d’Arc Basin, offshore Newfoundland. When the capability of modelling realistic earth models is fully realized, forward modelling may be used to assess the utility of the marine CSEM method as a tool for hydrocarbon detection and delineation in specific offshore scenarios

Large Scale Computing and Storage Requirements for Basic Energy Sciences Research

The National Energy Research Scientific Computing Center (NERSC) is the leading scientific computing facility supporting research within the Department of Energy's Office of Science. NERSC provides high-performance computing (HPC) resources to approximately 4,000 researchers working on about 400 projects. In addition to hosting large-scale computing facilities, NERSC provides the support and expertise scientists need to effectively and efficiently use HPC systems. In February 2010, NERSC, DOE's Office of Advanced Scientific Computing Research (ASCR) and DOE's Office of Basic Energy Sciences (BES) held a workshop to characterize HPC requirements for BES research through 2013. The workshop was part of NERSC's legacy of anticipating users future needs and deploying the necessary resources to meet these demands. Workshop participants reached a consensus on several key findings, in addition to achieving the workshop's goal of collecting and characterizing computing requirements. The key requirements for scientists conducting research in BES are: (1) Larger allocations of computational resources; (2) Continued support for standard application software packages; (3) Adequate job turnaround time and throughput; and (4) Guidance and support for using future computer architectures. This report expands upon these key points and presents others. Several 'case studies' are included as significant representative samples of the needs of science teams within BES. Research teams scientific goals, computational methods of solution, current and 2013 computing requirements, and special software and support needs are summarized in these case studies. Also included are researchers strategies for computing in the highly parallel, 'multi-core' environment that is expected to dominate HPC architectures over the next few years. NERSC has strategic plans and initiatives already underway that address key workshop findings. This report includes a brief summary of those relevant to issues raised by researchers at the workshop