Parallel solution methods and preconditioners for evolution equations

The recent development of the high performance computer platforms shows a clear trend towards heterogeneity and hierarchy. In order to utilize the computational power, particular attention must be paid to finding new algorithms or adjust existing ones so that they better match the HPC computer architecture. In this work we consider an alternative to classical time-stepping methods based on use of time-harmonic properties and discuss solution approaches that allow efficient utilization of modern HPC resources. The method in focus is based on a truncated Fourier expansion of the solution of an evolutionary problem. The analysis is done for linear equations and it is remarked on the possibility to use two- or multilevel mesh methods for nonlinear problems, which can enable further, even higher degree of parallelization. The arising block matrix system to be solved admits a two-by-two block form with square blocks, for which a very efficient preconditioner exists. It leads to tight eigenvalue bounds for the preconditioned matrix and, hence, to a very fast convergence of a preconditioned Krylov subspace or iterative refinement method. The analytical background is shown as well as some illustrating numerical examples

Robust and scalable 3-D geo-electromagnetic modelling approach using the finite element method

We present a robust and scalable solver for time-harmonic Maxwell's equations for problems with large conductivity contrasts, wide range of frequencies, stretched grids and locally refined meshes. The solver is part of the fully distributed adaptive 3-D electromagnetic modelling scheme which employs the finite element method and unstructured non-conforming hexahedral meshes for spatial discretization using the open-source software deal.II. We use the complex-valued electric field formulation and split it into two real-valued equations for which we utilize an optimal block-diagonal pre-conditioner. Application of this pre-conditioner requires the solution of two smaller real-valued symmetric problems. We solve them by using either a direct solver or the conjugate gradient method pre-conditioned with the recently introduced auxiliary space technique. The auxiliary space pre-conditioner reformulates the original problem in form of several simpler ones, which are then solved using highly efficient algebraic multigrid methods. In this paper, we consider the magnetotelluric case and verify our numerical scheme by using COMMEMI 3-D models. Afterwards, we run a series of numerical experiments and demonstrate that the solver converges in a small number of iterations for a wide frequency range and variable problem sizes. The number of iterations is independent of the problem size, but exhibits a mild dependency on frequency. To test the stability of the method on locally refined meshes, we have implemented a residual-based a posteriori error estimator and compared it with uniform mesh refinement for problems up to 200 million unknowns. We test the scalability of the most time consuming parts of our code and show that they fulfill the strong scaling assumption as long as each MPI process possesses enough degrees of freedom to alleviate communication overburden. Finally, we refer back to a direct solver-based pre-conditioner and analyse its complexity in time. The results show that for multiple right-hand sides the direct solver-based pre-conditioner can still be faster for problems of medium size. On the other hand, it also shows non-linear growth in memory, whereas the auxiliary space method increases only linearly.ISSN:0956-540XISSN:1365-246

Activities of the Institute for Computer Applications in Science and Engineering (ICASE)

Research conducted at the Institute for Computer Applications in Science and Engineering in applied mathematics, numerical analysis, and computer science during the period October 1, 1984 through March 31, 1985 is summarized

Computational Electromagnetic Methods for Transcranial Magnetic Stimulation.

Transcranial magnetic stimulation (TMS) is a noninvasive technique used both as a research tool for cognitive neuroscience and as a FDA approved treatment for depression. During TMS, coils positioned near the scalp generate electric fields and activate targeted brain regions. In this thesis, several computational electromagnetics methods that improve the analysis, design, and uncertainty quantification of TMS systems were developed. Analysis: A new fast direct technique for solving the large and sparse linear system of equations (LSEs) arising from the finite difference (FD) discretization of Maxwell’s quasi-static equations was developed. Following a factorization step, the solver permits computation of TMS fields inside realistic brain models in seconds, allowing for patient-specific real-time usage during TMS. The solver is an alternative to iterative methods for solving FD LSEs, often requiring run-times of minutes. A new integral equation (IE) method for analyzing TMS fields was developed. The human head is highly-heterogeneous and characterized by high-relative permittivities (10^7). IE techniques for analyzing electromagnetic interactions with such media suffer from high-contrast and low-frequency breakdowns. The novel high-permittivity and low-frequency stable internally combined volume-surface IE method developed. The method not only applies to the analysis of high-permittivity objects, but it is also the first IE tool that is stable when analyzing highly-inhomogeneous negative permittivity plasmas. Design: TMS applications call for electric fields to be sharply focused on regions that lie deep inside the brain. Unfortunately, fields generated by present-day Figure-8 coils stimulate relatively large regions near the brain surface. An optimization method for designing single feed TMS coil-arrays capable of producing more localized and deeper stimulation was developed. Results show that the coil-arrays stimulate 2.4 cm into the head while stimulating 3.0 times less volume than Figure-8 coils. Uncertainty quantification (UQ): The location/volume/depth of the stimulated region during TMS is often strongly affected by variability in the position and orientation of TMS coils, as well as anatomical differences between patients. A surrogate model-assisted UQ framework was developed and used to statistically characterize TMS depression therapy. The framework identifies key parameters that strongly affect TMS fields, and partially explains variations in TMS treatment responses.PhDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/111459/1/luisgo_1.pd

Proceedings of the FEniCS Conference 2017

Proceedings of the FEniCS Conference 2017 that took place 12-14 June 2017 at the University of Luxembourg, Luxembourg

SOLID-SHELL FINITE ELEMENT MODELS FOR EXPLICIT SIMULATIONS OF CRACK PROPAGATION IN THIN STRUCTURES

Crack propagation in thin shell structures due to cutting is conveniently simulated using explicit finite element approaches, in view of the high nonlinearity of the problem. Solidshell elements are usually preferred for the discretization in the presence of complex material behavior and degradation phenomena such as delamination, since they allow for a correct representation of the thickness geometry. However, in solid-shell elements the small thickness leads to a very high maximum eigenfrequency, which imply very small stable time-steps. A new selective mass scaling technique is proposed to increase the time-step size without affecting accuracy. New ”directional” cohesive interface elements are used in conjunction with selective mass scaling to account for the interaction with a sharp blade in cutting processes of thin ductile shells

Computational Engineering

The focus of this Computational Engineering Workshop was on the mathematical foundation of state-of-the-art and emerging finite element methods in engineering analysis. The 52 participants included mathematicians and engineers with shared interest on discontinuous Galerkin or Petrov-Galerkin methods and other generalized nonconforming or mixed finite element methods

Assessment of Linear Inverse Problems in Magnetocardiography and Lorentz Force Eddy Current Testing

Lineare inverse Probleme tauchen in vielen Bereichen von Wissenschaft und Technik auf. Effiziente Lösungsstrategien für diese inversen Probleme erfordern Informationen darüber, ob das Problem schlecht-gestellt und in welchem Ausmaß dies der Fall ist. In der vorliegenden Dissertation wird eine umfassende theoretische Analyse existierender Bewertungsmaße durchgeführt. Aus diesen Untersuchungen werden schließlich zwei neue Bewertungsmaße abgeleitet. Beide können bei einer Vielzahl linearer inverser Probleme angewendet werden, einschließlich biomedizinische Anwendungen oder der zerstörungsfreien Materialprüfung. Die theoretischen Betrachtungen zur Behandlung linearer inverser Probleme werden auf zwei Beispiele angewendet. Das erste ist die Magnetkardiographie, wo die Optimierung magnetischer Sensoren in einem westenähnlichen Sensorfeld untersucht wird. Für die Messungen der magnetischen Flussdichte werden üblicherweise monoaxiale Sensoren in einem Feld perfekt parallel angeordnet. Eine zufällige Variation ihrer Ausrichtungen kann die Kondition des entsprechenden linearen inversen Problems verbessern. Eine theoretische Definition des Falls, in dem zufällige Variationen monoaxialer Sensoren den Zustand der Kernmatrix mit einer Wahrscheinlichkeit gleich Eins verbessern wird ebenfalls in der Dissertation vorgestellt. Diese theoretische Beobachtung ist allgemein gültig.Positionen und Orientierungen der Magnetsensoren rund um den Oberkörper wurden mit drei aus der Literatur bekannten Bewertungsmaßen und einem neu in dieser Arbeit vorgeschlagenen Maß optimiert. Die besten Ergebnisse ergeben sich bei einer unregelmäßigen Verteilung der Sensoren auf der Oberfläche des Brustkorbes. Im Vergleich zu früheren Untersuchungsergebnissen kann daraus geschlussfolgert werden, dass mit geringfügig abweichenden Sensoranordnungen ebenso gute Ergebnisse erzielt werden können. Ein zweites Anwendungsbeispiel ist ein Verfahren der zerstörungsfreien Materialprüfung, das auch als Lorentzkraft-Wirbelstromprüfung bekannt geworden ist. In dieser Arbeit wird eine neue Methode für die kontaktlose, zerstörungsfreie Untersuchung leitfähiger Materialien vorgestellt. Dabei wird die Lorentzkraft gemessen, die auf einen Dauermagneten wirkt, der relativ zu einem Testkörper bewegt. Es wird eine neue Approximationsmethode für die Berechnung der magnetischen Felder und der Lorentzkräfte vorgeschlagen.Linear inverse problems arise throughout a variety of branches of science and engineering. Efficient solution strategies for these inverse problems need to know whether a problem is ill-conditioned as well as its degree of ill-conditioning. In this thesis, a comprehensive theoretical analysis of known figures of merit has been done and finally two new figures of merit have been developed. Both can be applied in a large variety of linear inverse problems, including biomedical applications and nondestructive testing of materials. Theoretical considerations of the conditioning of linear inverse problems are applied to two examples. The first one is magnetocardiography, where the optimization of magnetic sensors in a vest-like sensor array has been considered. When measuring magnetic flux density, usually mono-axial magnetic sensors are arranged in an array, perfectly in parallel. It has been shown that a random variation of their orientations can improve the condition of the corresponding linear inverse problem. Thus, in this thesis a theoretical definition of the case when random variations of mono-axial sensors orientations improve the condition of the kernel matrix with a probability equal to one is presented. This theoretical observation is valid in general. Positions and orientations of magnetic sensors around the torso have been optimized minimizing three figures of merit given in the literature and a novel one presented in the thesis. Best results have been found for non-uniform sensors distribution on the whole torso surface. In comparison to previous findings can be concluded that quite different sensor sets can perform equally well.The second application example is nondestructive testing of materials named Lorentz force eddy current testing, where the Lorentz force exerting on a permanent magnet, which is moving relative to the specimen, is determined. A novel approximation method for the calculation of the magnetic fields and Lorentz forces is proposed. Based on the new approximation method, a new inverse procedure for defect reconstruction is proposed. A successful reconstruction using data from finite elements method analysis and measurements is obtained

Edge-elements formulation of 3D CSEM in geophysics : a parallel approach

Electromagnetic methods (EM) are an invaluable research tool in geophysics whose relevance has increased rapidly in recent years due to its wide industrial adoption. In particular, the forward modelling of three-dimensional marine controlled-source electromagnetics (3D CSEM FM) has become an important technique for reducing ambiguities in the interpretation of geophysical datasets through mapping conductivity variations in the subsurface. As a consequence, the 3D CSEM FM has application in many areas such as hydrocarbon/mineral exploration, reservoir monitoring, CO2 storage characterization, geothermal reservoir imaging and many others due to there quantities often displaying conductivity contrasts with respect to their surrounding sediments. However, the 3D CSEM FM at real scale implies a numerical challenge that requires an important computational effort, often too high for modest multicore computing architectures, especially if it fuels an inversion process. On the other hand, although the HPC code development is dominated by compiled languages, the popularity of high-level languages for scientific computations has increased considerably. Among all of them, Python is probably the language that has shown more interest, mainly because of flexibility and its simple and clean syntax. However, its use for HPC geophysical applications is still limited, which suggests a path for research, development and improvement. Therefore, this thesis reports the attempts at designing and implementing a methodology that has not been systematically applied for solving 3D CSEM FM with an HPC application baked upon Python. The net contribution of this effort is the development and documentation of a new open-source modelling code for 3D CSEM FM in geophysics, namely, the Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM). The importance of having this modelling tools lies in the fact that they provide synthetic results that can be compared with real data which has a practical use both in the industry and academia. Still, available 3D CSEM FM codes are usually written in low-level languages whose implemented methods are often innaccessible to the scientific community since they are commercial. PETGEM is written mostly in Python and relies on mpi4py and petsc4py packages for parallel computations. Other scientific Python packages used include Numpy andScipy. This code is designed to cope with the main challenges encountered within the numerical simulation of the problem under consideration: tackle realistic problems with accuracy, efficiency and flexibility. It uses the Nédélec Edge Finite Element Method (EFEM) as discretisation technique because its divergence-free basis is very well suited for solving Maxwell¿s equations. Furthermore, it supports completely unstructured tetrahedral meshes which allows the representation of complex geometries and local refinement, positively impacting the accuracy of the solution. The parallel implementation of the code using shared/distributed-memory architectures is investigated and described throughout this document. In addition, the thesis deals with the numerical and physical challenges of the 3D CSEM FM problem. Through this work, frequency-domain Maxwell's equations have been discretised using EFEM and validated by comparison with analytical solutions and published data, proving that modelling results are highly accurate. Moreover, this work discusses an automatic mesh adaptation strategy and the convergence rate of the iterative solvers that are widely used in the literature for solving the EM problem is presented. In summary, this thesis shows that it is possible to integrate Python and HPC for the solution of 3D CSEM FM at large scale in an effective way. The new modelling tool is easy to use and the adopted algorithms are not only accurate and efficient but also have the possibility to easily add or remove components without having to rewrite large sections of the code.Los métodos electromagnéticos (EM) son una herramienta de investigación inestimable en geofísica, cuya relevancia ha aumentado rápidamente en los últimos años debido a su amplia adopción industrial. En particular, el modelado electromagnético de fuente controlada (3D CSEM FM) se ha convertido en una técnica importante para reducir las ambigüedades en la interpretación de datos geofísicos a través del mapeo de las variaciones de conductividad en el subsuelo. Como resultado, el 3D CSEM FM tiene aplicación en muchas áreas como la exploración de hidrocarburos/minerales, monitoreo de yacimientos, caracterización de almacenamiento de CO2, imágenes de yacimientos geotérmicos, entre otros, debido a que éstos muestran contrastes de conductividad con respecto a sus sedimentos circundantes. Sin embargo, el 3D CSEM FM a escala real implica un desafío numérico que requiere un esfuerzo computacional importante, a menudo demasiado exigente para arquitecturas multicore modestas, especialmente si éste forma parte de un proceso de inversión. Por otra parte, aunque el desarrollo aplicaciones HPC está dominado por lenguajes compilados, la popularidad de los lenguajes de alto nivel para cómputo científico ha aumentado considerablemente. Entre todos ellos, Python es probablemente el idioma que ha mostrado más interés, principalmente a su flexibilidad y sintaxis simple. Sin embargo, su uso para geocómputo con HPC sigue siendo limitado, lo que sugiere un camino para la investigación, el desarrollo y la mejora. Por lo tanto, esta tesis describe el diseño e implementación de una metodología que hasta ña fecha no se ha aplicado sistemáticamente para resolver el 3D CSEM FM con una aplicación HPC basada en Python. La contribución neta de este esfuerzo es el desarrollo y documentación de un nuevo código open-source para el modelado 3D CSEM FM en geofísica, es decir, Parallel Edge-based Tool for Geophysical Electromagnetic Modelling (PETGEM). La importancia del desarrollo de estas herramientas radica en el hecho de que proporcionan resultados sintéticos que pueden ser comparados con datos reales, lo cual tiene un uso práctico en la industria y el mundo académico. A pesar de ello, los códigos disponibles para 3D CSEM FM suelen estar escritos en lenguajes de bajo nivel, y en muchos casos sus métodos no son accesibles a la comunidad científica ya que son comerciales. PETGEM ha sido principalmente escrito en Python y se basa en paquetes mpi4py y petsc4py para cálculos paralelos. El código está diseñado para hacer frente a los principales desafíos que se encuentran en la simulación numérica del problema en cuestión: abordar problemas realistas con precisión, eficiencia y flexibilidad. Además, utiliza el Método de Elementos Finitos de Borde (EFEM) como técnica de discretización ya que sus bases son muy adecuadas para resolver las ecuaciones de Maxwell. Además, soporta mallas tetraédricas no estructuradas que permiten la representación de geometrías complejas y refinamiento local, impactando positivamente la precisión de la solución. A lo largo del documento se investiga la implementación paralela en arquitecturas de memoria compartida/distribuida. Además, la tesis revisa los desafíos numéricos y físicos del problema 3D CSEM FM. A través de este trabajo, las ecuaciones de Maxwell en el dominio de la frecuencia se han discretizado utilizando EFEM y validado contra soluciones analíticas y datos previamente publicados, lo que demuestra que los resultados del modelado son precisos. Por otra parte, este trabajo discute una estrategia de adaptación automática de malla y la tasa de convergencia de los solvers iterativos que se utilizan ampliamente en la literatura. En resumen, esta tesis muestra que es posible integrar Python y HPC para la solución de 3D CSEM FM a gran escala de una manera efectiva. La nueva herramienta de modelado es fácil de usar y los algoritmos adoptados no sólo son precisos y eficientes, sino también flexibles