Parallel algorithms for computational fluid dynamics on unstructured meshes

La simulació numèrica directa (DNS) de fluxos complexes és actualment una utopia per la majoria d'aplicacions industrials ja que els requeriments computacionals son massa elevats. Donat un flux, la diferència entre els recursos computacionals necessaris i els disponibles és cobreix mitjançant la modelització/simplificació d'alguns termes de les equacions originals que regeixen el seu comportament. El creixement continuat dels recursos computacionals disponibles, principalment en forma de super-ordinadors, contribueix a reduir la part del flux que és necessari aproximar. De totes maneres, obtenir la eficiència esperada dels nous super-ordinadors no és una tasca senzilla i, per aquest motiu, part de la recerca en el camp de la Mecànica de Fluids Computacional es centra en aquest objectiu. En aquest sentit, algunes contribucions s'han presentat en el marc d'aquesta tesis. El primer objectiu va ser el desenvolupament d'un codi de CFD de propòsit general i paral·lel, basat en la metodologia de volums finits en malles no estructurades, per resoldre problemes de multi-física. Aquest codi, anomenat TermoFluids (TF), té un disseny orientat a objectes i pensat per ser usat de forma altament eficient en els super-ordinadors actuals. Amb el temps, ha esdevingut pel grup una eina fonamental en projectes tant de recerca bàsica com d'interès industrial. En el context d'aquesta tesis, el treball s'ha focalitzat en el desenvolupament de dos de les llibreries més bàsiques de TermoFluids: i) La Basics Objects Library (BOL), que es una plataforma de software sobre la qual estan programades la resta de llibreries del codi, i que conté els mètodes algebraics i geomètrics fonamentals per la implementació paral·lela dels algoritmes de discretització, ii) la Linear Solvers Library (LSL), que conté un gran nombre de mètodes per resoldre els sistemes d'equacions lineals derivats de les discretitzacions. El primer capítol d'aquesta tesi conté les principals idees subjacents al disseny i la implementació de la BOL i la LSL, juntament amb alguns exemples i algunes aplicacions industrials. En els capítols posteriors hi ha una explicació detallada de solvers específics per algunes aplicacions concretes. En el segon capítol, es presenta un solver paral·lel i directe per la resolució de l'equació de Poisson per casos en els quals una de les direccions del domini té condicions d'homogeneïtat. En la simulació de fluxos incompressibles, l'equació de Poisson es resol almenys una vegada en cada pas de temps, convertint-se en una de les parts més costoses i difícils de paral·lelitzar del codi. El mètode que proposem és una combinació d'una descomposició directa de Schur (DDS) i una diagonalització de Fourier. La darrera descompon el sistema original en un conjunt de sub-sistemes 2D independents que es resolen mitjançant l'algorisme DDS. Atès que no s'imposen restriccions a les direccions no periòdiques del domini, aquest mètode és aplicable a la resolució de problemes discretitzats mitjançat l'extrusió de malles 2D no estructurades. L'escalabilitat d'aquest mètode ha estat provada amb èxit amb un màxim de 8192 CPU per malles de fins a ~10⁹ volums de control. En el darrer capitol capítol, es presenta un mètode de resolució per l'equació de Transport de Boltzmann (BTE). La estratègia emprada es basa en el mètode d'Ordenades Discretes i pot ser aplicat en discretitzacions no estructurades. El flux per a cada ordenada angular es resol amb un mètode de substitució equivalent a la resolució d'un sistema lineal triangular. La naturalesa seqüencial d'aquest procés fa de la paral·lelització de l'algoritme el principal repte. Diversos algorismes de substitució han estat analitzats, esdevenint una de les heurístiques proposades la millor opció en totes les situacions analitzades, amb excel·lents resultats. Els testos d'eficiència paral·lela s'han realitzat usant fins a 2560 CPU.Direct Numerical Simulation (DNS) of complex flows is currently an utopia for most of industrial applications because computational requirements are too high. For a given flow, the gap between the required and the available computing resources is covered by modeling/simplifying of some terms of the original equations. On the other hand, the continuous growth of the computing power of modern supercomputers contributes to reduce this gap, reducing hence the unresolved physics that need to be attempted with approximated models. This growth, widely relies on parallel computing technologies. However, getting the expected performance from new complex computing systems is becoming more and more difficult, and therefore part of the CFD research is focused on this goal. Regarding to it, some contributions are presented in this thesis. The first objective was to contribute to the development of a general purpose multi-physics CFD code. referred to as TermoFluids (TF). TF is programmed following the object oriented paradigm and designed to run in modern parallel computing systems. It is also intensively involved in many different projects ranging from basic research to industry applications. Besides, one of the strengths of TF is its good parallel performance demonstrated in several supercomputers. In the context of this thesis, the work was focused on the development of two of the most basic libraries that compose TF: I) the Basic Objects Library (BOL), which is a parallel unstructured CFD application programming interface, on the top of which the rest of libraries that compose TF are written, ii) the Linear Solvers Library (LSL) containing many different algorithms to solve the linear systems arising from the discretization of the equations. The first chapter of this thesis contains the main ideas underlying the design and the implementation of the BOL and LSL libraries, together with some examples and some industrial applications. A detailed description of some application-specific linear solvers included in the LSL is carried out in the following chapters. In the second chapter, a parallel direct Poisson solver restricted to problems with one uniform periodic direction is presented. The Poisson equation is solved, at least, once per time-step when modeling incompressible flows, becoming one of the most time consuming and difficult to parallelize parts of the code. The solver here proposed is a combination of a direct Schur-complement based decomposition (DSD) and a Fourier diagonalization. The latter decomposes the original system into a set of mutually independent 2D sub-systems which are solved by means of the DSD algorithm. Since no restrictions are imposed in the non-periodic directions, the overall algorithm is well-suited for solving problems discretized on extruded 2D unstructured meshes. The scalability of the solver has been successfully tested using up to 8192 CPU cores for meshes with up to 10 9 grid points. In the last chapter, a solver for the Boltzmann Transport Equation (BTE) is presented. It can be used to solve radiation phenomena interacting with flows. The solver is based on the Discrete Ordinates Method and can be applied to unstructured discretizations. The flux for each angular ordinate is swept across the computational grid, within a source iteration loop that accounts for the coupling between the different ordinates. The sequential nature of the sweep process makes the parallelization of the overall algorithm the most challenging aspect. Several parallel sweep algorithms, which represent different options of interleaving communications and calculations, are analyzed. One of the heuristics proposed consistently stands out as the best option in all the situations analyzed. With this algorithm, good scalability results have been achieved regarding both weak and strong speedup tests with up to 2560 CPUs

Spectral methods for CFD

One of the objectives of these notes is to provide a basic introduction to spectral methods with a particular emphasis on applications to computational fluid dynamics. Another objective is to summarize some of the most important developments in spectral methods in the last two years. The fundamentals of spectral methods for simple problems will be covered in depth, and the essential elements of several fluid dynamical applications will be sketched

Direct numerical simulation of turbulent wakes: flow past a sphere at Re=5000

The flow around bluff bodies is of great interest for a large number of engineering applications such as vehicle aerodynamics, cooling devices using forced convection, heat transfer enhancement using droplets, among others. Although the large number of experimental and numerical studies carried out for the flow past a sphere, there is still lack of detailed information about turbulent statistics in the wake of the sphere. Furthermore, most of the experimental studies consider that the sphere is supported by a stick by its rear side and aligned with the streamwise flow. The objectives of the present work are twofold: i) to carry out the DNS of the flow past a sphere at Re = 5000 in order to investigate the characteristics of turbulent flow in the wake of the sphere and provide detailed results of the first and second order turbulent statistics and, ii) to study the influence of the supporting mechanism in the wake con guration as in the turbulence statistics.Peer ReviewedPostprint (published version

Heterogeneous parallel algorithms for computational fluid dynamics on unstructured meshes

Frontiers of computational fluid dynamics (CFD) are constantly expanding and eagerly demanding more computational resources. Currently, we are experiencing an rapid evolution in the high performance computing systems driven by power consumption constraints. New HPC nodes incorporate accelerators that are used as math co-processors for increasing the throughput and the FLOP per watt ratio. On the other hand, multi-core CPUs have turned into energy efficient system-on-chip architectures. By doing so, the main components of the node are fused and integrated into a single chip reducing the energy costs. Nowadays, several institutions and governments are investing in the research and development of different aspects of HPC that could lead to the next generations of supercomputers. This initiatives have entitled the problem as the exascale challenge. This goal can only be achieved by incorporating major changes in computer architecture, memory design and network interfaces. The CFD community faces an important challenge: keep the pace at the rapid changes in the HPC resources. The codes and formulations need to be re-design in other to exploit the different levels of parallelism and complex memory hierarchies of the new heterogeneous systems. The main characteristics demanded to the new CFD software are: memory awareness, extreme concurrency, modularity and portability. This thesis is devoted to the study of a CFD algorithm re-factoring for the adoption of new technologies. Our application context is the solution of incompressible flows (DNS or LES) on unstructured meshes. The first approach was using GPUs for accelerating the Poisson solver, that is the most computational intensive part of our application. The positive results obtained in this first step motivated us to port the complete time integration phase of our application. This requires a major redesign of the code. We propose a portable implementation model for CFD applications. The main idea was substituting stencil data structures and kernels by algebraic storage formats and operators. By doing so, the algorithm was restructured into a minimal set of algebraic operations. The implementation strategy consisted in the creation of a low-level algebraic layer for computations on CPUs and GPUs, and a high-level user-friendly discretization layer for CPUs that is fully localized at the preprocessing stage where performance does not play an important role. As a result, at the time-integration phase the code relies only on three algebraic kernels: sparse-matrix-vector product (SpMV), linear combination of two vectors (AXPY) and dot product (DOT). Such a simple set of basic linear algebra operations naturally provides the desired portability to any computing architecture. Special attention was paid at the development of data structures compatibles with the stream processing model. A detailed performance analysis was studied in both sequential and parallel execution engaging up to 128 GPUs in a hybrid CPU/GPU supercomputer. Moreover, we tested the portable implementation model of TermoFluids code in the Mont-Blanc mobile-based supercomputer. The re-design of the kernels exploits a heterogeneous execution model using both computing devices CPU and GPU of the ARM-based nodes. The load balancing between the two computing devices exploits a tabu search strategy that tunes the workload distribution during the preprocessing stage. A comparison of the Mont-Blanc prototypes with high-end supercomputers in terms of the achieved net performance and energy consumption provided some guidelines of the behavior of CFD applications in ARM-based architectures. Finally, we present a memory aware auto-tuned Poisson solver for problems with one Fourier diagonalizable direction. This work was developed and tested in the BlueGene/Q Vesta supercomputer, and aims at demonstrating the relevance of vectorization and memory awareness for fully exploiting the modern energy efficient CPUs.Las fronteras de la dinámica de fluidos computacional (CFD) están en constante expansión y demandan más y más recursos computacionales. Actualmente, estamos experimentando una evolución en los sistemas de computación de alto rendimiento (HPC) impulsado por restricciones de consumo de energía. Los nuevos nodos HPC incorporan aceleradores que se utilizan como co-procesadores para incrementar el rendimiento y la relación FLOP por vatio. Por otro lado, CPUs multi-core se han convertido en arquitecturas system-on-chip. Hoy en día, varias instituciones y gobiernos están invirtiendo en la investigación y desarrollo de los diferentes aspectos de HPC que podrían llevar a las próximas generaciones de superordenadores. Estas iniciativas han titulado el problema como el "exascale challenge". Este objetivo sólo puede lograrse mediante la incorporación de cambios importantes en: la arquitectura de ordenador, diseño de la memoria y las interfaces de red. La comunidad de CFD se enfrenta a un reto importante: mantener el ritmo a los rápidos cambios en las infraestructuras de HPC. Los códigos y formulaciones necesitan ser rediseñados para explotar los diferentes niveles de paralelismo y complejas jerarquías de memoria de los nuevos sistemas heterogéneos. Las principales características exigidas al nuevo software CFD son: estructuras de datos, la concurrencia extrema, modularidad y portabilidad. Esta tesis está dedicada al estudio de un modelo de implementation CFD para la adopción de nuevas tecnologías. Nuestro contexto de aplicación es la solución de los flujos incompresibles (DNS o LES) en mallas no estructuradas. El primer enfoque se basó en utilizar GPUs para acelerar el solver de Poisson. Los resultados positivos obtenidos en este primer paso nos motivaron a la portabilidad completa de la fase de integración temporal de nuestra aplicación. Esto requiere un importante rediseño del código. Proponemos un modelo de implementacion portable para aplicaciones de CFD. La idea principal es sustituir las estructuras de datos de los stencils y kernels por formatos de almacenamiento algebraicos y operadores. La estrategia de implementación consistió en la creación de una capa algebraica de bajo nivel para los cálculos de CPU y GPU, y una capa de discretización fácil de usar de alto nivel para las CPU. Como resultado, la fase de integración temporal del código se basa sólo en tres funciones algebraicas: producto de una matriz dispersa con un vector (SPMV), combinación lineal de dos vectores (AXPY) y producto escalar (DOT). Además, se prestó especial atención en el desarrollo de estructuras de datos compatibles con el modelo stream processing. Un análisis detallado de rendimiento se ha estudiado tanto en ejecución secuencial y paralela utilizando hasta 128 GPUs en un superordenador híbrido CPU / GPU. Por otra parte, hemos probado el nuevo modelo de TermoFluids en el superordenador Mont-Blanc basado en tecnología móvil. El rediseño de las funciones explota un modelo de ejecución heterogénea utilizando tanto la CPU y la GPU de los nodos basados en arquitectura ARM. El equilibrio de carga entre las dos unidades de cálculo aprovecha una estrategia de búsqueda tabú que sintoniza la distribución de carga de trabajo durante la etapa de preprocesamiento. Una comparación de los prototipos Mont-Blanc con superordenadores de alta gama en términos de rendimiento y consumo de energía nos proporcionó algunas pautas del comportamiento de las aplicaciones CFD en arquitecturas basadas en ARM. Por último, se presenta una estructura de datos auto-sintonizada para el solver de Poisson en problemas con una dirección diagonalizable mediante una descomposicion de Fourier. Este trabajo fue desarrollado y probado en la superordenador BlueGene / Q Vesta, y tiene por objeto demostrar la relevancia de vectorización y las estructuras de datos para aprovechar plenamente las CPUs de los superodenadores modernos

A bibliography on parallel and vector numerical algorithms

This is a bibliography of numerical methods. It also includes a number of other references on machine architecture, programming language, and other topics of interest to scientific computing. Certain conference proceedings and anthologies which have been published in book form are listed also

Numerical analysis of conservative unstructured discretisations for low Mach flows

This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving. https://authorservices.wiley.com/author-resources/Journal-Authors/licensing-and-open-access/open-access/self-archiving.htmlUnstructured meshes allow easily representing complex geometries and to refine in regions of interest without adding control volumes in unnecessary regions. However, numerical schemes used on unstructured grids have to be properly defined in order to minimise numerical errors. An assessment of a low-Mach algorithm for laminar and turbulent flows on unstructured meshes using collocated and staggered formulations is presented. For staggered formulations using cell centred velocity reconstructions the standard first-order method is shown to be inaccurate in low Mach flows on unstructured grids. A recently proposed least squares procedure for incompressible flows is extended to the low Mach regime and shown to significantly improve the behaviour of the algorithm. Regarding collocated discretisations, the odd-even pressure decoupling is handled through a kinetic energy conserving flux interpolation scheme. This approach is shown to efficiently handle variable-density flows. Besides, different face interpolations schemes for unstructured meshes are analysed. A kinetic energy preserving scheme is applied to the momentum equations, namely the Symmetry-Preserving (SP) scheme. Furthermore, a new approach to define the far-neighbouring nodes of the QUICK scheme is presented and analysed. The method is suitable for both structured and unstructured grids, either uniform or not. The proposed algorithm and the spatial schemes are assessed against a function reconstruction, a differentially heated cavity and a turbulent self-igniting diffusion flame. It is shown that the proposed algorithm accurately represents unsteady variable-density flows. Furthermore, the QUICK schemes shows close to second order behaviour on unstructured meshes and the SP is reliably used in all computations.Peer ReviewedPostprint (author's final draft

An algebraic multigrid method for Q2−Q1Q_2-Q_1 mixed discretizations of the Navier-Stokes equations

Algebraic multigrid (AMG) preconditioners are considered for discretized systems of partial differential equations (PDEs) where unknowns associated with different physical quantities are not necessarily co-located at mesh points. Specifically, we investigate a $Q_2-Q_1$ mixed finite element discretization of the incompressible Navier-Stokes equations where the number of velocity nodes is much greater than the number of pressure nodes. Consequently, some velocity degrees-of-freedom (dofs) are defined at spatial locations where there are no corresponding pressure dofs. Thus, AMG approaches leveraging this co-located structure are not applicable. This paper instead proposes an automatic AMG coarsening that mimics certain pressure/velocity dof relationships of the $Q_2-Q_1$ discretization. The main idea is to first automatically define coarse pressures in a somewhat standard AMG fashion and then to carefully (but automatically) choose coarse velocity unknowns so that the spatial location relationship between pressure and velocity dofs resembles that on the finest grid. To define coefficients within the inter-grid transfers, an energy minimization AMG (EMIN-AMG) is utilized. EMIN-AMG is not tied to specific coarsening schemes and grid transfer sparsity patterns, and so it is applicable to the proposed coarsening. Numerical results highlighting solver performance are given on Stokes and incompressible Navier-Stokes problems.Comment: Submitted to a journa

Multiscale Modeling, Homogenization and Nonlocal Effects: Mathematical and Computational Issues

In this work, we review the connection between the subjects of homogenization and nonlocal modeling and discuss the relevant computational issues. By further exploring this connection, we hope to promote the cross fertilization of ideas from the different research fronts. We illustrate how homogenization may help characterizing the nature and the form of nonlocal interactions hypothesized in nonlocal models. We also offer some perspective on how studies of nonlocality may help the development of more effective numerical methods for homogenization

Numerical resolution of turbulent flows on complex geometries.

This thesis aims at developing a numerical methodology suitable for the direct numerical simulation (DNS) and large-eddy simulation (LES) of turbulent flows in order to be used in complex flows, currently encountered in industrial application. At the same time, the study of such turbulent flows can be an opportunity for gaining insight into the complex physics associated with them. To accomplish these goals, the mathematical formulation, conservative spatial discretization on unstructured grids and time- integration scheme for solving the Navier-Stokes equations are presented. The spatial discretization proposed preserves the symmetry properties of the continuous differential operator and ensure both, stability and conservation of the global kinetic energy balance on any grid. Furthermore, the time-integration technique proposed is an efficient self-adaptive strategy, based on a one-parameter second-order-explicit scheme, which has been successfully tested on both Cartesian staggered and unstructured collocated codes, leading to CPU cost reductions of up to 2.9 and 4.3, respectively. After presenting the general methodology for computing flows in complex geometries with unstructured grids, different LES models and regularization models suitable for these kind of meshes are presented and assessed by means of the analysis of different flows. First, regularization models are tested by means of the simulation of different cases with different level of complexity of the mesh. From a structured grid to a very complex mesh, with zones composed of prism and tetrahedral control volumes. It has been shown, that regularization models are very dependent on the quality of the filtering process. Although good results can be obtained with structured or smooth unstructuredmeshes, their performance is affected under fully irregular unstructured grids. A possible remedy to circumvent this issue is also presented. The main idea is to formulate the C4 model within a LES template. Although preliminary results are promising, further testing is still required. After regularization model assessment, LES models are also tested in a natural convection flow. It is shown that, although first order statistics are well solved for most of the models tested (with the exception of the Smagorinsky model), QR- and dynamic-Smagorinsky models present a better prediction of the second-order statistics. However, if CPU time is considered, then QR model is the best alternative. The second part of the thesis is devoted to the study of turbulent flows past bluff bodies. The cases studied are: the flow past a sphere, the flow past a circular cylinder and the flow past a NACA 0012 airfoil. All these cases shares some characteristics encountered in turbulent flows with massive separations, i.e., flow separation, transition to turbulence in the separated shear-layers and turbulent wakes with periodic shedding of vortices. However there are intrinsic characteristics of the turbulence in each of them, which make them interesting for the studying of the turbulence. Furthermore, the results presented for the flow past a sphere at Re = 3700 and 10000, together with the flow past a NACA 0012 at Re=50000 and AoA = 8 are the first DNS results presented in the literature for both flows. Conclusions drawn from the good results obtained point out that the use of the conservative formulation presented in this thesis, is one of the keys for the success of the SGS models used. This formulation, together with the use of unstructured grids might be a step towards the use of LES models for solving industrial flows on complex geometries at high Reynolds numbers.La present tesi proposa una metodologia apte per a realitzar simulacions directes de la turbulència (DNS) i simulacions de les grans escales (LES) de fluxos turbulents en geometries complexes. Tanmateix també s'estudia detalladament els mecanismes bàsics de funcionament dels fluxos turbulents en diferents situacions d'interès industrial i acadèmic. Per acomplir aquest objectiu s'ha desenvolupat una innovadora formulació matemàtica que permet conservar discretament les propietats continues de les equacions governants en malles no estructurades. La formulació proposada preserva la simetries originals dels operadors diferencials, assegurant així l'estabilitat i la conservació de l'energia cinètica turbulent en qualsevol mallat. Posteriorment s'ha proposat una metodologia d'integració temporal basada en una formulació explicita de segon ordre. Aquesta nova tècnica ha demostrat ser entre 2.9 i 4.3 més rapida que les tècniques anteriorment utilitzades per la comunitat. Un cop presentada la formulació per a simular fluxos turbulents en geometries complexes, s'han validat diferents models LES adaptats a malles no estructurades. Els models s'han testejat usant diferents solucions de referencia de la literatura i simulacions d'alt nivell generades en el context de la present tesi. Finalment s'ha conclòs que la conjunció de la formulació bàsica proposada amb alguns del models LES sorgits en els darrers anys es molt efectiva per a simular fluxos turbulents en situacions complexes, essent el Variational Multiscale WALE i el model QR els més adequats per a simular situacions de interes industrial. La segona part de la tesi es dedicada a l'estudi aerodinàmic del flux turbulent al voltant de diferents perfils. El perfils seleccionats son: el flux al voltant d'una esfera, flux al voltant d'un cilindre i flux al voltant d'un perfil NACA 0012. Els tres casos comparteixen fenomenologies com ara separació massiva de capes límits, esteles turbulentes i desprendiment periòdic de remolins. Tot i així cadascun d'ells es comporta diferent a nivell turbulent així que es d'interès estudiar-los i entendre quins son les causes de les diferencies físiques que es troben. Cal recordar que la física estudiada es la que es pot trobar posteriorment en ales d'avió, perfils de turbines de vent, aerodinàmica de cotxes, etc. Finalment recalcar que els resultats DNS del flux al voltant de l'esfera a Re=3700 i Re=10000 conjuntament amb els DNS del flux al voltant del perfil NACA a Re=50000 i AoA =8 son els primers presentats en la literatura internacional en el seu àmbit. Finalment es pot concloure que la formulació conservativa presentada en la tesis juntament amb els diferents models LES d'última generació testejats en la tesis, han demostrat ser una eina eficaç tan per a resoldre fluxos turbulents d'interès acadèmic com per simular situacions d'interès industrial.Postprint (published version

Parallel harmonic balance method for analysis of nonlinear mechanical systems

Mechanical vibration analysis and modelling are essential tools used in the design of various mechanical components and structures. In the case of turbine engine design specifically, the ability to accurately predict vibration of various parts is crucial to ensure their safe operation while maintaining efficiency. As the designs become increasingly complex and margins for errors get smaller, high fidelity numerical vibration models are necessary for their analysis. Research of parallel algorithms has progressed significantly in the last decades, thanks to the exponential growth of the world's available computational resources. This work explores the possibilities for parallel implementations for solving large scale nonlinear vibration problems. A C++ code using MPI was developed to validate these implementations in practice. The harmonic balance method is used in combination with finite elements discretisation and applied to an elastic body with the Green-Lagrange nonlinear model for large deformations. A parameter continuation scheme using a predictor-corrector approach is included to compute frequency response functions. A Newton-Raphson solver is used to solve the bordered nonlinear system of equations in the frequency domain. Three different parallel algorithms for solving the linearised problem in each Newton iteration are analysed - a sparse direct solver (using MUMPS library), GMRES (using PETSc library) and an inhouse implementation of FETI. The performance of the solvers is analysed using beam testcases and a fan blade geometry. Scalability of MUMPS and the FETI solver is assessed. Full nonlinear frequency response functions with turning points are also computed. Use of artificial coarse space and preconditioning in FETI is discussed as it greatly impacts convergence properties of the solver. The presented parallel linear solvers show promising scalability results and an ability to solve nonlinear systems of several million degrees of freedom.Open Acces