Optimised hybrid parallelisation of a CFD code on Many Core architectures

COSA is a novel CFD system based on the compressible Navier-Stokes model for unsteady aerodynamics and aeroelasticity of fixed structures, rotary wings and turbomachinery blades. It includes a steady, time domain, and harmonic balance flow solver. COSA has primarily been parallelised using MPI, but there is also a hybrid parallelisation that adds OpenMP functionality to the MPI parallelisation to enable larger number of cores to be utilised for a given simulation as the MPI parallelisation is limited to the number of geometric partitions (or blocks) in the simulation, or to exploit multi-threaded hardware where appropriate. This paper outlines the work undertaken to optimise these two parallelisation strategies, improving the efficiency of both and therefore reducing the computational time required to compute simulations. We also analyse the power consumption of the code on a range of leading HPC systems to further understand the performance of the code.Comment: Submitted to the SC13 conference, 10 pages with 8 figure

Computational Aerodynamics on unstructed meshes

New 2D and 3D unstructured-grid based flow solvers have been developed for simulating steady compressible flows for aerodynamic applications. The codes employ the full compressible Euler/Navier-Stokes equations. The Spalart-Al Imaras one equation turbulence model is used to model turbulence effects of flows. The spatial discretisation has been obtained using a cell-centred finite volume scheme on unstructured-grids, consisting of triangles in 2D and of tetrahedral and prismatic elements in 3D. The temporal discretisation has been obtained with an explicit multistage Runge-Kutta scheme. An "inflation" mesh generation technique is introduced to effectively reduce the difficulty in generating highly stretched 2D/3D viscous grids in regions near solid surfaces. The explicit flow method is accelerated by the use of a multigrid method with consideration of the high grid aspect ratio in viscous flow simulations. A solution mesh adaptation technique is incorporated to improve the overall accuracy of the 2D inviscid and viscous flow solutions. The 3D flow solvers are parallelised in a MIMD fashion aimed at a PC cluster system to reduce the computing time for aerodynamic applications. The numerical methods are first applied to several 2D inviscid flow cases, including subsonic flow in a bump channel, transonic flow around a NACA0012 airfoil and transonic flow around the RAE 2822 airfoil to validate the numerical algorithms. The rest of the 2D case studies concentrate on viscous flow simulations including laminar/turbulent flow over a flat plate, transonic turbulent flow over the RAE 2822 airfoil, and low speed turbulent flows in a turbine cascade with massive separations. The results are compared to experimental data to assess the accuracy of the method. The over resolved problem with mesh adaptation on viscous flow simulations is addressed with a two phase mesh reconstruction procedure. The solution convergence rate with the aspect ratio adaptive multigrid method and the direct connectivity based multigrid is assessed in several viscous turbulent flow simulations. Several 3D test cases are presented to validate the numerical algorithms for solving Euler/Navier-Stokes equations. Inviscid flow around the M6 wing airfoil is simulated on the tetrahedron based 3D flow solver with an upwind scheme and spatial second order finite volume method. The efficiency of the multigrid for inviscid flow simulations is examined. The efficiency of the parallelised 3D flow solver and the PC cluster system is assessed with simulations of the same case with different partitioning schemes. The present parallelised 3D flow solvers on the PC cluster system show satisfactory parallel computing performance. Turbulent flows over a flat plate are simulated with the tetrahedron based and prismatic based flow solver to validate the viscous term treatment. Next, simulation of turbulent flow over the M6 wing is carried out with the parallelised 3D flow solvers to demonstrate the overall accuracy of the algorithms and the efficiency of the multigrid method. The results show very good agreement with experimental data. A highly stretched and well-formed computational grid near the solid wall and wake regions is generated with the "inflation" method. The aspect ratio adaptive multigrid displayed a good acceleration rate. Finally, low speed flow around the NREL Phase 11 Wind turbine is simulated and the results are compared to the experimental data

Strategies and tools for the exploitation of massively parallel computer systems

The aim of this thesis is to develop software and strategies for the exploitation of parallel computer hardware, in particular distributed memory systems, and embedding these strategies within a parallelisation tool to allow the automatic generation of these strategies. The parallelisation of four structured mesh codes using the Computer Aided Parallelisation Tools provided a good initial parallelisation of the codes. However, investigation revealed that simple optimisation of the communications within these codes provided an even better improvement in performance. The dominant factor within the communications was the data transfer time with communication start-up latencies also significant. This was significant throughout the codes but especially in sections of pipelined code where there were large amounts of communication present. This thesis describes the development and testing of the methods used to increase the performance of these communications by overlapping them with unrelated calculation. This method of overlapping the communications was applied to the exchange of data communications as well as the pipelined communications. The successful application by hand provided the motivation for these methods to be incorporated and automatically generated within the Computer Aided Parallelisation Tools. These methods were integrated within these tools as an additional stage of the parallelisation. This required a generic algorithm that made use of many of the symbolic algebra tests and symbolic variable manipulation routines within the tools. The automatic generation of overlapped communications was applied to the four codes previously parallelised as well as a further three codes, one of which was a real world Computational Fluid Dynamics code. The methods to apply automatic generation of overlapped communications to unstructured mesh codes were also discussed. These methods are similar to those applied to the structured mesh codes and their automation is viewed to be of a similar fashion

Large eddy simulation of primary liquid-sheet breakup

This research project aims at providing the aeronautical industry with a modelling capability to simulate the fuel injection in gas turbine combustion chambers. The path to this objective started with the review of state-of-the-art numerical techniques to model the primary breakup of liquid fuel into droplets. Based on this and keeping in mind the requirements of the industry, our modelling strategy led to the generation of a mass-conservative method for efficient atomisation modelling on unstructured meshes. This goal has been reached with the creation of high-order numerical schemes for unstructured grids, the development of an efficient numerical method that transports the liquid-vapour interface accurately while conserving mass and the implementation of an algorithm that outputs the droplet boundary conditions to separate combustion codes. Both high-order linear and WENO schemes have been created for general polyhedral meshes. The notorious complexity of high-order schemes on 3D mixed-element meshes has been handled by the creation of a series of algorithms. These include the tetrahedralisation of the mesh, which allows generality of the approach while remaining efficient and affordable, together with a novel approach to stencil generation and a faster interpolation of the solution. The performance of the scheme has been demonstrated on typical two-dimensional and three-dimensional test cases for both linear and non-linear hyperbolic partial differential equations. The conservative level set method has been extended to unstructured meshes and its performance has been improved in terms of robustness and accuracy. This was achieved by solving the equations for the transport of the liquid volume fraction with our novel WENO scheme for polyhedral meshes and by adding a flux-limiter algorithm. The resulting method, named robust conservative level set, conserves mass to machine accuracy and its ability to capture the physics of the atomisation is demonstrated in this thesis. To be readily applicable to the simulation of atomisation, the novel interfacecapturing technique has been embedded in a framework — within the open source CFD code OpenFOAM — that solves the velocity and pressure fields, outputs droplet characteristics and runs in parallel. In particular, the production of droplet boundary conditions involves a set of routines handling the selection of drops in the level set field, the calculation of relevant droplet characteristics and their storage into data files. An n-halo parallelisation method has been implemented in OpenFOAM to perform the computations at the expected order of accuracy. Finally, the modelling capability has been demonstrated on the simulation of primary liquid-sheet breakup with relevance to fuel injection in aero-engine combustors. The computation has demonstrated the ability of the code to capture the physics accurately and further illustrates the potential of the numerical approach

Time- and frequency-domain turbulent flow analysis of wind turbine unsteady aerodynamics

The main objective of the research work presented in this thesis is the development of a single aerodynamic CFD code for the analysis of complex turbulent flow unsteady aerodynamics such as those encountered in horizontal and vertical axis wind turbines. The finite volume parallel CFD Optimized Structured multi-block Algorithm (COSA) research code solves the Navier-Stokes equations on structured multi-block grids and models turbulence effects with Menter's shear stress transport turbulence model. The novel algorithmic contribution of this research is the successful development of a Harmonic Balance (HB) solver which can reduce the run-time required to compute nonlinear periodic flow fields with respect to the conventional time-domain (TD) approach. The thesis also presents a semi-implicit integration based on LU factorisation and a successfully LAPACK libraries integration to massively improve the computational efficiency of the integration of the HB RANS equations and the turbulence model of Menter. The main computational results of this research are for two low-speed renewable energy applications. The former application is a turbulent unsteady flow analysis of a vertical axis wind turbine working in a low-speed turbulent regime for a wide range of operating conditions. The test case is first solved using the COSA TD turbulent solver to analyse and discuss in great detail the unsteady aerodynamic phenomena occurring in all regimes of this complex device. During the turbine rotation there is a generation of blade vortex shedding and wakes all around the rotor which interacts with the blades itself on the returning side. The most important features of the investigated devices were captured with CFD. In addition, a series of investigations have been conducted to analyse the effects of computational domain refinement, number of time steps per revolution and distance of the farfield boundary from the rotor centre on prediction accuracy. The solution of the turbulent flow solver is validated by comparing torque and power coefficients with experimental data and numerical solutions obtained with a state-of-the-art time-domain of commercial package regularly used by the industry and the Academia worldwide. A detailed selection of results is presented, dealing with the various investigated issues. Afterwards, the COSA HB turbulent solver is used to solve the problem and compare the HB resolution and speed-ups with the TD results. The main motivation for analysing this problem is to highlight the predictive capabilities and the numerical robustness of the developed turbulent HB flow solver for complex realistic problems with a strong nonlinearity and to shed more light on the complex physics of this renewable energy device. The latter application regards the turbulent unsteady flow analysis of horizontal axis wind turbine blade sections in yawed wind regime. The TD and HB turbulent flow analysis of a 164 m-diameter wind turbine rotor is performed. CFD represents an accurate design tool to get a better understanding of the physical behaviour of the flow field past wind turbine rotors and the importance of accurate design is increased as the machines tend to become larger. A study at 30% and at 85% blade section is carried out, allowing the analysis of the unsteady forces acting on two different blade sections. The aim of these analyses is to assess the computational benefits achievable by using the HB method for a common nonlinear flow problem and also to further demonstrate the predictive capabilities of the developed CFD system. The turbulent HB solutions highlight that is possible to obtain an accurate analysis as its TD counterparts can do. Moreover, the results highlight that the turbulent HB solver can compute the hysteresis force cycles of the turbine blade more than 10 times faster than the TD approach. The purpose of proving the turbulent COSA HB capabilities for studying the flow field of wind turbines rotor has been fully achieved and this research represent one of the first turbulent HB RANS applications to the analysis of periodic horizontal axis wind turbine flows, and the first application to vertical axis wind turbine flows