Proceedings of: Second International Workshop on Sustainable Ultrascale Computing Systems (NESUS 2015). Krakow (Poland), September 10-11, 2015.This work was supported by the European Regional Development Fund in the IT4Innovations Centre of Excellence project (CZ.1.05/1.1.00/02.0070) and the COST NESUS project with an additional CZ MEYS LD15105 support

Blaheta, Radim

Georgiev, Ivan

Jakl, Ondrej

Kolcun, Alexej

Soucek, Kamil

Starý, Jirí

Stary, Jiri

Name not available

HPC in Computational Micromechanics of Composite Materials

Universidad Carlos III de Madrid e-Archivo

Proceedings of the Second International Workshop on SustainableUltrascale Computing Systems (NESUS 2015)Krakow, PolandJesus Carretero, Javier Garcia BlasRoman Wyrzykowski, Emmanuel Jeannot.(Editors)September 10-11, 2015NESUSNetwork for Sustainable Ultrascale ComputingIC1305Book paper template • September 2015 • Vol. I, No. 1HPC in Computational Micromechanicsof Composite Materials(Poster Summary)Radim Blaheta, Alexej Kolcun, Ondrˇej Jakl, Kamil Soucˇek, Jirˇí Starý, Ivan GeorgievInstitute of Geonics, Czech Academy of Sciences, Ostrava, Czech RepublicInstitute of Information and Communication Technologies, Bulgarian Academy of Sciences, Sofia, Bulgaria[blaheta, kolcun, jakl, soucek, stary]@ugn.cas.cz, ivan.georgiev@parallel.bas.bgI. IntroductionBy micromechanics we understand analysis of themacroscale response of materials through investiga-tion of processes in their microstructure. Here by themacroscale, we mean the scale of applications, wherewe solve engineering problems involving materials.Examples could be analysis of aircraft constructionswith different composite materials and analysis of rockbehaviour and concrete properties in geo- and civil en-gineering applications. Analysis of bio-materials withmany medicine applications is also rapidly developing.Different applications are distinguished by differentcharacteristic size. At macroscale the materials mostlylook as homogeneous or they are idealized as homo-geneous or piecewise homogeneous. A substantialheterogeneity is hidden and appears only after moredetailed zooming view into the material. This hiddenheterogeneity can be called microstructure. In metalsit is created by crystals and grains, in composite ma-terials by matrix and inclusions, in concrete by graveland mortar or iron reinforcement, in rock by mineralcomposition and possible grouting, etc. When the ratiobetween the characteristic dimensions on macro andmicrostructure subjects is sufficiently large, then we saythat the scales are well separated. In this case, it is notpossible to perform the macroscale analysis going intothe microstructure details, but it is possible to analysethe macroscopic problems with the use of effective (ho-mogenized) material properties, which are obtained bytesting smaller samples of materials. In computationalmicromechanics, the testing of such samples meanssolution of boundary value problems on test domainsinvolving the microstructure with loading provided bysuitable boundary conditions.In this work, we focus on X-ray CT image basedmicromechanics of geomaterials and concrete with theuse of continuum mechanics and finite element (FE)computations of the microscale strains and stresses,see [2]. This means that basic information about themicrostructure is provided by the analysis (segmenta-tion) of 3D images of real samples. This informationshould be complemented by information on local mate-rial properties, i.e. material properties of the individualmaterial constituents.There is a strong need for high performance comput-ing (HPC) at several levels of computational microme-chanics, namely at:• analysis of CT scans,• high resolution finite element solution of bound-ary value problems,• solution of inverse problems for determination orcalibration of local material properties.This contribution deals with the second point, i.e.solution of high resolution FE systems with tens orhundreds million degrees of freedom (DOF). We reportabout the performance of in-house solvers exploitingthe Schwarz domain decomposition method with aggrega-tion, c.f. [3], and outline possible future developmentin the area of ultrascale computing, which is necessaryfor building efficient solution methods for inverse mate-rial identification problems, see [4] and work in progress.1Radim Blaheta, Alexej Kolcun, Ondrej Jakl, Kamil Soucek, Jiri Stary,Ivan Georgiev 143Book paper template • September 2015 • Vol. I, No. 1II. High resolution FE systemsIn the analysis of heterogeneous materials with mi-crostructure (composites) (see [2]), our test sample do-main Ω is a cube with a relatively complicated mi-crostructure. Its FE mesh is constructed from dataobtained from industrial CT scanning, performed atthe CT-lab of the Institute of Geonics. Two types ofcomposites are considered in this work: A coal-resingeocomposites and reinforced concrete composites. SeeTab. 1 for the main characteristics of the resulting FEproblems (benchmarks).Benchmark Discretization Size in DOF Data sizeGEOC-2l 257 × 257 × 1025 203 100 675 33.5 GBFIBER-3 401 × 401 × 401 193 443 603 32.2 GBTable 1: Benchmark problems: Name, discretization, size ofthe resulting linear system and storage requirements.The elastic response of a representative volume Ωis characterized by homogenized elasticity C or com-pliance S tensors (S = C−1). The elasticity and com-pliance tensors are basically determined from the rela-tionsC〈ε〉 = Cε0 = 〈σ〉 and S〈σ〉 = Sσ0 = 〈ε〉, (1)respectively. Here 〈σ〉 and 〈ε〉 are volume averagedstresses and strains computed from the solution ofelasticity problem−div(σ) = 0, σ = Cmε, ε = (∇u+(∇u)T)/2 in Ω,(2)with boundary conditionsu(x) = ε0 · x on ∂Ω and σ · n = σ0 · n on ∂Ω, (3)respectively. Above, σ and ε denote stress and strainin the microstructure, Cm is the variable local elasticitytensor, u and n denote the displacement and the unitnormal, respectively. The use of pure Dirichlet andpure Neumann boundary conditions allows us to geta upper and lower bounds for the upscaled elasticitytensor, see e.g. [4].III. GEM solversGEM is an in-house FE software described in detailin [1], which makes use of linear tetrahedral finiteelements for discretization. Arising systems of lin-ear equations are processed by solvers based on thepreconditioned conjugate gradient (PCG) method, withstabilization in the singular case [3]. PCG uses overlap-ping domain decomposition preconditioners. To solve thebenchmarks, we employ two types of GEM solvers:GEM-DD solver uses one-level additive Schwarz domaindecomposition preconditioner with subproblemsreplaced by displacement decomposition incom-plete factorization, see ref. in [3]. The resultingpreconditioner is symmetric positive definite evenfor the singular case.GEM-DD+CG solver implements two-level Schwarzdomain decomposition preconditioning, arisingfrom the GEM-DD above by additive involvementof a coarse problem correction. The coarse problem iscreated by a regular aggressive aggregation with3 DoF’ per aggregation. In the singular case, thecoarse problem is also singular with a smaller nullspace containing only the rigid shifts. The coarseproblem is solved only approximately by inner(not stabilized) CG method with a lower solutionaccuracy – relative residual accuracy ε0 ≤ 0.01.IV. Computing resourcesThe computations were performed on two parallelplatforms:Enna 64-core NUMA multiprocessor, Institute of Geon-ics AS CR: eight octa-core Intel Xeon E7-8837 /2.66 GHz processors; 512 GB of DDR2 RAM; Cen-tOS 6.3, Intel Cluster Studio XE 2013.Anselm multicomputer (cluster, 209 compute nodes),IT4Innovations National Supercomputing Center:two octa-core Intel E5-2665 / 2.4 GHz processorsper node; 64 GB RAM per node; Infiniband QDRinterconnection, fully non-blocking, fat-tree; BullxLinux Server 6.3 (Red Hat clone), Intel ParallelStudio 13.1.2144 HPC in Computational Micromechanics of Composite MaterialsBook paper template • September 2015 • Vol. I, No. 1V. Computational experimentsTable 2 shows the timings of the GEM-DD+CG solver(with coarse grid problem applied) obtained for thecoal-resin geocomposite benchmark GEOC-2l both onEnna and Anselm, and demonstrates the impact ofthe coarse grid size on the time of the solution. Thestopping criterion ‖r‖/‖b‖ ≤ ε, based on the relativeresidual accuracy, was 10−5. On Enna, the best results(2483.6 s) were observed with aggregation 9× 9× 9.# SdEnna AnselmDD + CG DD + CG DD + CG DD + CG9× 9× 9 9× 9× 18 9× 9× 27 9× 9× 27# It Titer # It Titer # It Titer # It Titer4 751 13719.0 858 15757.6 997 18518.4 997 12671.48 690 6237.7 800 6960.8 917 8062.9 917 5803.916 585 2717.4 674 4010.6 777 4815.6 777 2576.632 585 2483.6 622 2923.8 708 3452.5 708 1157.564 627 3637.0 627 558.8128 652 358.5256 631 299.6512 649 333.5Table 2: Timings of the GEOC-2l benchmark with pureNeumann boundary conditions and achieved by the GEM-DD+CG solver on the multiprocessor Enna and multicom-puter Anselm: Iteration counts (# It) and wall-clock time (inseconds) for the solution time (Titer) are provided now fordifferent sizes of CG problem involved in computations andfor various numbers of subdomains (# Sd).The experiments confirm the advantage of multicom-puters (systems with distributed memory) for greaternumber of subdomains, when the multiprocessors ingeneral suffer from the memory-processor bandwidthcontention. Thus, while on Enna the scalability fadesout at about 32 cores, the turning point on Anselm isaround 256 processing elements, when the small sizeof subdomains deteriorates the ratio between computa-tion and communication. In absolute figures, we wereable to solve the benchmark 8 times faster on Anselmthan on Enna. A part of Anselm’s advantage is to becredited to its newer Intel Sandy Bridge CPU architec-ture, which outperforms Enna’s Westmere CPU in ourapplications by 20 - 40 % (separate test).A bit surprising decrease of the number of iterationswith increasing number of subdomains (processors) asreported in the above Tables, can be explained by thefact that smaller subdomain problems are solved moreaccurately in our implementation.Anselm/GEOC-2l best time in Table 2 (299.6 s with256 processing elements and aggregation 9× 9× 27)was surpassed by another experiment (not shown inthe table): aggregation 15× 15× 31, 910 iterations, 512subdomains (32 compute nodes employed), 249.8 s.The experiments carried out on the fiber-reinforcedconcrete FIBER-3 benchmark delivered similar results.So far, just Enna has been used for computations, con-firming limited scalability on its multi-processor archi-tecture. We observed great importance of the coarsegrid and its proper dimensioning for efficient solution(computing times of GEM-DD are multiples of thecomputing times of GEM-DD+CG).VI. Conclusions and future workMicromechanics leads to large-scale problems, as il-lustrated by the presented benchmarks. The compu-tational requirements can be further substantially in-creased in the case of an inverse analysis for identificationlocal material properties (see e.g. [4]).At the IT4Innovations National SupercomputingCenter, there is a new massively parallel computeravailable, called Salomon. This multi-computer (clus-ter) has 1008 compute nodes and we plan to employ itin future experiments.The approach described so far employs classical do-main decomposition philosophy. Both facts, computa-tional demands and availability of massively parallelcomputers, motivate further research in algorithms,which are efficient from the point of view of arithmeticoperations and (even more important) from the pointof view of communication.In the future, we plan to test the effect of communi-cation avoiding (CA) algorithms. For example, usingthe same ingredients as in our domain decompositionsolvers, we can employ CA conjugate gradients anda deflation type implementation of the aggregationbased coarse space.Acknowledgement: This work was supportedby the European Regional Development Fund inthe IT4Innovations Centre of Excellence project3Radim Blaheta, Alexej Kolcun, Ondrej Jakl, Kamil Soucek, Jiri Stary,Ivan Georgiev 145Book paper template • September 2015 • Vol. I, No. 1(CZ.1.05/1.1.00/02.0070) and the COST NESUS projectwith an additional CZ MEYS LD15105 support.References[1] R. Blaheta, O. Jakl, R. Kohut, J. Starý: GEM - APlatform for Advanced Mathematical Geosimulations.In: R. Wyrzykowski et al. (eds.): PPAM 2009, PartI, LNCS 6067, Springer-Verlag, 2010, pp. 266-275.[2] R. Blaheta et al: Material parameter identificationwith parallel processing and geo-applications. In: R.Wyrzykowski et al. (eds.): PPAM 2011, Part I, LNCS7203, Springer-Verlag, 2012, pp. 366-375.[3] R. Blaheta, O. Jakl, J. Starý, E. Turan: Parallel solversfor numerical upscaling. In: P. Manninen, P. Oster(eds.): PARA 2012, LNCS 7782, Springer-Verlag,2013, pp. 375-386.[4] R. Blaheta, R. Kohut, A. Kolcun, K. Soucˇek, L. Staš,L. Vavro: Digital image based numerical micromechan-ics of geocomposites with application to chemical grout-ing. International Journal of Rock Mechanics andMining Sciences 77, 2015, pp. 77-88.4146 HPC in Computational Micromechanics of Composite Materials

http://e-archivo.uc3m.es/bitstream/handle/10016/22008/hpc_NESUS_2015.pdf?sequence=1&isAllowed=y

HPC in Computational Micromechanics of Composite Materials

Abstract

Similar works

Full text

Available Versions

Name not available

Name not available

Universidad Carlos III de Madrid e-Archivo