We propose a new problem formulation for graph partitioning that is tailored to the needs of time-critical simulations on modern heterogeneous supercomputers

Langguth, Johannes

Schlag, Sebastian

Schulz, Christian

English

arXiv

KITopen

Load-Balanced Bottleneck Objectives in Process Mapping∗Johannes Langguth† Sebastian Schlag‡ Christian Schulz§AbstractWe propose a new problem formulation for graph partition-ing that is tailored to the needs of time-critical simulationson modern heterogeneous supercomputers.1 IntroductionAmong the many combinatorial problems, graph parti-tioning (GP) has a central role in the area of parallelhigh performance computing. Irregular inputs such asgraphs, sparse matrices and many meshes typically haveto be distributed among the nodes of a supercomputerto allow parallel processing, and the distribution shouldboth balance the workload and minimize the communi-cation. In the last two decades, sophisticated softwarehas been developed for computing high-quality parti-tionings fast. However, the underlying model, which as-sumes homogeneous nodes connected by a full-bisectionbandwidth network, no longer represents current super-computers. We therefore propose a new problem formu-lation which overcomes these drawbacks and discuss itssuitability for future challenges in parallel computing.Irregular inputs can roughly be divided into twomain groups. The first consists of problems whosedata access pattern resembles sparse-matrix-dense-vector multiplications (SpMV), where the communica-tion is irregular but follows a repeated pattern, suchas PageRank [21] and many scientific computations onirregular meshes. The second group represents prob-lems resembling sparse-matrix-sparse-vector multiplica-tions (SpMSpV). Here, only a subset of vertices is ac-tive at a time, leading to changing communication pat-terns. While minimizing cut sizes helps to reduce thecommunication cost in both cases, in the first case, theoverall communication time is determined by the max-imum communication per node, assuming the networklinks of a distributed memory computer operate inde-pendently. Thus, minimizing the maximum (i.e., the∗This work was partially supported by the Austrian ScienceFoundation (FWF, project P 31763-N31), and it was partiallysupported by the Research Council of Norway (project RCN251186).†Simula Research Laboratory, Oslo, Norway‡Karlsruhe Institute of Technology, Karlsruhe, Germany§University of Vienna, Faculty of Computer Science, Vienna,Austriamakespan) is a better partitioning objective, and as-suming there are no slowdowns such as concurrent traf-fic in the network, it is actually optimal in this case. Inthe second case, it is harder to compare the partitioningobjectives. In low-diameter graphs such as social net-works, breadth-first search and similar algorithms typi-cally perform a low number of high-volume communica-tion rounds. Consequently, minimizing the maximumcommunication among the nodes may lead to betterresults than cut-size optimization. For high diametergraphs this no longer holds, which renders makespanpartitioning less attractive in this case.Finally, current single-criterion partitioning doesnot allow trade-offs between load balance and cut size.For SPMV type computations, where both commu-nication and computation performance is typically toa certain degree predictable, incorporating this trade-off has the potential to provide better solutions thanusing a fixed load-balance constraint on the vertices.And while SpMSpV computations are typically less pre-dictable, they are often highly communication-bound,which means that load-balance is less important thancommunication minimization and might benefit fromthe trade-off formulation. All this however implies thatwe must provide a ratio between the cost of communi-cation and computation for the trade-off to function.2 Related WorkThere has been an enormous amount of research onGP, and we refer the reader to Refs. [2, 3, 26] for ex-tensive material and references. The most commonobjective function is to minimize the total cut size∑i<j ω(Eij). There are well known software packagesbased to find partitions minimizing this objective in-cluding KaHIP [24], Jostle [29], Metis [15], and Scotch[22]. A related objective is to minimize the maximumcut size maxi<j ω(Eij). The first accurate communica-tion volume metrics for sparse-matrix vector multipli-cation, using hypergraph models, are due to Catalyurekand Aykanat [5]. When graph partitioning is usedin parallel computing to map the graph nodes to dif-ferent processors, the communication volume is oftenmore representative than the cut [13]. Let V` be theblock containing vertex v. Then, we let D(v) denotethe number of blocks in which vertex v has a neigh-bor, excluding the block containing v and c(v) be avertex weight. That is, D(v) = |{j | ∃u ∈ Vj 6=V` s.t. {u, v} ∈ E}|. For a block Vi, the communica-tion volume is cvol(Vi) :=∑v∈Vi c(v)D(v). Similar tocut size, one can minimize the maximum communica-tion volume max1≤i≤k cvol(Vi), or the total communi-cation volume∑1≤i≤k cvol(Vi). The choice of eitherthe total or maximum variant of the objective functionhere is typically dictated by the topology of the inter-connection network between the machines [13]. Otherobjectives are less common, these include the maxi-mum degree in the quotient graph (the graph formedby blocks and their connections), expansion and conduc-tance [16], and optimizing the “shape” of partitions [20].Specialty partitioners tailor-made for applications cangive better quality [7]. Furthermore, there are meth-ods for maximizing multiple objective functions simul-taneously [25, 28, 6], and finding Pareto-optimal solu-tions [11].There is likewise a large literature on process map-ping, i.e., decomposition techniques that takes the givenconnection network into account, often in the context ofscientific applications using MPI (Message-Passing In-terface). Refs. [30, 23, 4, 12] were among the first totackle the process mapping problem. Hatazaki [12] andWalshaw [30] proposed graph partitioning to solve theMPI process mapping problem for unweighted processtopologies. Mercier and Clet-Ortega and later Jean-not [18, 19] simplify the mapping problem by only con-sidering the topology inside the compute nodes them-selves and ignoring the topology of the network. Multi-ple placement policies are investigated to enhance over-all system performance. Yu et al. [31] discuss and im-plement embedding heuristics for the BlueGene 3d torussystem. Hoefler and Snir [14] instead optimize the con-gestion of the mapping and show that this problem isNP-complete. Routing-aware mapping heuristics tak-ing the hierarchy of specific hardware topologies intoaccount were discussed in Ref. [1]. Later, based on avariety of techniques, more sophisticated algorithms tocompute mappings for hierarchical systems have beenpublished [27, 10, 8].The Lynx code [17] is an SpMV-type computationfor large scale simulations in cardiac electrophysiologyon heterogeneous supercomputers. As the problem isrelatively communication heavy, high-quality partition-ing is crucial for its performance, especially when usingnodes equipped with multiple GPUs. Scalability wasachieved via hierarchical partitioning. However, dueto the lack of actual hierarchical partitioning software,it was emulated by applying conventional partitioningtwice. This proved to be highly effective, but difficultto program. As this type of compute node has becomecommon in modern supercomputers, we believe that theneed for hierarchical partitioning software will increasein the near future.3 Problem SpecificationWe define the basic version of the graph-constrainedmakespan partitioning problem as follows:Given a tree C = (B,L), an undirected graphG = (V,E), and a communication cost factor F . Find apartitioning P : V → B which maps each vertex v ∈ Vto a vertex b ∈ B such that the makespan is minimized.We refer to the vertices B of C as bins, and the edgesL as links. The makespan M is defined as:M(P ) = max(maxb∈Bcomp(b), F ·maxl∈Lcomm(l))where comp(b) = |{v ∈ V | P (v) = b}| is the com-putational load of vertex b ∈ B, comm(l) = |{{u, v} ∈E | l ∈ STP (P (u), P (v))}| is the communication vol-ume along edge l ∈ L, and STP (P (u), P (v)) is theshortest path (as a set of edges) between the bins P (u)and P (v) that the endpoints u and v of some edge wereassigned to in P . Because C is a tree, the shortest pathis unique. Note that C is unweighted, which means thatthe length of a path is simply the number of edges con-tained in it. This is based on the intuition that if anedge is cut, communication will be necessary betweenthe compute nodes represented by these bins.3.1 Generalizations. The most basic generalizationis the introduction of a set R ⊂ B of routers, i.e.,bins r ∈ R that cannot be assigned any work (i.e.,load(r) = 0 ∀r ∈ R). This is needed to correctly modelthe networks of most supercomputers. We refer to thisvariant as the interconnect graph-constrained makespanpartitioning problem.To take more complicated interconnects with dif-ferent routing protocols into account, the protocols be-come part of the problem formulation. Thus, the routinggraph-constrained partitioning problem differs from thestandard version in that C is no longer required to be atree. Any algorithm for this problem is given access toan oracle (such as a routing table) which for every pairof bins returns a unique path between them. If we allowfor multipath routing, the oracle may instead return aset of paths. In that case, assuming k paths, each pathP adds 1/k to the load of each edge l ∈ P.The hierarchical network structure of interconnecttopologies such as fat trees implies that some linksneed to have much higher capacities than the injectionbandwidth of a single node. In order to model this,we change the communication factor F from globallyapplying to all links to a link-specific factor Fl, therebyobtaining the edge-weighted variant. By the same token,we can define a vertex-weighted variant where everyv ∈ V is given a weight w(v). The load of a bin isthen computed as the sum of the weights of all verticesassigned to that bin.3.2 NP Hardness. Unlike the traditional graph par-titioning problem, graph constrained makespan parti-tioning does not have a simple reduction from Mini-mum Bisection. However, the vertex weighted casecan easily be proven to be NP-Hard by reduction fromMinimum multiprocessor scheduling [9].References[1] A. H. Abdel-Gawad, M. Thottethodi, and A. Bhatele.RAHTM: routing algorithm aware hierarchical taskmapping. In Intl. Conference for High PerformanceComputing, Networking, Storage and Analysis (SC),pages 325–335, 2014.[2] C. Bichot and P. Siarry, editors. Graph Partitioning.Wiley, 2011.[3] A. Buluc¸, H. Meyerhenke, I. Safro, P. Sanders, andC. Schulz. Recent Advances in Graph Partition-ing. In Algorithm Engineering – Selected Topics,ArXiv:1311.3144, 2014.[4] U¨. V Catalyu¨rek and C. Aykanat. Hypergraph modelfor mapping repeated sparse matrix-vector productcomputations onto multicomputers. In Proceedings ofInternational Conference on High Performance Com-puting (HiPC), 1995.[5] U¨. V. C¸atalyu¨rek and C. Aykanat. Hypergraph-partitioning-based decomposition for parallel sparse-matrix vector multiplication. IEEE Trans. ParallelDistrib. Syst., 10(7):673–693, 1999.[6] U¨. V. C¸atalyu¨rek, M. Deveci, K. Kaya, and B. Uc¸ar.UMPa: A Multi-objective, Multi-level Partitioner forCommunication Minimization. In 10th DIMACS Impl.Challenge Workshop: Graph Partitioning and GraphClustering. Georgia Institute of Technology, Atlanta,GA, February 13-14 2012.[7] D. Delling, A. V. Goldberg, I. Razenshteyn, andR. F. Werneck. Graph partitioning with naturalcuts. In 2011 IEEE International Parallel DistributedProcessing Symposium, pages 1135–1146, May 2011.[8] M. Fonseca Faraj, A. van der Grinten, H. Meyerhenke,J. L. Tra¨ff, and C. Schulz. High-Quality HierarchicalProcess Mapping. CoRR, abs/2001.07134, 2020.[9] M. R. Garey and D. S. Johnson. Computersand Intractability: A Guide to the Theory of NP-Completeness (Series of Books in the MathematicalSciences). W. H. Freeman, first edition edition, 1979.[10] R. Glantz, M. Predari, and H. Meyerhenke. Topology-induced enhancement of mappings. In Proceedings ofthe 47th International Conference on Parallel Process-ing, ICPP 2018, pages 9:1–9:10. ACM, 2018.[11] M. Hamann and B. Strasser. Graph Bisection withPareto-Optimization. In Proc. of the 18th AlgorithmEngineering and Experiments, pages 90–102. SIAM,2016.[12] T. Hatazaki. Rank reordering strategy for MPI topol-ogy creation functions. In 5th European PVM/MPIUser’s Group Meeting, volume 1497, pages 188–195,1998.[13] B. Hendrickson and T. G. Kolda. Graph PartitioningModels for Parallel Computing. Parallel Computing,26(12):1519–1534, 2000.[14] T. Hoefler and M. Snir. Generic topology mappingstrategies for large-scale parallel architectures. In Proc.25th Intl. Conf. on Supercomputing (ICS), pages 75–84,2011.[15] G. Karypis and V. Kumar. Multilevel k-way Partition-ing Scheme for Irregular Graphs. Journal on Paralleland Distributed Compututing, 48(1):96–129, 1998.[16] K. Lang and S. Rao. A Flow-Based Method for Im-proving the Expansion or Conductance of Graph Cuts.In Proceedings of 10th International Integer Program-ming and Combinatorial Optimization Conference, vol-ume 3064 of LNCS, pages 383–400. Springer, 2004.[17] J. Langguth, M. Sourouri, G. T. Lines, S. B. Baden,and X. Cai. Scalable heterogeneous cpu-gpu computa-tions for unstructured tetrahedral meshes. IEEE Mi-cro, 35(4):6–15, July 2015.[18] G. Mercier and J. Clet-Ortega. Towards an efficientprocess placement policy for MPI applications in mul-ticore environments. In European Parallel Virtual Ma-chine/Message Passing Interface Users Group Meeting,pages 104–115. Springer, 2009.[19] G. Mercier and E. Jeannot. Improving MPI appli-cations performance on multicore clusters with rankreordering. In 18th Eur. MPI Users’ Group Meeting,pages 39–49, 2011.[20] H. Meyerhenke and T. Sauerwald. Beyond good parti-tion shapes: An analysis of diffusive graph partitioning.Algorithmica, 64(3):329–361, Nov 2012.[21] L. Page, S. Brin, R. Motwani, and T. Winograd.The pagerank citation ranking: Bringing order to theweb. Technical Report 1999-66, Stanford InfoLab,November 1999. Previous number = SIDL-WP-1999-0120.[22] F. Pellegrini. Scotch Home Page.http://www.labri.fr/perso/pelegrin/scotch/.[23] P. Sadayappan, F. Erc¸al, and J. Ramanujam. Clusterpartitioning approaches to mapping parallel programsonto a hypercube. Parallel Computing, 13(1):1–16,1990.[24] P. Sanders and C. Schulz. KaHIP – Karl-sruhe High Qualtity Partitioning Homepage.http://algo2.iti.kit.edu/documents/kahip/index.html.[25] K. Schloegel, G. Karypis, and V. Kumar. A new algo-rithm for multi-objective graph partitioning. In Pro-ceedings of the 5th International Euro-Par Conferenceon Parallel Processing (Euro-Par 1999), volume 1685of LNCS, pages 322–331. Springer, 1999.[26] C. Schulz and D. Strash. Graph partitioning: Formu-lations and applications to big data. In Encyclopediaof Big Data Technologies. Springer, 2019.[27] C. Schulz and J. L. Tra¨ff. Better process mappingand sparse quadratic assignment. In 16th InternationalSymposium on Experimental Algorithms, volume 75 ofLIPIcs, pages 4:1–4:15, 2017.[28] N. Selvakkumaran and G. Karypis. Multi-objectivehypergraph partitioning algorithms for cut and max-imum subdomain degree minimization. In ICCAD-2003. International Conference on Computer AidedDesign (IEEE Cat. No.03CH37486), pages 726–733,Nov 2003.[29] C. Walshaw and M. Cross. JOSTLE: Parallel Multi-level Graph-Partitioning Software – An Overview. InMesh Partitioning Techniques and Domain Decompo-sition Techniques, pages 27–58. 2007.[30] C. Walshaw, M. Cross, M. G. Everett, S. P. Johnson,and K. McManus. Partitioning & Mapping of Un-structured Meshes to Parallel Machine Topologies. InAfonso Ferreira and Jose´ D. P. Rolim, editors, Pro-ceedings of Parallel Algorithms for Irregularly Struc-tured Problems, Second International Workshop, IR-REGULAR ’95, volume 980 of LNCS, pages 121–126.Springer, 1995.[31] H. Yu, I-H. Chung, and J. E. Moreira. Topology map-ping for Blue Gene/L supercomputer. In ACM/IEEESupercomputing, page 116, 2006.

Load-Balanced Bottleneck Objectives in Process Mapping

https://publikationen.bibliothek.kit.edu/1000117914/65913240

Load-Balanced Bottleneck Objectives in Process Mapping

Abstract

Similar works

Full text

Available Versions

KITopen