International audienceWe show how monotone interpretations – a termination analysis technique for term rewritingsystems – can be used to assess the inherent parallelism of recursive programs manipulatinginductive data structures. As a side effect, we show how monotone interpretations specify aparallel execution order, and how our approach extends naturally affine scheduling – a powerfulanalysis used in parallelising compilers – to recursive programs. This work opens new perspectivesin automatic parallelisation

Alias, Christophe

Fuhs, Carsten

Gonnord, Laure

English

INRIA a CCSD electronic archive server

HAL Id: hal-01345914https://hal.archives-ouvertes.fr/hal-01345914Submitted on 20 Jul 2016HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Distributed under a Creative Commons Attribution| 4.0 International LicenseEstimation of Parallel Complexity with RewritingTechniquesChristophe Alias, Carsten Fuhs, Laure GonnordTo cite this version:Christophe Alias, Carsten Fuhs, Laure Gonnord. Estimation of Parallel Complexity with RewritingTechniques. Workshop on Termination, Sep 2016, Obergurgl, Austria. ￿hal-01345914￿Estimation of Parallel Complexity with Rewriting TechniquesAuthor version, accepted to WST 2016∗Christophe AliasINRIA & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, FranceCarsten FuhsBirkbeck, University of London, United KingdomLaure GonnordUniversity of Lyon & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, FranceJuly 20, 2016AbstractWe show how monotone interpretations – a termination analysis technique for termrewriting systems – can be used to assess the inherent parallelism of recursive programsmanipulating inductive data structures. As a side effect, we show how monotone interpre-tations specify a parallel execution order, and how our approach extends naturally affinescheduling – a powerful analysis used in parallelising compilers – to recursive programs. Thiswork opens new perspectives in automatic parallelisation.1 IntroductionThe motivation of this work is the automatic transformation of sequential code into parallel codewithout changing its (big-step operational) semantics, only changing the computation order isallowed. We want to find out the limits of these approaches, by characterising the “maximumlevel of parallelism” that we can find for a given sequential implementation of an algorithm.In this paper, we propose a way to estimate the parallel complexity which can be informallydefined as the complexity of the program if it were executed on an machine with unboundedparallelism.Such a result could come as by-product of the polyhedral-based automatic parallelisationtechniques for array-based programs [6]. However, for general programs with complex data flowand inductive structures, such techniques have not been explored so far.The contribution of this paper is a novel unifying way to express program dependenciesfor general programs with inductive data structures (lists, trees. . . ) as well as a way to usecomplexity bounds of term rewriting systems in order to derive an estimation of this parallelcomplexity.∗This work was partially supported by a “BQR” funding at ENS De Lyon.12 Intuitions behind the notion of “parallel complexity”for(i=0; i<=N; i++)for (j=0; j<=N; j++)// Block S{m1[i][j] = Integer.MIN_VALUE;for(k=1; k<=i; k++)m1[i][j] = max(m1[i][j],H[i-k][j] + W[k]);m2[i][j] = Integer.MIN_VALUE;for(k=1; k<=j; k++)m2[i][j] = max(m2[i][j],H[i][j-k] + W[k]);H[i][j] = max(0,H(i-1,j-1)+s(a[i],b[i]),m1[i][j],m2[i][j]);}jiFigure 1: The Smith-Waterman sequence alignment algorithm and its dependencies. Each point(i, j) represents an execution of the block S, denoted by 〈S, i, j〉.public class Tree {private int val;private Tree left;private Tree right;public int treeMax () {int leftMax = Integer.MIN_VALUE;int rightMax = Integer.MIN_VALUE;if (this.left != null) {leftMax = this.left.treeMax (); // S1}if (this.right != null) {rightMax = this.right.treeMax ();// S2}return Math.max(this.val , Math.max(leftMax , rightMax ));}}valleftrightFigure 2: Maximum element of a tree algorithm and its call tree.In this paper, we consider the derivation of a lower bound for the intrinsic parallelism of asequential imperative program P and thus an upper bound for the complexity of a fully paral-lelised version of P . Figures 1 and 2 depict the two motivating examples that will be studiedin the remainder of this paper. Figure 1 is a loop kernel computing the Smith-Waterman op-timal sequence alignment algorithm1. Figure 2 is a simple recursive function to compute themaximum element of a binary tree.Usually, the parallelism is found by analysing the data dependencies between the operationsexecuted by the program. There is a data dependency from operation o1 to operation o2 inthe execution of a program if o1 is executed before o2 and both operations access the same1See https://en.wikipedia.org/wiki/Smith-Waterman_algorithm. We consider two sequences of the samelength N .2memory location. In particular, we have a flow dependency when the result of o1 is required byo2. Non-flow dependencies (write after write, write after read) can be removed by playing onthe memory allocation [3] and can thus be ignored. Thus, we consider only flow dependenciesreferred as dependencies in the remainder of this paper. If there is no dependency between twooperations, then they may be executed in parallel. While the dependency relation is in generalundecidable, in practice we can use decidable over-approximations such that a statement thattwo operations are independent is always correct.In Figure 1, each point represents an execution of the block S computing H[i][j] for a giveni and j in J0, NK. Such an operation is written 〈S, i, j〉. The arrows represents the dependenciestowards a given 〈S, i, j〉. For instance the diagonal arrow means that H[i][j] requires the valueof H[i-1][j-1] to be computed.In Figure 2, we depicted the execution trace of the recursive program on a tree. Here, thedependencies between computations resemble the recursive calls: the dependency graph (in thesense used for imperative programs) is the call tree.All reorderings of computations respecting the dependencies are valid orderings (that wewill name schedule in the following). In both cases, the dependencies we draw show a certainpotential parallelism. Indeed, each pair of computations that do not transitively depend on eachother can be executed in parallel; moreover, even a machine with infinite memory and infinitelyparallel cannot do the computation in an amount of time which is shorter than the longest pathin the dependency graph. The length of this longest path, referred to as parallel complexity, isthus an estimation of the potential parallelism of the program (its execution time with suitablyreordered instructions on an idealised parallel machine).Our goal in this paper is, given a sequential program P and an over-approximation of itsdependency relation, to find bounds on the parallel complexity of P and the over-approximationof its dependency relation. Via the representation of the dependencies via (possibly constrained)rewrite rules, we can apply existing techniques to find such bounds for (constrained) rewritesystems (e.g., [8, 4]).3 Computing the parallel complexity of programsIn this section we explain on the two running examples the relationship between termination andscheduling (this relationship was already explored a bit in [1]), and polynomial interpretations(more generally, proofs of complexity bounds for term rewriting) and parallel complexity. As thelong-term goal is to devise compiler optimisations by automatic parallelisation for imperativeprograms, we consider programs with data structures such as arrays, structs, records andeven classes. The main idea is that all these constructions can be classically representedas terms via the notion of position, and the dependencies as term rewriting rules. Contrarilyto other approaches for proving termination of programs by an abstraction to rewriting (e.g.,[11, 5]), we only encode the dependencies (and forget about the control flow).3.1 First example: Smith-Waterman algorithmFor the Smith-Waterman program of Figure 1, we can derive (with polyhedral array dataflowanalysis, as in [3]), forgetting about the local computations of W scores, the following depen-dencies as constrained rewrite rules:dep(i, j)→ dep(i− 1, j − 1) : 0 ≤ i ≤ n, 0 ≤ j ≤ ndep(i, j)→ dep(i− k, j) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ k ≤ idep(i, j)→ dep(i, j − `) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ ` ≤ j3To get an estimation of the parallel complexity, following the inspiration of previous work ontermination proofs for complexity analysis (e.g., [8, 1, 4]), we first try to generate a polynomialinterpretation [10] to map function symbols and terms to polynomials:Pol(dep(x1, x2)) = x1 + x2Essentially, this says that iteration 〈S, x1, x2〉 can be executed at time stamp x1 + x2. As0 ≤ x1, x2 ≤ N , the maximal timestamp is 2N = O(N). Not only does this mean that theprogram may be parallelised, but it provides an actual reordering of the computation, along theparallel wavefront x1 + x2.This linear interpretation (or ranking function) provides us with a bound in O(N1) for theparallel complexity of the program. Thus, the degree of sequentiality is 1. As the overall runtimeof the program is in O(N2), this gives us a degree of parallelism of 2− 1 = 1 [9].Let us point out that we would have obtained the same results using affine schedulingtechniques from the polyhedral model [6]. The interesting fact here is that our apparatus isnot restricted to regular programs (for loops, arrays) as the polyhedral model. Also, currentcomplexity analysis tools like KoAT [4] are able to compute similar results within a reasonableamount of time. The next section show how our technique applies to a recursive program oninductive data structures.3.2 Second example: computing the maximum element in a tree23 47Figure 3: The tree t.An object of class Tree is represented by the term Tree(val , left , right),for some terms val , left , right that represent its attributes [11]. Forinstance, the Java object defined in Figure 3 corresponds to the followingterm:t = Tree(2,Tree(3, null, null),Tree(4,Tree(7, null, null), null))Recall that to address entries of an n-dimensional array, we use vec-tors (i1, . . . , in) ∈ Nn as indices or “positions”. Then for an array A,we say that A[i1]. . .[in] is the “entry” at the array’s position (i1, . . . , in). Similarly to arrayentries, we would also like to address particular “entries” of a term, i.e., its subterms.For more general terms, we can use a similar notion: positions. For our tree t, we havePos(t) = {ε, 1, 2, 3, 21, 22, 23, . . .}, giving an absolute way to access each element or subarray.For instance, 21 denotes the first element of the left child of the tree (i.e., the number 3).Now let us consider the program in Figure 2. This function computes recursively the max-imum value of an integer binary tree. Clearly, the computation of the maximum of a treedepends on the computation of each of its children. However, the computation of the max ofeach child is independent from the other. There is thus potential parallelism.Here we observe structural dependencies from the accesses to the children of the currentnode. Like in the previous example, from the program we generate the following “dependency-rewrite rules” (note that similar to the dependency pair setting for termination proving [2], itsuffices to consider “top-rewriting” with rewrite steps only at the root of the term):dep(Tree(val , left , right))→ dep(left) (S1)dep(Tree(val , left , right))→ dep(right) (S2)We can use the following polynomial interpretation (analogous to the notion of a rankingfunction) to prove termination and also a complexity bound:Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = x2 + x3 + 14Or, using interpretations also involving the maximum function [7]:Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = max(x2, x3) + 1Thus we interpret a tree as the maximum of its two children plus one to prove termination ofthe dependency relation of the original sequential program, essentially mapping a tree to itsheight. This means that the parallel complexity (i.e., the length of a chain in the dependencyrelation) of the program is bounded by the height of the input data structure.Indeed, the two recursive calls could be executed in parallel, with a runtime bounded by theheight of the tree on a machine with unbounded parallelism. The interpretation Pol(Tree(x1, x2, x3)) =max(x2, x3) + 1 induces a wavefront for the parallel execution along the levels of the tree, i.e.,the nodes at the same depth in the tree.In contrast, the overall runtime of the original sequential program is bounded only by thesize of the input tree, which may be exponentially larger than its depth.4 Conclusion and Future WorkIn this paper we showed some preliminary results on the (automatable) computation of theparallel complexity of programs with inductive data structures.In the future, we will investigate a complete formalisation of these preliminary results, andtest for applicability in more challenging programs like heapsort and prefixsum. As we said in theintroduction, expressing the parallel complexity is the first step toward more ambitious use ofrewriting techniques for program optimisation. The work in progress includes the computationof parallel schedules from the rewriting rules or their interpretation, and then parallel codegeneration from the obtained schedules.References[1] Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. Multi-dimensional rankings, program termination,and complexity bounds of flowchart programs. In Proc. SAS ’10, pages 117–133, 2010.[2] Thomas Arts and Jürgen Giesl. Termination of term rewriting using dependency pairs. Theoretical Computer Science,236:133–178, 2000.[3] Denis Barthou, Albert Cohen, and Jean-François Collard. Maximal static expansion. International Journal of ParallelProgramming, 28(3):213–243, June 2000.[4] Marc Brockschmidt, Fabian Emmes, Stephan Falke, Carsten Fuhs, and Jürgen Giesl. Analyzing runtime and sizecomplexity of integer programs. ACM TOPLAS, 2016. To appear.[5] Stephan Falke, Deepak Kapur, and Carsten Sinz. Termination analysis of C programs using compiler intermediatelanguages. In Proc. RTA ’11, pages 41–50, 2011.[6] Paul Feautrier. Some efficient solutions to the affine scheduling problem, I, one-dimensional time. InternationalJournal of Parallel Programming, 21(5):313–348, October 1992.[7] Carsten Fuhs, Jürgen Giesl, Aart Middeldorp, Peter Schneider-Kamp, René Thiemann, and Harald Zankl. Maximaltermination. In Proc. RTA ’08, pages 110–125, 2008.[8] Nao Hirokawa and Georg Moser. Automated complexity analysis based on the dependency pair method. In Proc.IJCAR ’08, pages 364–379, 2008.[9] Richard M. Karp, Raymond E. Miller, and Shmuel Winograd. The organization of computations for uniform recurrenceequations. Journal of the ACM, 14(3):563–590, 1967.[10] Dallas S. Lankford. On proving term rewriting systems are Noetherian. Technical Report MTP-3, Louisiana TechnicalUniversity, Ruston, LA, USA, 1979.[11] Carsten Otto, Marc Brockschmidt, Christian von Essen, and Jürgen Giesl. Automated ter-mination analysis of Java Bytecode by term rewriting. In RTA ’10, pages 259–276, 2010.5

Estimation of Parallel Complexity with Rewriting Techniques

HAL-ENS-LYON

Estimation of Parallel Complexity with RewritingTechniquesChristophe Alias, Carsten Fuhs, Laure GonnordTo cite this version:Christophe Alias, Carsten Fuhs, Laure Gonnord. Estimation of Parallel Complexity withRewriting Techniques. Workshop on Termination, Sep 2016, Obergurgl, Austria. Workshop onTermination, 2016, Workshop on Termination. <http://cl-informatik.uibk.ac.at/events/wst-2016/>. <hal-01345914>HAL Id: hal-01345914https://hal.archives-ouvertes.fr/hal-01345914Submitted on 20 Jul 2016HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Distributed under a Creative Commons Attribution 4.0 International LicenseEstimation of Parallel Complexity with Rewriting TechniquesAuthor version, accepted to WST 2016∗Christophe AliasINRIA & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, FranceCarsten FuhsBirkbeck, University of London, United KingdomLaure GonnordUniversity of Lyon & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, FranceJuly 20, 2016AbstractWe show how monotone interpretations – a termination analysis technique for termrewriting systems – can be used to assess the inherent parallelism of recursive programsmanipulating inductive data structures. As a side effect, we show how monotone interpre-tations specify a parallel execution order, and how our approach extends naturally affinescheduling – a powerful analysis used in parallelising compilers – to recursive programs. Thiswork opens new perspectives in automatic parallelisation.1 IntroductionThe motivation of this work is the automatic transformation of sequential code into parallel codewithout changing its (big-step operational) semantics, only changing the computation order isallowed. We want to find out the limits of these approaches, by characterising the “maximumlevel of parallelism” that we can find for a given sequential implementation of an algorithm.In this paper, we propose a way to estimate the parallel complexity which can be informallydefined as the complexity of the program if it were executed on an machine with unboundedparallelism.Such a result could come as by-product of the polyhedral-based automatic parallelisationtechniques for array-based programs [6]. However, for general programs with complex data flowand inductive structures, such techniques have not been explored so far.The contribution of this paper is a novel unifying way to express program dependenciesfor general programs with inductive data structures (lists, trees. . . ) as well as a way to usecomplexity bounds of term rewriting systems in order to derive an estimation of this parallelcomplexity.∗This work was partially supported by a “BQR” funding at ENS De Lyon.12 Intuitions behind the notion of “parallel complexity”for(i=0; i<=N; i++)for (j=0; j<=N; j++)// Block S{m1[i][j] = Integer.MIN_VALUE;for(k=1; k<=i; k++)m1[i][j] = max(m1[i][j],H[i-k][j] + W[k]);m2[i][j] = Integer.MIN_VALUE;for(k=1; k<=j; k++)m2[i][j] = max(m2[i][j],H[i][j-k] + W[k]);H[i][j] = max(0,H(i-1,j-1)+s(a[i],b[i]),m1[i][j],m2[i][j]);}jiFigure 1: The Smith-Waterman sequence alignment algorithm and its dependencies. Each point(i, j) represents an execution of the block S, denoted by 〈S, i, j〉.public class Tree {private int val;private Tree left;private Tree right;public int treeMax () {int leftMax = Integer.MIN_VALUE;int rightMax = Integer.MIN_VALUE;if (this.left != null) {leftMax = this.left.treeMax (); // S1}if (this.right != null) {rightMax = this.right.treeMax ();// S2}return Math.max(this.val , Math.max(leftMax , rightMax ));}}valleftrightFigure 2: Maximum element of a tree algorithm and its call tree.In this paper, we consider the derivation of a lower bound for the intrinsic parallelism of asequential imperative program P and thus an upper bound for the complexity of a fully paral-lelised version of P . Figures 1 and 2 depict the two motivating examples that will be studiedin the remainder of this paper. Figure 1 is a loop kernel computing the Smith-Waterman op-timal sequence alignment algorithm1. Figure 2 is a simple recursive function to compute themaximum element of a binary tree.Usually, the parallelism is found by analysing the data dependencies between the operationsexecuted by the program. There is a data dependency from operation o1 to operation o2 inthe execution of a program if o1 is executed before o2 and both operations access the same1See https://en.wikipedia.org/wiki/Smith-Waterman_algorithm. We consider two sequences of the samelength N .2memory location. In particular, we have a flow dependency when the result of o1 is required byo2. Non-flow dependencies (write after write, write after read) can be removed by playing onthe memory allocation [3] and can thus be ignored. Thus, we consider only flow dependenciesreferred as dependencies in the remainder of this paper. If there is no dependency between twooperations, then they may be executed in parallel. While the dependency relation is in generalundecidable, in practice we can use decidable over-approximations such that a statement thattwo operations are independent is always correct.In Figure 1, each point represents an execution of the block S computing H[i][j] for a giveni and j in J0, NK. Such an operation is written 〈S, i, j〉. The arrows represents the dependenciestowards a given 〈S, i, j〉. For instance the diagonal arrow means that H[i][j] requires the valueof H[i-1][j-1] to be computed.In Figure 2, we depicted the execution trace of the recursive program on a tree. Here, thedependencies between computations resemble the recursive calls: the dependency graph (in thesense used for imperative programs) is the call tree.All reorderings of computations respecting the dependencies are valid orderings (that wewill name schedule in the following). In both cases, the dependencies we draw show a certainpotential parallelism. Indeed, each pair of computations that do not transitively depend on eachother can be executed in parallel; moreover, even a machine with infinite memory and infinitelyparallel cannot do the computation in an amount of time which is shorter than the longest pathin the dependency graph. The length of this longest path, referred to as parallel complexity, isthus an estimation of the potential parallelism of the program (its execution time with suitablyreordered instructions on an idealised parallel machine).Our goal in this paper is, given a sequential program P and an over-approximation of itsdependency relation, to find bounds on the parallel complexity of P and the over-approximationof its dependency relation. Via the representation of the dependencies via (possibly constrained)rewrite rules, we can apply existing techniques to find such bounds for (constrained) rewritesystems (e.g., [8, 4]).3 Computing the parallel complexity of programsIn this section we explain on the two running examples the relationship between termination andscheduling (this relationship was already explored a bit in [1]), and polynomial interpretations(more generally, proofs of complexity bounds for term rewriting) and parallel complexity. As thelong-term goal is to devise compiler optimisations by automatic parallelisation for imperativeprograms, we consider programs with data structures such as arrays, structs, records andeven classes. The main idea is that all these constructions can be classically representedas terms via the notion of position, and the dependencies as term rewriting rules. Contrarilyto other approaches for proving termination of programs by an abstraction to rewriting (e.g.,[11, 5]), we only encode the dependencies (and forget about the control flow).3.1 First example: Smith-Waterman algorithmFor the Smith-Waterman program of Figure 1, we can derive (with polyhedral array dataflowanalysis, as in [3]), forgetting about the local computations of W scores, the following depen-dencies as constrained rewrite rules:dep(i, j)→ dep(i− 1, j − 1) : 0 ≤ i ≤ n, 0 ≤ j ≤ ndep(i, j)→ dep(i− k, j) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ k ≤ idep(i, j)→ dep(i, j − `) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ ` ≤ j3To get an estimation of the parallel complexity, following the inspiration of previous work ontermination proofs for complexity analysis (e.g., [8, 1, 4]), we first try to generate a polynomialinterpretation [10] to map function symbols and terms to polynomials:Pol(dep(x1, x2)) = x1 + x2Essentially, this says that iteration 〈S, x1, x2〉 can be executed at time stamp x1 + x2. As0 ≤ x1, x2 ≤ N , the maximal timestamp is 2N = O(N). Not only does this mean that theprogram may be parallelised, but it provides an actual reordering of the computation, along theparallel wavefront x1 + x2.This linear interpretation (or ranking function) provides us with a bound in O(N1) for theparallel complexity of the program. Thus, the degree of sequentiality is 1. As the overall runtimeof the program is in O(N2), this gives us a degree of parallelism of 2− 1 = 1 [9].Let us point out that we would have obtained the same results using affine schedulingtechniques from the polyhedral model [6]. The interesting fact here is that our apparatus isnot restricted to regular programs (for loops, arrays) as the polyhedral model. Also, currentcomplexity analysis tools like KoAT [4] are able to compute similar results within a reasonableamount of time. The next section show how our technique applies to a recursive program oninductive data structures.3.2 Second example: computing the maximum element in a tree23 47Figure 3: The tree t.An object of class Tree is represented by the term Tree(val , left , right),for some terms val , left , right that represent its attributes [11]. Forinstance, the Java object defined in Figure 3 corresponds to the followingterm:t = Tree(2,Tree(3, null, null),Tree(4,Tree(7, null, null), null))Recall that to address entries of an n-dimensional array, we use vec-tors (i1, . . . , in) ∈ Nn as indices or “positions”. Then for an array A,we say that A[i1]. . .[in] is the “entry” at the array’s position (i1, . . . , in). Similarly to arrayentries, we would also like to address particular “entries” of a term, i.e., its subterms.For more general terms, we can use a similar notion: positions. For our tree t, we havePos(t) = {ε, 1, 2, 3, 21, 22, 23, . . .}, giving an absolute way to access each element or subarray.For instance, 21 denotes the first element of the left child of the tree (i.e., the number 3).Now let us consider the program in Figure 2. This function computes recursively the max-imum value of an integer binary tree. Clearly, the computation of the maximum of a treedepends on the computation of each of its children. However, the computation of the max ofeach child is independent from the other. There is thus potential parallelism.Here we observe structural dependencies from the accesses to the children of the currentnode. Like in the previous example, from the program we generate the following “dependency-rewrite rules” (note that similar to the dependency pair setting for termination proving [2], itsuffices to consider “top-rewriting” with rewrite steps only at the root of the term):dep(Tree(val , left , right))→ dep(left) (S1)dep(Tree(val , left , right))→ dep(right) (S2)We can use the following polynomial interpretation (analogous to the notion of a rankingfunction) to prove termination and also a complexity bound:Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = x2 + x3 + 14Or, using interpretations also involving the maximum function [7]:Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = max(x2, x3) + 1Thus we interpret a tree as the maximum of its two children plus one to prove termination ofthe dependency relation of the original sequential program, essentially mapping a tree to itsheight. This means that the parallel complexity (i.e., the length of a chain in the dependencyrelation) of the program is bounded by the height of the input data structure.Indeed, the two recursive calls could be executed in parallel, with a runtime bounded by theheight of the tree on a machine with unbounded parallelism. The interpretation Pol(Tree(x1, x2, x3)) =max(x2, x3) + 1 induces a wavefront for the parallel execution along the levels of the tree, i.e.,the nodes at the same depth in the tree.In contrast, the overall runtime of the original sequential program is bounded only by thesize of the input tree, which may be exponentially larger than its depth.4 Conclusion and Future WorkIn this paper we showed some preliminary results on the (automatable) computation of theparallel complexity of programs with inductive data structures.In the future, we will investigate a complete formalisation of these preliminary results, andtest for applicability in more challenging programs like heapsort and prefixsum. As we said in theintroduction, expressing the parallel complexity is the first step toward more ambitious use ofrewriting techniques for program optimisation. The work in progress includes the computationof parallel schedules from the rewriting rules or their interpretation, and then parallel codegeneration from the obtained schedules.References[1] Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. Multi-dimensional rankings, program termination,and complexity bounds of flowchart programs. In Proc. SAS ’10, pages 117–133, 2010.[2] Thomas Arts and Ju¨rgen Giesl. Termination of term rewriting using dependency pairs. Theoretical Computer Science,236:133–178, 2000.[3] Denis Barthou, Albert Cohen, and Jean-Franc¸ois Collard. Maximal static expansion. International Journal of ParallelProgramming, 28(3):213–243, June 2000.[4] Marc Brockschmidt, Fabian Emmes, Stephan Falke, Carsten Fuhs, and Ju¨rgen Giesl. Analyzing runtime and sizecomplexity of integer programs. ACM TOPLAS, 2016. To appear.[5] Stephan Falke, Deepak Kapur, and Carsten Sinz. Termination analysis of C programs using compiler intermediatelanguages. In Proc. RTA ’11, pages 41–50, 2011.[6] Paul Feautrier. Some efficient solutions to the affine scheduling problem, I, one-dimensional time. InternationalJournal of Parallel Programming, 21(5):313–348, October 1992.[7] Carsten Fuhs, Ju¨rgen Giesl, Aart Middeldorp, Peter Schneider-Kamp, Rene´ Thiemann, and Harald Zankl. Maximaltermination. In Proc. RTA ’08, pages 110–125, 2008.[8] Nao Hirokawa and Georg Moser. Automated complexity analysis based on the dependency pair method. In Proc.IJCAR ’08, pages 364–379, 2008.[9] Richard M. Karp, Raymond E. Miller, and Shmuel Winograd. The organization of computations for uniform recurrenceequations. Journal of the ACM, 14(3):563–590, 1967.[10] Dallas S. Lankford. On proving term rewriting systems are Noetherian. Technical Report MTP-3, Louisiana TechnicalUniversity, Ruston, LA, USA, 1979.[11] Carsten Otto, Marc Brockschmidt, Christian von Essen, and Ju¨rgen Giesl. Automated ter-mination analysis of Java Bytecode by term rewriting. In RTA ’10, pages 259–276, 2010.5

Hal-Diderot

Archive Ouverte en Sciences de l'Information et de la Communication

HAL Id: hal-01345914
https://hal.archives-ouvertes.fr/hal-01345914
Submitted on 20 Jul 2016
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Distributed under a Creative Commons Attribution| 4.0 International License
Estimation of Parallel Complexity with Rewriting
Techniques
Christophe Alias, Carsten Fuhs, Laure Gonnord
To cite this version:
Christophe Alias, Carsten Fuhs, Laure Gonnord. Estimation of Parallel Complexity with Rewriting
Techniques. Workshop on Termination, Sep 2016, Obergurgl, Austria. ￿hal-01345914￿
Estimation of Parallel Complexity with Rewriting Techniques
Author version, accepted to WST 2016
∗
Christophe Alias
INRIA & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, France
Carsten Fuhs
Birkbeck, University of London, United Kingdom
Laure Gonnord
University of Lyon & LIP (UMR CNRS/ENS Lyon/UCB Lyon1/INRIA), Lyon, France
July 20, 2016
Abstract
We show how monotone interpretations – a termination analysis technique for term
rewriting systems – can be used to assess the inherent parallelism of recursive programs
manipulating inductive data structures. As a side effect, we show how monotone interpre-
tations specify a parallel execution order, and how our approach extends naturally affine
scheduling – a powerful analysis used in parallelising compilers – to recursive programs. This
work opens new perspectives in automatic parallelisation.
1 Introduction
The motivation of this work is the automatic transformation of sequential code into parallel code
without changing its (big-step operational) semantics, only changing the computation order is
allowed. We want to find out the limits of these approaches, by characterising the “maximum
level of parallelism” that we can find for a given sequential implementation of an algorithm.
In this paper, we propose a way to estimate the parallel complexity which can be informally
defined as the complexity of the program if it were executed on an machine with unbounded
parallelism.
Such a result could come as by-product of the polyhedral-based automatic parallelisation
techniques for array-based programs [6]. However, for general programs with complex data flow
and inductive structures, such techniques have not been explored so far.
The contribution of this paper is a novel unifying way to express program dependencies
for general programs with inductive data structures (lists, trees. . . ) as well as a way to use
complexity bounds of term rewriting systems in order to derive an estimation of this parallel
complexity.
∗This work was partially supported by a “BQR” funding at ENS De Lyon.
1
2 Intuitions behind the notion of “parallel complexity”
for(i=0; i<=N; i++)
for (j=0; j<=N; j++)
// Block S
{
m1[i][j] = Integer.MIN_VALUE;
for(k=1; k<=i; k++)
m1[i][j] = max(m1[i][j],H[i-k][j] + W[k]);
m2[i][j] = Integer.MIN_VALUE;
for(k=1; k<=j; k++)
m2[i][j] = max(m2[i][j],H[i][j-k] + W[k]);
H[i][j] = max(0,H(i-1,j-1)+s(a[i],b[i]),
m1[i][j],m2[i][j]);
}
j
i
Figure 1: The Smith-Waterman sequence alignment algorithm and its dependencies. Each point
(i, j) represents an execution of the block S, denoted by 〈S, i, j〉.
public class Tree {
private int val;
private Tree left;
private Tree right;
public int treeMax () {
int leftMax = Integer.MIN_VALUE;
int rightMax = Integer.MIN_VALUE;
if (this.left != null) {
leftMax = this.left.treeMax (); // S1
}
if (this.right != null) {
rightMax = this.right.treeMax ();// S2
}
return Math.max(this.val , Math.max(leftMax , rightMax ));
}
}
val
le
ft
right
Figure 2: Maximum element of a tree algorithm and its call tree.
In this paper, we consider the derivation of a lower bound for the intrinsic parallelism of a
sequential imperative program P and thus an upper bound for the complexity of a fully paral-
lelised version of P . Figures 1 and 2 depict the two motivating examples that will be studied
in the remainder of this paper. Figure 1 is a loop kernel computing the Smith-Waterman op-
timal sequence alignment algorithm1. Figure 2 is a simple recursive function to compute the
maximum element of a binary tree.
Usually, the parallelism is found by analysing the data dependencies between the operations
executed by the program. There is a data dependency from operation o1 to operation o2 in
the execution of a program if o1 is executed before o2 and both operations access the same
1See https://en.wikipedia.org/wiki/Smith-Waterman_algorithm. We consider two sequences of the same
length N .
2
memory location. In particular, we have a flow dependency when the result of o1 is required by
o2. Non-flow dependencies (write after write, write after read) can be removed by playing on
the memory allocation [3] and can thus be ignored. Thus, we consider only flow dependencies
referred as dependencies in the remainder of this paper. If there is no dependency between two
operations, then they may be executed in parallel. While the dependency relation is in general
undecidable, in practice we can use decidable over-approximations such that a statement that
two operations are independent is always correct.
In Figure 1, each point represents an execution of the block S computing H[i][j] for a given
i and j in J0, NK. Such an operation is written 〈S, i, j〉. The arrows represents the dependencies
towards a given 〈S, i, j〉. For instance the diagonal arrow means that H[i][j] requires the value
of H[i-1][j-1] to be computed.
In Figure 2, we depicted the execution trace of the recursive program on a tree. Here, the
dependencies between computations resemble the recursive calls: the dependency graph (in the
sense used for imperative programs) is the call tree.
All reorderings of computations respecting the dependencies are valid orderings (that we
will name schedule in the following). In both cases, the dependencies we draw show a certain
potential parallelism. Indeed, each pair of computations that do not transitively depend on each
other can be executed in parallel; moreover, even a machine with infinite memory and infinitely
parallel cannot do the computation in an amount of time which is shorter than the longest path
in the dependency graph. The length of this longest path, referred to as parallel complexity, is
thus an estimation of the potential parallelism of the program (its execution time with suitably
reordered instructions on an idealised parallel machine).
Our goal in this paper is, given a sequential program P and an over-approximation of its
dependency relation, to find bounds on the parallel complexity of P and the over-approximation
of its dependency relation. Via the representation of the dependencies via (possibly constrained)
rewrite rules, we can apply existing techniques to find such bounds for (constrained) rewrite
systems (e.g., [8, 4]).
3 Computing the parallel complexity of programs
In this section we explain on the two running examples the relationship between termination and
scheduling (this relationship was already explored a bit in [1]), and polynomial interpretations
(more generally, proofs of complexity bounds for term rewriting) and parallel complexity. As the
long-term goal is to devise compiler optimisations by automatic parallelisation for imperative
programs, we consider programs with data structures such as arrays, structs, records and
even classes. The main idea is that all these constructions can be classically represented
as terms via the notion of position, and the dependencies as term rewriting rules. Contrarily
to other approaches for proving termination of programs by an abstraction to rewriting (e.g.,
[11, 5]), we only encode the dependencies (and forget about the control flow).
3.1 First example: Smith-Waterman algorithm
For the Smith-Waterman program of Figure 1, we can derive (with polyhedral array dataflow
analysis, as in [3]), forgetting about the local computations of W scores, the following depen-
dencies as constrained rewrite rules:
dep(i, j)→ dep(i− 1, j − 1) : 0 ≤ i ≤ n, 0 ≤ j ≤ n
dep(i, j)→ dep(i− k, j) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ k ≤ i
dep(i, j)→ dep(i, j − `) : 0 ≤ i ≤ n, 0 ≤ j ≤ n, 1 ≤ ` ≤ j
3
To get an estimation of the parallel complexity, following the inspiration of previous work on
termination proofs for complexity analysis (e.g., [8, 1, 4]), we first try to generate a polynomial
interpretation [10] to map function symbols and terms to polynomials:
Pol(dep(x1, x2)) = x1 + x2
Essentially, this says that iteration 〈S, x1, x2〉 can be executed at time stamp x1 + x2. As
0 ≤ x1, x2 ≤ N , the maximal timestamp is 2N = O(N). Not only does this mean that the
program may be parallelised, but it provides an actual reordering of the computation, along the
parallel wavefront x1 + x2.
This linear interpretation (or ranking function) provides us with a bound in O(N1) for the
parallel complexity of the program. Thus, the degree of sequentiality is 1. As the overall runtime
of the program is in O(N2), this gives us a degree of parallelism of 2− 1 = 1 [9].
Let us point out that we would have obtained the same results using affine scheduling
techniques from the polyhedral model [6]. The interesting fact here is that our apparatus is
not restricted to regular programs (for loops, arrays) as the polyhedral model. Also, current
complexity analysis tools like KoAT [4] are able to compute similar results within a reasonable
amount of time. The next section show how our technique applies to a recursive program on
inductive data structures.
3.2 Second example: computing the maximum element in a tree
2
3 4
7
Figure 3: The tree t.
An object of class Tree is represented by the term Tree(val , left , right),
for some terms val , left , right that represent its attributes [11]. For
instance, the Java object defined in Figure 3 corresponds to the following
term:
t = Tree(2,Tree(3, null, null),Tree(4,Tree(7, null, null), null))
Recall that to address entries of an n-dimensional array, we use vec-
tors (i1, . . . , in) ∈ Nn as indices or “positions”. Then for an array A,
we say that A[i1]. . .[in] is the “entry” at the array’s position (i1, . . . , in). Similarly to array
entries, we would also like to address particular “entries” of a term, i.e., its subterms.
For more general terms, we can use a similar notion: positions. For our tree t, we have
Pos(t) = {ε, 1, 2, 3, 21, 22, 23, . . .}, giving an absolute way to access each element or subarray.
For instance, 21 denotes the first element of the left child of the tree (i.e., the number 3).
Now let us consider the program in Figure 2. This function computes recursively the max-
imum value of an integer binary tree. Clearly, the computation of the maximum of a tree
depends on the computation of each of its children. However, the computation of the max of
each child is independent from the other. There is thus potential parallelism.
Here we observe structural dependencies from the accesses to the children of the current
node. Like in the previous example, from the program we generate the following “dependency-
rewrite rules” (note that similar to the dependency pair setting for termination proving [2], it
suffices to consider “top-rewriting” with rewrite steps only at the root of the term):
dep(Tree(val , left , right))→ dep(left) (S1)
dep(Tree(val , left , right))→ dep(right) (S2)
We can use the following polynomial interpretation (analogous to the notion of a ranking
function) to prove termination and also a complexity bound:
Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = x2 + x3 + 1
4
Or, using interpretations also involving the maximum function [7]:
Pol(dep(x1)) = x1 and Pol(Tree(x1, x2, x3)) = max(x2, x3) + 1
Thus we interpret a tree as the maximum of its two children plus one to prove termination of
the dependency relation of the original sequential program, essentially mapping a tree to its
height. This means that the parallel complexity (i.e., the length of a chain in the dependency
relation) of the program is bounded by the height of the input data structure.
Indeed, the two recursive calls could be executed in parallel, with a runtime bounded by the
height of the tree on a machine with unbounded parallelism. The interpretation Pol(Tree(x1, x2, x3)) =
max(x2, x3) + 1 induces a wavefront for the parallel execution along the levels of the tree, i.e.,
the nodes at the same depth in the tree.
In contrast, the overall runtime of the original sequential program is bounded only by the
size of the input tree, which may be exponentially larger than its depth.
4 Conclusion and Future Work
In this paper we showed some preliminary results on the (automatable) computation of the
parallel complexity of programs with inductive data structures.
In the future, we will investigate a complete formalisation of these preliminary results, and
test for applicability in more challenging programs like heapsort and prefixsum. As we said in the
introduction, expressing the parallel complexity is the first step toward more ambitious use of
rewriting techniques for program optimisation. The work in progress includes the computation
of parallel schedules from the rewriting rules or their interpretation, and then parallel code
generation from the obtained schedules.
References
[1] Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. Multi-dimensional rankings, program termination,
and complexity bounds of flowchart programs. In Proc. SAS ’10, pages 117–133, 2010.
[2] Thomas Arts and Ju¨rgen Giesl. Termination of term rewriting using dependency pairs. Theoretical Computer Science,
236:133–178, 2000.
[3] Denis Barthou, Albert Cohen, and Jean-Franc¸ois Collard. Maximal static expansion. International Journal of Parallel
Programming, 28(3):213–243, June 2000.
[4] Marc Brockschmidt, Fabian Emmes, Stephan Falke, Carsten Fuhs, and Ju¨rgen Giesl. Analyzing runtime and size
complexity of integer programs. ACM TOPLAS, 2016. To appear.
[5] Stephan Falke, Deepak Kapur, and Carsten Sinz. Termination analysis of C programs using compiler intermediate
languages. In Proc. RTA ’11, pages 41–50, 2011.
[6] Paul Feautrier. Some efficient solutions to the affine scheduling problem, I, one-dimensional time. International
Journal of Parallel Programming, 21(5):313–348, October 1992.
[7] Carsten Fuhs, Ju¨rgen Giesl, Aart Middeldorp, Peter Schneider-Kamp, Rene´ Thiemann, and Harald Zankl. Maximal
termination. In Proc. RTA ’08, pages 110–125, 2008.
[8] Nao Hirokawa and Georg Moser. Automated complexity analysis based on the dependency pair method. In Proc.
IJCAR ’08, pages 364–379, 2008.
[9] Richard M. Karp, Raymond E. Miller, and Shmuel Winograd. The organization of computations for uniform recurrence
equations. Journal of the ACM, 14(3):563–590, 1967.
[10] Dallas S. Lankford. On proving term rewriting systems are Noetherian. Technical Report MTP-3, Louisiana Technical
University, Ruston, LA, USA, 1979.
[11] Carsten Otto, Marc Brockschmidt, Christian von Essen, and Ju¨rgen Giesl. Automated ter-
mination analysis of Java Bytecode by term rewriting. In RTA ’10, pages 259–276, 2010.
5


https://hal.archives-ouvertes.fr/hal-01345914/file/rewriting_parallelism_articleHAL.pdf

Estimation of Parallel Complexity with Rewriting Techniques

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

HAL-ENS-LYON

Hal-Diderot

Archive Ouverte en Sciences de l'Information et de la Communication