A Reduction -- an accumulation over a set of values, using an associative and
commutative operator -- is a common computation in many numerical computations,
including scientific computations, machine learning, computer vision, and
financial analytics.
Contemporary polyhedral-based compilation techniques make it possible to
optimize reductions, such as prefix sums, in which each component of the
reduction's output potentially shares computation with another component in the
reduction. Therefore an optimizing compiler can identify the computation shared
between multiple components and generate code that computes the shared
computation only once.
These techniques, however, do not support reductions that -- when phrased in
the language of the polyhedral model -- span multiple dependent statements. In
such cases, existing approaches can generate incorrect code that violates the
data dependences of the original, unoptimized program.
In this work, we identify and formalize the optimization of dependent
reductions as an integer bilinear program. We present a heuristic optimization
algorithm that uses an affine sequential schedule of the program to determine
how to simplfy reductions yet still preserve the program's dependences.
We demonstrate that the algorithm provides optimal complexity for a set of
benchmark programs from the literature on probabilistic inference algorithms,
whose performance critically relies on simplifying these reductions. The
complexities for 10 of the 11 programs improve siginifcantly by factors at
least of the sizes of the input data, which are in the range of 104 to
106 for typical real application inputs. We also confirm the significance of
the improvement by showing speedups in wall-clock time that range from
1.1x to over 106x