Robust Ranking of Equivalent Algorithms via Relative Performance

In scientific computing, it is common that one target computation can be translated into many different sequences of library calls, each identifying an algorithm. Although mathematically equivalent, those algorithms might exhibit significant differences in terms of performance. In practice, we observe that subsets of algorithms show comparable performance characteristics. For this reason, we aim to identify and separate not one algorithm, but the subset of algorithms, that are reliably faster than the rest. One of the motivations for this setup is that it makes it then possible to select an algorithm based on additional performance metrics such as those based on energy or scalability. To this end, instead of using the usual methods of quantifying the performance of an algorithm in absolute terms, we present a measurement-based approach that assigns a relative score to the algorithms in comparison to one another. The relative performance is encoded by sorting the algorithms based on pair-wise comparisons and by ranking them into equivalence (or performance) classes, so that multiple algorithms can obtain the same rank. We show that this approach, based on relative performance, leads to robust identification of the fastest algorithms, that is, reliable identification even under noisy system conditions