The experimental analysis of meta-heuristic algorithm performance is usually
based on comparing average performance metric values over a set of algorithm
instances. When algorithms getting tight in performance gains, the additional
consideration of significance of a metric improvement comes into play. However,
from this moment the comparison changes from an absolute to a relative mode.
Here the implications of this paradigm shift are investigated. Significance
relations are formally established. Based on this, a trade-off between
increasing cycle-freeness of the relation and small maximum sets can be
identified, allowing for the selection of a proper significance level and
resulting ranking of a set of algorithms. The procedure is exemplified on the
CEC'05 benchmark of real parameter single objective optimization problems. The
significance relation here is based on awarding ranking points for relative
performance gains, similar to the Borda count voting method or the Wilcoxon
signed rank test. In the particular CEC'05 case, five ranks for algorithm
performance can be clearly identified.Comment: 6 pages, 2 figures, 1 tabl