mk:isda13

Summary

Significance Relations for the Benchmarking of Meta-Heuristic Algorithms. Mario Köppen and Kei Ohnishi. In Proc. 2013 Intl. Conference on Intelligent Systems Design and Applications (ISDA 2013), pages 253-258, December 2013. (URL)

Abstract

The experimental analysis of meta-heuristic algorithm performance is usually based on comparing average per- formance metric values over a set of algorithm instances. When algorithms getting tight in performance gains, the additional consideration of significance of a metric improvement comes into play. However, from this moment the comparison changes from an absolute to a relative mode. Here the implications of this paradigm shift are investigated. Significance relations are formally established. Based on this, a trade-off between increasing cycle-freeness of the relation and small maximum sets can be identified, allowing for the selection of a proper significance level and resulting ranking of a set of algorithms. The procedure is exemplified on the CEC'05 benchmark of real parameter single objective optimization problems. The significance relation here is based on awarding ranking points for relative performance gains, similar to the Borda count voting method or the Wilcoxon signed rank test. In this particular case, five ranks for algorithm performance can be clearly identified.

Bibtex entry

@INPROCEEDINGS { mk:isda13,
    ABSTRACT = { The experimental analysis of meta-heuristic algorithm performance is usually based on comparing average per- formance metric values over a set of algorithm instances. When algorithms getting tight in performance gains, the additional consideration of significance of a metric improvement comes into play. However, from this moment the comparison changes from an absolute to a relative mode. Here the implications of this paradigm shift are investigated. Significance relations are formally established. Based on this, a trade-off between increasing cycle-freeness of the relation and small maximum sets can be identified, allowing for the selection of a proper significance level and resulting ranking of a set of algorithms. The procedure is exemplified on the CEC'05 benchmark of real parameter single objective optimization problems. The significance relation here is based on awarding ranking points for relative performance gains, similar to the Borda count voting method or the Wilcoxon signed rank test. In this particular case, five ranks for algorithm performance can be clearly identified. },
    AUTHOR = { Mario Köppen and Kei Ohnishi },
    BOOKTITLE = { Proc. 2013 Intl. Conference on Intelligent Systems Design and Applications (ISDA 2013) },
    ADDED = { 2013-11-07 11:59:45 +0000 },
    MODIFIED = { 2014-01-05 06:49:15 +0000 },
    MONTH = { December },
    PAGES = { 253-258 },
    TITLE = { Significance Relations for the Benchmarking of Meta-Heuristic Algorithms },
    URL = { http://arxiv.org/abs/1311.1338 },
    YEAR = { 2013 },
    1 = { http://arxiv.org/abs/1311.1338 },
}