I have three heuristics H1, H2 and H3 that have been tested on a set of instances. For each triple (n,m,p) we have 100 instances. The performance criteria that have been used to compare between them are the average deviation from the best lower bound (Dev_H) and the number of times a given heuristic yields to the best solution (Best_H), as shown in the given table.
My question is: how to use p-value to see if the differences between H1 and H2 are significant and to show that H3 outperforms H1 and H2 for n=200 and H1 and H2 outperform H3 for n=10
I am not sure what n, m and p are, but to compare multiple (meta-)heuristic algorithms you can use a non-parametric test called Friedman test. You can get a good explanation from
https://juangvillegas.files.wordpress.com/2011/08/friedman-test-24062011.pdf
hope this help
Dear Surafel, thank you for your answer. n,m and p are the parameters of the instances, i.e. each instance is characterised by given values of these parameters. But why choosing a non-parametric test and not a paired t-test for example?
Non-parametric tests are used because they don't assume the normality of the data. However, if you use t-test then you are assuming the data you have has a normal distribution.
ok, so we cannot assume that the data given in the table have a normal distribution (such that we have 45 sets of instances)?
On the other hand, Friedman test does not consider the differences between the values given by the (meta-)heuristics and I need to know if these differences are significant.
The Friedman test works based on the rank of the results by the algorithms for the given benchmark problems. The difference is not taken into consideration but the rank.
Yes this is why I am asking if I can use the t-test which considers these differences.
Dear Nour
The parametric test, like t-test, are highly sensitive to outliers. One significantly different result might change the final conclusions drastically. On the other hand, non-parametric tests, Wilcoxon Test, Mann Whitney Test, Friedman Test etc., are not much effected by outliers as they deal with ranks only but not actual parametric measures. Distribution based test are useful when the complete experimental results are available for large sample sizes ( i. e; >30 ), but in many cases only final results in the form of mean, median, best results of all runs are provided. In that case only non-parametric tests could be used.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics