You can use ANOVA (Analysis of Variance) to compare more than two means.

You can try Friedman test. If you are doing the simulations in Matlab, then store the best values in each run. As for one algorithm, you will get 30 best values for 30 runs. For 4 algorithms you will get 120 best values, and then store the best values in a matrix and call the Matlab toolbox by writing "[p,tbl,stats] = friedman(Best_Sol)". If you need any more suggestions, feel free to msg me.

You can compare among four algorithms based on the evaluation performance metrics like Mean Squared Error (MSE), Mean Absolute Error (MAE), Correlation coefficient (R2), Mean absolute percent error (MAPE), Mean absolute deviation (MAD) etc.

You can compare algorithms based on runtime complexity, or you can use anova test used for statistical validity of the solutions...

Hello,

Anova should not be used, it supposes the executons follows a normal distribution, and it is not.

Using stochastic algorithms may also lead to bad convergence for one (or more) execution(s), so working with medians should be more appropriate than means.

To compare algorithms to see if there is a significant difference between two, you could use a a p-value test such as Wilcoxon test.

You can also represent boxplots to see the repartition of quartiles and median of the executions.

Good luck !

Friedman & Wilcoxon test

https://stats.stackexchange.com/questions/83904/friedman-test-vs-wilcoxon-test

I have used Friedman’s test.

"This function returns the p-value for the nonparametric Friedman's test to compare column effects in a two-way layout. friedman tests the null hypothesis that the column effects are all the same against the alternative that they are not all the same[1]"

[1] https://au.mathworks.com/help/stats/friedman.html

And also it is able to show the ANOVA table.

In the following links, you can find how to compare the performance of metaheuristic algorithms using statistical tests of Mean Absolute Error (MAE), Friedman, and Wilcoxon tests.

Article CCSA: Conscious Neighborhood-based Crow Search Algorithm for...

Article MTDE: An effective multi-trial vector-based differential evo...

Article An improved grey wolf optimizer for solving engineering problems

You can use these parameters Mean Squared Error (MSE), Mean Absolute Error (MAE), Correlation coefficient (R2) and their are some latest also you can use about your datasets based on what parameter you looking.

It's not clear as to what you want to evaluate. You described a task where you have an optimisation problem and there are several algorithms to find the optimal value. Do you know the true optimum? You say "Apart from maximum, minimum, average, and standard deviation", but what what do these measures relate to? More details are needed.

you can follow Rasel Ahmed 's comment

Hi. You might refer to this paper where 10 metaheuristic algorithms are compared by statistical tests. To perform a comparison among several algorithms, hypothesis testing is carried out. The null hypothesis (Ho) and the alternative hypothesis (Ha) is formulated to test the performance of the best algorithms.

Performance Studies of 10 Metaheuristic Techniques in Determination of Damages for Large-Scale Spatial Trusses from Changes in Vibration Responses

https://ascelibrary.org/doi/abs/10.1061/%28ASCE%29CP.1943-5487.0000872

Using the Friedman test and it is easily realized by matlab

Other than mean, standard deviation, best and worst values you can add the following,

Number of success

Average function evaluation

Average time elapsed

Providing convergence characteristics

Apart from all that you can find perform Friedman test.

I hope these will help you

You can use the Friedman Test [1, 2] and its corresponding Nemenyi Post-hoc Test [3, 4] to evaluate the statistical significance of differences in ranks.

Madjarov et al. [5] and Demsar [6] provide details on statistical tests used for algorithms comparison with multiple datasets.

[1] M. Friedman, A comparison of alternative tests of significance for the problem of m rankings, Annals of Mathematical Statistics 11 (1940) 86-92.

[2] G. W. Corder, D. I. Foreman, Nonparametric statistics for non-statisticians: A step-by-step approach, John Wiley & Sons, 2011.

[3] P. B. Nemenyi, Distribution-free multiple comparisons, Ph.D. thesis, Princeton University, New Jersey, 1963.

[4] I. Brown, C. Mues, An experimental comparison of classification algorithms for imbalanced credit scoring data sets, Expert Systems with Applications 39 (3) (2012) 3446-53.

[5] G. Madjarov, D. Kocev, D. Gjorgjevikj, S. Dzeroski, An extensive experimental comparison of methods for multi-label learning, Pattern Recognition 45 (9) (2012) 3084-104.

[6] J. Demsar, Statistical comparison of classifiers over multiple data sets, Journal of Machine Learning Research 7 (2006) 1-30.

The following article (R-GWO) shows how to compare the performance of an improvement of the GWO algorithm with eight comparative algorithms using three statistical tests, Wilcoxon rank-sum, mean absolute error (MAE), and Friedman.

Article R-GWO: Representative-based grey wolf optimizer for solving ...

In the following article, the performance of a new metaheuristic algorithm named QANA was compared with ten comparative algorithms using three statistical tests, Wilcoxon signed-rank sum, analysis of variance (ANOVA), and mean absolute error (MAE).

Article QANA: Quantum-based avian navigation optimizer algorithm

You can basically use two statistical tests to show the significance of your results.: Wilxcon's signed-rank sum and Friedman's tests.

The non parametric statistical test (wilcoxon rank-sum) is frequently used in this domain to judge whether the differences between results are significant or not. You can also apply Friedman test to rank the competetive algorithms over multiple cases.

The Wilcoxon test, which can refer to either the Rank Sum test or the Signed Rank test version, is a nonparametric statistical test that compares two paired groups. The tests essentially calculate the difference between sets of pairs and analyzes these differences to establish if they are statistically significantly different from one another.

Wilcoxon test! The implementation in Matlab is very easy!

ANOVA or Wilcoxon depending upon if you think it to be parametric or non-parametric in the distribution.

Hello sir,

If you want to compare in pairs, the best method is the Wilcoxon method. Also, if you want to compare 3 or more statistical tests together, you can use Kruskall wallis. For example: (A new fusion of ASO with SA algorithm and its applications to MLP training and DC motor speed control)

1-Find the average of fitness functions (30 runs) for two algorithms separately: find the p value using t-test.

2-Find the average of fitness functions (30 runs) for more than 2 algorithms separately: find the p value using ANOVA.

Mr. Soumen Atta, please view these articles :

- Deep Statistical Comparison for Multi-Objective Stochastic Optimization

Algorithms;

- Novel Approach to statistical comparison of meta-heuristic

stochastic optimization algorithms using deep statistics;

- DSCTool: A web-service-based framework for statistical comparison of

stochastic optimization algorithms;

- Performance assessment of meta-heuristics for composite layup

optimisation.

Personally, I think the Wilcoxon and Friedmann statistical tests suffice as both well-established and robust comparison methodologies. If you're feeling whimsical, however, you can try implementing the variations of the Friedmann tests widely available in literature.

let's say that we have algorithm A and B. firstly, we have to find out the mean of both algorithms (mean of 30 runs). if they are equal, we will no need any other statistic value . but, if not, we usually use rank sum test (wilcoxon rank-sum) to know whether the difference between the means is significant or not. by the way , if the P-value is less than 0.05 that indicate the difference between them is significant, which what we are looking for, normally.

Non-parametric tests such as Friedmann statistical tests, and Wilcoxon rank-sum tests are very common.

Dear Soumen Atta

I suggest you consult the Paper :

https://doi.org/10.1016/j.swevo.2011.02.002

Best regards

Professor Legón.

A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms

Joaquín Derraca,

Salvador Garcíab,

Daniel Molina,

Francisco Herreraa.

Keywords: Statistical analysis Nonparametric statistics Pairwise comparisons Multiple comparisons Evolutionary algorithms Swarmintelligence algorithms

a b s t r a c t

The interest in nonparametric statistical analysis has grown recently in the field of computational intelligence. In many experimental studies, the lack of the required properties for a proper application of parametric procedures –independence, normality, and homoscedasticity –yields to nonparametric ones the task of performing a rigorous comparison among algorithms. In this paper, we will discuss the basics and give a survey of a complete set of nonparametric procedures developed to perform both pairwise and multiple comparisons, for multi-problem analysis. The test problems of the CEC’2005 special session on real parameter optimization will help to illustrate the use of the tests throughout this tutorial, analyzing the results of a set of well-known evolutionary and swarm intelligence algorithms. This tutorial is concluded with a compilation of considerations and recommendations, which will guide practitioners when using these tests to contrast their experimental results.

I would like to invite you to have a look on the following links.

SSO: https://www.mathworks.com/matlabcentral/fileexchange/92150-sperm-swarm-optimization-sso

HSSOGSA: https://www.mathworks.com/matlabcentral/fileexchange/92130-hssogsa?s_tid=srchtitle

MOSFP: https://github.com/sh7adeh1990/MOSFP

Thank you very much for your time and efforts.

FYI

• Article The SOS Platform: Designing, Tuning and Statistically Benchm...

• https://doi.org/10.1016/j.swevo.2020.100665

The Wilcoxon signed-rank sum, analysis of variance (ANOVA), and mean absolute error (MAE).

The attached article discussed many such relevant parameters.

you can do that by using benchmark functions, you can check the link may be useful for you:

https://www.google.com/search?q=steps+tutorail+benchmark+function&tbm=isch&ved=2ahUKEwjLgtXUn-f2AhU-Q0EAHWK3CQEQ2-cCegQIABAA&oq=steps+tutorail+benchmark+function&gs_lcp=CgNpbWcQAzoFCAAQgAQ6BAgAEBg6BAgAEAE6BAgAEEM6BggAEAcQHjoICAAQBxAFEB46BggAEAgQHjoICAAQCBAHEB5QphRYjbUBYOPDAWgDcAB4AIAByASIAccskgEMMC4yNC4xLjIuMS4ymAEAoAEBqgELZ3dzLXdpei1pbWfAAQE&sclient=img&ei=s9hAYovdD76GhbIP4u6mCA&bih=696&biw=1519&hl=en-GB

Check

Bettinger, P., J. Sessions, and K. Boston. 2009. A review of the status and use of validation procedures for heuristics used in forest planning. Mathematical and Computational Forestry & Natural-Resource Sciences. 1(1): 26-37.

Use student t distribution test for comparison between the mean of small samples (n

I think the work of LaTorre, Antonio, et al. (2021) provides good guidelines for statistical comparison. Their second guideline related to the Validation of the Results and especially the paragraph 4.1 explain in which case each statistical test can/should be used when comparing several meta-heuristics: Article A Prescription of Methodological Guidelines for Comparing Bi...