Hi Ivan,

Efficiency can be measured several ways. Of course speed and quality of the best solution found are important but you can also consider additional criteria such as number of calls to the evaluation function, stability or robustness for instance. You can also provide some statistics (deviation, asymmetry) on your results, provided you perform enough runs on the same instance. TTT (Time To Target) plots also help visualizing the way your metaheuristic behaves. You can download a Perl program to compute such plots at Mauricio Resende's homepage (http://www2.research.att.com/~mgcr/tttplots/)

Besides, it is always tricky to compare methods, especially non-deterministic heuristics. Ranking methods (computing the rank of each method on a set of instances, given the value they get) are less sensitive to noise. Bonferroni statistical tests can be used as well. I also suggest you read the work from Eric Taillard (http://mistic.heig-vd.ch/taillard/codes.dir/comparaison.dir/comparaison.htm) on a way to compare non-deterministic heuristics. It is a bit old, but it is still valuable.

Hi Duhamel,

Thank you for your quick reply. I will check the links and look for a way to include that in the comparison.

In my opinion, your topic was discussed before. Please have a look at the links:

https://www.researchgate.net/post/Compare_the_complexity_of_metaheuristic_algorithms

https://www.researchgate.net/post/Which_techniques_are_best_to_measure_the_performance_of_an_optimization_algorithm_solving_test_functions_like_rosenbrock_langerman

I guess you will compare single-objective meta-heuristics. In this case, two typical ways to do that are:

1) Execute (using a certain test problem) both meta-heuristics during a given number of iterations/evaluations of the fitness functions/time and to determine which of both obtains the best result according to the fitness function

2) Execute (using a certain test problem) one meta-heuristic during a given number of iterations/evaluations of the fitness functions/time and to execute the other one, then analyzing if the latter reach the solution of the former, and if it is so, which of them is faster.

I remark, that it is very important to determine the termination criterion. The number of iterations is not suitable, because if you compare different approaches (e.g. Simulated Annealing and Evolutionary Algorithms), surely one of the methods will perform much more iterations in the same runtime than the other one. The number of evaluations of the fitness function (usually 25.000) is often used in this context. Finally, the runtime is also applied to carry out comparisons (the runtime is a good way to determine the computational cost of running an algorithm). On the other hand, it is very important to consider the statistical analysis of the results. If, for example, you run both meta-heuristics only once, your conclusions would not be so robust as if you run both methods in each test problem 15, 25 or more times, and return the best, mean, and other statistical results. I suggest to read papers dealing with the design of "sensitivity analysis" in metaheuristics.

In case you compare multi-objective approaches, you can use any of the large number of metrics proposed for this purpose (e.g.: http://sci2s.ugr.es/keel/pdf/specific/congreso/okabe03a.pdf )

Thank you for your feedbacks. I have seen articles talking about function evaluations, but I was not so sure about using that criterion, since I think that swarm strategies (e.g. PSO or CFO) would be in some disadvantage against single point ones (since they evaluate several times the objective function on each iteration). Anyway, I will try and run some tests considering this, and see what happens... Thanks again...

Hi, first of all, remember the No Free Lunch Theorem! As the theorem implies, the result of any comparison between metaheuristics depends on the test functions that you use. You might get completely different outcomes just by changing the test functions. Therefore, the first consideration is what do you want to do with your metaheuristic. If you are interested in some kind of real-life problem, choose test functions that reflect the features of the problem that you want to solve. Regarding the criteria to evaluate the performance, I would use the number of fitness function evaluations performed (if the optimum is found), or the value of the evaluated function (if the optimum is not found). You could also set a number of thresholds on how close is the best solution found to the global optimum, and compare how quickly the different metaheuristics reach these thresholds of precision. In any case, I would use the number of fitness function evaluations as they best reflect the "effort" made by the algorithm to find the optimum.

Hi all,

To evaluate performances of metaheuristics (only empirically), you can use benchmark functions (for more details, please see http://www.lri.fr/~hansen/Tech-Report-May-30javascript:-05.pdf ) and you must to set for all algorithms the same parameters like: maximum number of function evaluations, dimension D, number of independent runs, etc.

Then, to compare the performance of your algorithm with the other state-of-the-art algorithms, you can perform a Friedman test (for more details, please see http://juangvillegas.files.wordpress.com/2011/08/friedman-test-24062011.pdf).

https://www.researchgate.net/publication/297245624_Particle_Swarm_Optimization_Algorithm_and_its_Codes_in_MATLAB

Research Particle Swarm Optimization: Algorithm and its Codes in MATLAB

https://www.researchgate.net/publication/296636431_Codes_in_MATLAB_for_Particle_Swarm_Optimization

Code Codes in MATLAB for Particle Swarm Optimization

https://www.researchgate.net/publication/280244004_A_comparative_study_of_metaheuristic_optimization_approaches_for_directional_overcurrent_relays_coordination

Article A comparative study of metaheuristic optimization approaches...

https://www.researchgate.net/publication/280244004_A_comparative_study_of_metaheuristic_optimization_approaches_for_directional_overcurrent_relays_coordination

Article A comparative study of metaheuristic optimization approaches...

https://www.researchgate.net/publication/297245624_Particle_Swarm_Optimization_Algorithm_and_its_Codes_in_MATLAB

Research Particle Swarm Optimization: Algorithm and its Codes in MATLAB

https://www.researchgate.net/publication/296636431_Codes_in_MATLAB_for_Particle_Swarm_Optimization

Code Codes in MATLAB for Particle Swarm Optimization

The 7th section, namely "Results, Data Analysis, and Comparison", of the following current-state-of-the-art research paper have a sufficient answer for this question:

Article Sea Lion Optimization Algorithm for Solving the Maximum Flow Problem