You should use benchmark functions and real world engineering problems if the hybridization is not related to any specific problem. you have to set same iteration number and population or maximum number of evaluations to have a fair comparison. Also you can do p test

There may be n cases where heuristic 1 has provided better solutions than heuristic 2. There may be m cases where heuristic 2 has provided better solutions than heuristic 1. I would suggest trying n+m cases and see if the hybrid heuristic outperforms both the heuristics.

Jahedul Islam Anmar Abuhamdah C.P. Ravikumar

In order to combine two metaheuristics, can I just use the operators of one metaheuristic in another? for example if GA is the first algorithm, can I employ crossover and mutation operators in my second algorithm in order to generate new solutions and develop a hybrid algorithm? If yes, should the crossover and mutation operators completely replace the operators of the second algorithm or can I assign a probability for using both the GA operators and the second algorithm's original operators? Is there a rule for this?

Also if this adds a setting parameter, and I find the optimal setting by trial and error, how am I supposed to prove its optimality?

Also, my research is more focused on my developed model that I want to solve with these metaheuristics rather than the solution approach. Therefore, I wonder if I apply each metaheuristic and the hybrid on my own problem (instead of benchmark problems) with same number of population, and iteration (e.g. 1000) and report and compare the mean, SD and best solution of 1 time of performing each algorithm, would that be a valid comparison?

As far as I know, There is no rule. Also, you can make changes as long as it gives better result.

yes, you can apply on your own problem, however, when comparing with other techniques use the parameters that have been used in other research articles.

Dear Anmar Abuhamdah ,

Thank you so much for the information.

My research is more focused on my developed model that I want to solve with these metaheuristics rather than the solution approach. Therefore, I wonder if it is possible that I apply each metaheuristic and the hybrid on my own problem (instead of benchmark problems) with same number of population, and iteration (e.g. 1000) and report and compare the mean, SD and best solution of performing each algorithm. Should I run each algorithm on my own problem multiple times and report the mean and SD of the best solutions I get every time I run the same algorithm for a specific no of iterations or should I run each algorithm for a specific number of iterations once and report the SD and mean of the best solution in each iteration?

Dear Shervin Ataeian,

in order to assess whether the hybrid is better than the original or not as

Jahedul Islam mentioned earlier, one should use benchmark functions and real engineering problems. From my point of view, for each problem the following must be done:

For further reading, I suggest you read my paper entitled "Elitism in intensity-based image registration", where I tried to hyper-optimize Elitist Genetic Algorithm for Rigid Image Registration. Hope it helps.

Hi Shervin Ataeian

Are there other studies that investigate the same problem that you are solving where the authors report the efficiency of their particular solution method on a set of benchmark instances? If so, it is a good practice to use the same benchmark instances to replicate the tests and present a performance comparison of your method with respect to the other solution methods.

However, if no one has published such information about the problem you are studying, such a comparison is not necessary. More specifically, you need not implement any methods that you do not use in solving your problems even if your hybrid heuristic would combine different ideas from different heuristic methods. However, one thing you could do in this case, although not necessary, is to construct new benchmark instances which could consist of randomly generated instances of different sizes along with some real-world problems.

Finally, even if there are no existing benchmarks to compare with, you could of course implement the two original ones if you wish to provide a more comprehensive evaluation on the benefits of your hybrid method compared to the two individual ones. In such case, you could follow the suggestions by Constantinos Spanakis who provided excellent instructions on how to tune the parameters of your solution method and finally compare the results. Especially if you are writing a scientific article, is is likely that referees will ask you to somehow justify the selected parameter values and just writing that they were obtained by trial and error during development may not be enough.

I would perhaps add that the more repetitions you are able to perform, the better. I find that, for example, 10 is typically too few since it's easy to "cheat" and handpick "random" solutions that yield good overall results. You might also want to report the worst solution for each instance to set an example for other researchers that typically only report the best and average ones. It is also important to get an idea on the quality of not so good runs, and I believe reporting these by no means undermine the performance of your algorithm.

Best regards,

Juho

Hi,

You can compare the algorithms according to their average, best outcomes and also their run time.

Dear Shervin,

You should evaluate the new hybrid algorithm with a similar termination criterion and computational time as the baseline algorithms. I generally recommend to measure performance (average and best solution quality over several runs with different seeds) and (average) computational time on a wide enough set of benchmark instances for the problem at hand, and ideally compare with other studies published in your field.

Finally, I recommend demonstrating the "added-value" of your new components by evaluating the performance of the same method when you change some key parameters or deactivate some of these components (this is sometimes called "ablation study"). This information is essential for future works since other authors are likely to re-use some components of your approach but not necessarily the entire algorithm. Therefore, they will wish to know what is the relative contribution of all the design choices that you made.

If you wish some examples of component analyses in hybrid algorithms, you can consult Section 4.2 of the following paper:

https://link.springer.com/article/10.1007%2Fs10732-018-9372-2

or Section 5.3 of this one:

https://www.sciencedirect.com/science/article/abs/pii/S0377221719309804

Good luck!

--Thibaut

Dear Shervin,

To compare the performance of your hybrid algorithm with the two individual algorithms you need to first find the best parameters for each algorithm, then run the algorithms on a common platform and compare statistically them based on some performance measure. And yes the common algorithmic parameters need to be the same.

Dear my friend,

At the first step, if it is assumed that both algorithms are used in the same computer, we should consider some standard problems with different characteristics in that problem. Now, every algorithm is applied for several tests and their best, worst and mean are compared. Also, we can use some statistical index for more comparison.

Dear Dr. Yousefikhoshbakht

The aim of using the meta-heuristics is to find a good solution in a reasonable time. Of course, along with the objective function, the runtime of the algorithms must also be compared. In such a comparison, it should be noted that the differences between algorithms should be examined by the NHST (Null Hypothesis Significance Testing) in the terms of the objective function and runtime. If the difference is not significant (given the P-value), it cannot be concluded that one is better than the other.

Check

Bettinger, P., J. Sessions, and K. Boston. 2009. A review of the status and use of validation procedures for heuristics used in forest planning. Mathematical and Computational Forestry & Natural-Resource Sciences. 1(1): 26-37.

Besides validation mentioned by Dr. Bettinger. I plotted the solutions against the number of iterations. I could often see the improvement after initiating the new solution procedure.