How to compare two methods resolving a real world optimization problem? Is the canonical evaluation using instance of this problem sufficient? Is it more accurate to test their complexity or convergence or .. ? If yes how ?
I'm afraid, there is no simple (and general) method to grade such results.
The very problem in applying optimization to real-world problems is you can quite simply find a precise optimum of the wrong function.
Often, in fact, you want to optimize a few criteria and you have to use either multicriteria analysis, instead of optimization, or - aggregate the criteria somehow...
Also, the formula for the objective you have might be only an approximation of your actual objective - in that case finding the optimum, precisely, might give you no good...
It is all the matter of a specific application... and it is an art, I'm afraid...
Good luck with *your* application!
# Bartlomiej Kubica , I understood from your reply is :
To mesure the quality of a metaheuristic in resolving a real world problem is the quality of the optimum that it can reach ? I'm I correct ?
so using the basic "Mean" and "Standard_deviation" of the reached results using this metheuristic is enough to mesure its quality and performance ?
No, that was not really what I meant.
If you *can* grade the quality of the optimum, then yes, you can stick to using measures like Standard deviation, etc.
But what I wanted to say is that in real-world problems, it might be difficult to compare different solutions, as you - in fact - have several criteria and some of them are hard to express quantitatively.
Example: if you want to optimize the location of a plant, you optimize your outcome. Yes, but this outcome is uncertain.
And also, you should minimize the number of accidents and maximize your fraction of the market and etc. etc. How to put that all in an objective function???
it's true that every single criterion can't be incorporated in one objective function.. but then again there is a need for future research... a single research work is just one step in the bigger domain...
You can compare the solutions obtained by different soft-computing method in terms of their goodness... u can conduct statistical analysis such as paired-t-test to see the consistencies...
there is always sensitivity or qualitative analysis for any methodology depending upon your problems... i.e. for an example this would not be same for clustering problems and optimisation problems...
so comparison or analysis are problem specific more precisely what you exactly want to compare...
#Tamal Ghosh ,#Bartlomiej Kubica :
So the paired-t-test will be a good option to evaluate the performance of metaheuristic.
- What is the basis of such test ? what doses is mesure precisily ? what is it's principal?
Is there any other statistical test like this one for evalutating metaheuristics performances ?
Generally speaking, you should compare different methods simultaneously for 2 criteria - accuracy and calculation time. It isn't easy, so really it is recommended to fix calculation time (e.g., 1h or 10 h) and compare only according accuracy. Please, see Article: The Cross-Entropy Method for Continuous Multi-Extremal Optimization. (Dirk P. Kroese, Sergey Porotsky, Reuven Y. Rubinstein). Methodology And Computing In Applied Probability 08/2006; 8(3):383-407.
https://www.researchgate.net/profile/Sergey_Porotsky/publications
- Badges
- Science topic