Dear researchers,
I've a selection problem, in which I need to select the top N items from a given list. The selection should be optimised based on some weighted selection criteria. I've used three optimization methods, gravitational search algorithm (GSA), particle swarm intelligence (PSO) and genetic algorithm (GA). Each method is returning different selection sets with different ranking. My questions are:
1. How do I assess which ranking is best?
2. Is there any mathematical model to assess the correctness of the ranking?
3. How else can I compare the performance of the algorithms in the context of this particular problem?
Any help would be appreciated.
One thing you could try is through backtesting of your optimised value. You take an in-sample rolling window, and predict an out-sample data, and see whether it matches the result you got for that time point.
Next thing, you assess the convergence characteristic of the different methods you used.
Alternatively, you could compare which method is more rapid.
Do a convergence analysis for performance.
Model to assess is your defined fitness function.
First you need to recognize that, for each method you get different answers based upon initial conditions. In that context, you should go through multiple trials for each method (with differing initial conditions) to determine how much variance to expect. Then, consider the genetic algorithm and the attached powerpoint.
1. On what basis the optimizers are returning a ranking? Based on some weighted selection criteria, right?
Do the optimizers return the same value for your objective function with different rankings? Are you running your algorithms only once or more?
Because of the involvement of random variables in each of your algorithms, it would be better if you run them a good number of times (say, 30). Then as suggested by Maude Josee Blondin you should go for some standard statistical tests to find out any statistically significant differences among your algorithms. Hopefully, after 30 runs, you would be better placed to compare your algorithms.
However, if your algorithms are returning the same value for the objective function even after 30 runs, then you may go for any ranking provided by any algorithm under test.
2. You should compare your ranking against those obtained by previous researchers. Look for the mathematical model developed by them; you may use the same one and try to prove that your one is providing a different ranking with (hopefully) lower objective value (considering minimization problem).
3. If you get the same objective function value even after 30 runs for each algorithm, check with the speed of the convergence and the number of function evaluations required by the optimizers.
If you get different objective values after 30 runs, the mean/ median/ standard deviation should give you an idea of which one is better. Moreover, you may also comment on the abilities of the exploration/ exploitation for each of the optimizers.
Thanks Vaina Venkiah Rohail Gulbaz Barry Fox Maude Josee Blondin Ashik Ahmed for your valuable suggestions.
The precise problem is not clear to me: what are the objective functions, what the constraints. What is here "list" : is this a finite ordered collection of numbers?
Etc. Only when you have the precise mathematical problem one could help you a bit further...
You can select a simple numerical example and solve it mathematically to get the best point. This solution is used as a measure for your solution.
Metaheuristics are problem dependent. Search landscapes differ and you need to run the algorithm for a required problem in order to get the relevant statistical metrics for comparison. Your best bet moving forward for your particular problem is with either the Wilcoxon test or Friedmann rank test. Studying the convergence behavior of your curves will also give you an indication of which algorithm performs the best.
- Badges
- Science topic
- Similar topics
- Mathematics