A multi-objective EA with 2 objectives should produce a Pareto set as it's output. I can easily graph these and produce nice charts. But remembering the advice of my PhD supervisor I must run my algorithm multiple times before considering the results from it.
Suppose I have to MOEA algorithms both of which optimise the same two objectives on the same problem. After running them both for, say 20 times, I'm left with a group of Pareto sets from each algorithm.
I can establish if a specific objective is significantly different between algorithms by using a T-Test on the two sets of results for that objective.
In terms of visualizing the results there would appear to be two things that I could do, firstly graph all of solutions in each of the Pareto sets, but that leads to a very crowed graph. Alternatively create a final Pareto set of the non-dominated solutions from each set created by each algorithm.
So my question is this, are either of the above approaches particularly advisable or not and is there another approach that I've missed out on?
Hi Neil,
I doubt there is an "optimal" method, because it depends on what you value about each Pareto front you obtain. Of course be cautious about t-tests: they depend on some assumptions, such as near-normality of the data, that may be violated. You might like to look at M. Li, S. Yang, and X. Liu. "Diversity comparison of Pareto front approximations in many-objective optimization". IEEE Transactions on Cybernetics, 44(12): 2568-2584, 2014, IEEE Press (see http://www.brunel.ac.uk/~cspgmml1/Publication.html for a preprint) which proposes a diversity comparison indicator and usefully lists a number of other approaches.
Metrics of Performance for comparing different multi-objective optimization algorithms are discussed in section 5 (p 179) of attached highly cited article:
- The distance of the resulting nondominated set to the Pareto-optimal front should be minimized.
- A good (in most cases uniform) distribution of the solutions found is desirable. The assessment of this criterion might be based on a certain distance metric.
- The extent of the obtained nondominated front should be maximized, i.e., for each objective, a wide range of values should be covered by the nondominated solutions
Zitzler, Eckart, Kalyanmoy Deb, and Lothar Thiele. "Comparison of multiobjective evolutionary algorithms: Empirical results." Evolutionary computation 8.2 (2000): 173-195.
http://www.tik.ee.ethz.ch/sop/publicationListFiles/zdt2000a.pdf
I think minimum 30 runs on each problem instances is required to make a fair comparison between the meta-heuristic algorithms.
An important question to ask well before plotting is: are these points guaranteed to be Pareto optimal? That is, do the properties of (a) the problem and (b) the algorithm ACTUALLY guarantee that these point definitely are Pareto optimal solutions? Without such guarantees you cannot draw any such conclusions.