I was wondering if a portfolio hyper-heuristics mechanism could be an efficient method to analyse the performance of various algorithms. I am aware they have been used in optimising the choice of algorithm. However, i am not sure they would be suitable for analysing the performance. I would appreciate it if you could help me. Many thanks in advanced for your contributions.
What is a "portfolio hyper-heuristic mechanism"?
The best performance measure I have seen so far takes into account both how well an algorithm solves a problem AND how fas, AND in comparison with others. It is called the "performance profile" and is presented in a very well-cited paper by Dolan and Moré:
http://link.springer.com/article/10.1007%2Fs101070100263#page-1
Thank you Michael for this very interesting answer. Now is this method suitable for non-deterministic algorithms?
The algorithm search for the answer. We are making a trade-off for
That may depend on your definition of "success". Suppose you know the optimal solutions to those problems that you want your non-deterministic algorithms to compete on. I would in fact argue that you SHOULD test your algorithms with that knowledge beforehand, since your comparison will be much less precise otherwise (you simply would not know what the golden goal is!). If you know the solutions, then the performance profile will yield the perfect picture of both how well the non-deterministic methods can solve these problems, AND you will know how the competitors you have coded will compare against each other. Then you may assess the time it took for the exact algorithms to reach the optimal solutions in comparison with the time the heuristics used, and say something about the trade-off between time and accuracy, perhaps.
We followed a closely related idea. The main difference that we did not refer to "hyper-heuristics". Instead, we posed the following question:
When assessing the performance of different solvers it is customary practice to use the Virtual Best Solver (VBS) defined as the best solver instance per instance and the Simple Best Solver (SBS) defined as the solver which solved more problems or did so with better quality. Does it make sense to compute a portfolio and then to compare the performance of each solver wrt the portfolio.
For this, we generated a portfolio as the solution found by a Linear Programming Task. In case one solver is much better than all the others, the solution would consist of that specific solver. The typical (linear) combination, however, considers a subset of the candidate solvers. Those that are not part of the portfolio have a behaviour that can be explained as a linear combination of the others and thus, we regard them as not contributing to the advancement in the field. Others, however, if are considered in the solution then they have abilities for efficiently solving specific problems.
We applied this technique for various purposes:
1. To assess the performance of the planners in the 2011 International Planning Competition: https://www.researchgate.net/publication/259517147_How_Good_is_the_Performance_of_the_Best_Portfolio_in_IPC-2011
2. To automatically generate portfolios that won the Learning track of the 2014 International Planning Competition and that was awarded with two silver medals in the 2013 SAT Competition: https://www.researchgate.net/publication/277338269_Automatic_construction_of_optimal_static_sequential_portfolios_for_AI_planning_and_beyond
For a general discussion of the approach followed you can also see: https://www.researchgate.net/publication/259502995_Performance_Analysis_of_Planning_Portfolios which is specific to solvers that optimally solve problems. The other papers describe also the approach to suboptimal solvers.
Hope this helps,
Conference Paper How Good is the Performance of the Best Portfolio in IPC-2011?
Article Automatic construction of optimal static sequential portfoli...
Conference Paper Performance Analysis of Planning Portfolios
Hi,
You may have a look to this paper:
Portfolio Diversification and International Corporate Bonds.
By: Liu, Edith X. Journal of Financial & Quantitative Analysis. Jun2016, Vol. 51 Issue 3, p959-983. 25p. DOI: 10.1017/S002210901600034X.Sujets: Diversification (Finance);
Corporate bonds; Financial crises; Rate of return; Loss control; Liquidity (Economics); Stocks (Finance); Central credit unions; Financial Transactions Processing, Reserve, and Clearinghouse Activities; Capitalists & financiers -- United States
Best regards
- Badges
- Science topic
- Similar topics
- Political Science
- International History and Politics
- Globalization