I am looking for some analytical, probabilistic, statistical or any other way to compare the results of a different number of approaches implemented on the same test model. These approaches can be different optimization techniques implemented on the similar problem or different type of sensitivity analysis implemented on a design. I am looking for a metric (generic or application-specific) that can be used to compare the results in a more constructive and structured way.
I would like to hear if there is a technique that you use in your field, as I might able to drive something for my problem.
Thank you very much.
Given that you know the parameter value, you can compare the results in terms of bias (absolute or relative), standard error, mean squared error, and 95% coverage probability. Otherwise, you can use the cross-validation technique to calculate the mean squared prediction error (for continuous outcome) or misclassification error rate (for binary outcome).
Best results obtained, computational time and standard deviation of the solutions can be used for comperison. Also if you use bench mark dataset then it is easy for you to compare.
https://www.academia.edu/20440556/Comparison_of_Different_Approaches_to_Evaluate_Proficiency_Test_Data
Well, it's really a good question in the field of computational and applied mathematics because such things are usually faced in these fields.
Suppose your problem (in any domain) is of a single variable (independent variable) only i.e. scalar problem for example flow in uni-direction, optimization of single-variable function, the root of a non-linear equation in one variable etc. Now we have two solutions from two different methods say S1 and S2.
Case I: If there is the availability of analytical (exact) solution (Sa)-
then at each x belongs to the domain of the problem, find the summation of |Sa - S1| and |Sa-S2| over the domain which represents the error in both methods respectively. The one which has less error will be the better method.
Case II: If the exact solution is not available -
Then we directly put the value of discretized x in function and compare its value with the same obtained from each solution, the one which has less error will be better.
Now if the problem has several variables then they can be represented in vector form and now the modulus will convert into the norm.
I hope this answer will help you.
The first that comes to mind are to use performance profiles of various kinds. (Look up the term!)
This is a starting place:
E. D. Dolan and J. J. Moré. 2002. Benchmarking optimization software with performance profiles. Mathematical Programming 91, 2 (2002), 201–213.
There have been some criticisms on this and other profiles, as it is not easy to trim them to be fool-proof. I suggest that you perform a forward search to find some later paper on performance profiles that have made them more robust.
There are well-established statistical methodologies for comparing optimization algorithms. I recommend you to review the following paper, it is from 2020.
LaTorre, A., Molina, D., Osaba, E., Del Ser, J., & Herrera, F. (2020). Fairness in Bio-inspired Optimization Research: A Prescription of Methodological Guidelines for Comparing Meta-heuristics. arXiv preprint arXiv:2004.09969.
Let me recommend this forecast evaluation framework:
Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis