Hello Ankit,

1. You could conduct a simulation study in which you create a generating function for some "true" model(s), then compare your two algorithms with respect to RMSR of parameter estimates (vs. the true/generating model values) over several thousand replications.

2. Run-time might be an interesting comparison as well, especially for persons interested in doing Monte-Carlo/simulation studies. For the typical person running a (single) regression analysis, chances are that run-time differences wouldn't be noticed.

Good luck with your work.

https://machinelearningmastery.com/statistical-significance-tests-for-comparing-machine-learning-algorithms/

https://www.hindawi.com/journals/sp/2019/8706505/

Ankit -

Better in what way? For prediction? Considering estimated variance of prediction error? You could compare results in each case by using a graphical residual analysis with both sets of results on the same scatterplot each time (with predicted y on the x-axis, and estimated residuals on the y-axis). After examining them you might decide to measure an overall average performance, perhaps for the estimated random factors of estimated residuals, if you account for heteroscedasticity (where e = (e0)(z^gamma), where e0 is the estimated random factor of an estimate residual, z is a size measure, say predicted y, and gamma is the coefficient of heteroscedasticity). Or you might lookout for worst performances.

Anyway, the term "better" can mean a lot of things.

Best wishes - Jim

I order to find out which method is better, you need to

1) define a good error measure

In our chapter "Forecast Error Measures: Critical Review and Practical Recommendations" we looked at the task where we compare two forecasting methods:

Chapter Forecast Error Measures: Critical Review and Practical Recommendations

One requirement for the error measure (the above chapter, page 3) is "the use of the same loss function that was used for optimisation"

So your error measure must correspond to the optimization criteria.

Generally, you "must first obtain the density forecast using a statistical model, and then adjust our estimate given our specific loss function" (page 3). But "Fitting a statistical model usually delivers forecasts optimal under quadratic loss."

Let's assume you optimized your forecasts for the quadratic loss.

Then you need to find the loss differential for the quadratic loss function for each obervation i:

Li = methodA_error_i^2 - methodB_error_i^2

2) apply a statistical test:

Usually, the distribution of Li will not be normal.

On page 9 of the above chapter we suggest:

"Assess the statistical significance of changes by testing the mean of 𝑙𝑖 against zero. For this purpose, the Wilcoxon’s one-sample signed rank test can be used (assuming that the distribution of 𝑙𝑖 is symmetric, but not necessarily normal). If the distribution of 𝑙𝑖 is non-symmetric, the binomial test can be used to test the median of 𝑙𝑖 against zero. If the distribution has a negative skew then it is likely that the negative median will indicate negative mean as well."

References:

Davydenko, A., & Fildes, R. (2016). Forecast Error Measures: Critical Review and Practical Recommendations. In Business Forecasting: Practical Problems and Solutions. John Wiley & Sons Inc.

You can compare with their variances. The less variance method is better .

Use. IMSE or BIC

Use the ratio performance deviation (RPD)