Pooria Behnam in the case of RMSE takes the root of the mean square error thus the numeric value seems to be lower. But in the case of MAPE, The performance of a forecasting model should be the baseline for determining whether your values are good. It is irresponsible to set arbitrary forecasting performance targets (such as MAPE < 10% is Excellent, MAPE < 20% is Good) without the context of the forecastability of your data.

Pooria -

Perhaps this is because your numbers are small, so your RMSE might actually be relatively high. As for R-square, it can be misleading. You can search the internet for info on that. (You might also recalculate to make sure you did that correctly.)

For comparing performances of regressions, I prefer "graphical residual analyses." You could plot the predicted y on the x-axis, and estimated residuals on the y-axis. You could plot for more than one model on the same scatterplot to facilitate comparisons. ("One picture is worth a thousand words," or statistics in this case.)

A "graphical residual analysis" is for checking model fit. "Cross-validation" is for studying whether your model may be overfit to your specific data sample.

By the way, "y" is usually used for your dependent variable, and "y_hat," or better "y*" (for weighted least squares regression, using notation from G.S. Maddala) is the predicted y, so you might write y = y* + e.

You might look at the following to see if there is anything of use to you:

https://www.researchgate.net/publication/313925533_Comparing_Model_Performances_Graphically

Cheers - Jim

PS -

For the skeletal format y = y* + e, very useful for predictions for finite populations, you might consider the following, which discusses the fundamental nature of heteroscedasticity in regression:

https://www.researchgate.net/project/OLS-Regression-Should-Not-Be-a-Default-for-WLS-Regression, including updates

Dear Ahan Chatterjee,

Thank you for your response. According to your words, the performance of a regression algorithm should be evaluated based on the forecastability of the data, and a single low/high value of MAPE does not reflect its goodness. My question is how to evaluate the forecastability of the data and its corresponding acceptable MAPE?

Imagine we had the following results for 2 different regression model:

model 1 : R2=0.99, RMSE= 0.02, MAPE= 1%

model 2 : R2=0.99, RMSE=0.01, MAPE= 25 %

Do you mean that model 2 can be a good regression model?

Many thanks

Dear James R Knaub

Thank you very much for your answer. The idea of plotting the residual plot is really interesting and helpful.

Let me recommend this chapter where we illustrate the problems of MAPE:

Chapter Measuring Forecasting Accuracy: Problems and Recommendations...

Generally, MAPE is not recommended due to the reasons described in the chapter. A shorter version of this chapter is available here: Chapter Forecast Error Measures: Critical Review and Practical Recommendations

The above chapter also introduces the following rule: the criteria you used to optimize forecasts should correspond to the evaluation metric. In your case, as I understand, it is out-of-sample MSE.

It is well-known from the times of (Zellner, 1979) that optimal forecasts in terms of linear loss will correspond to the median of density forecast and optimal forecasts in terms of quadratic loss will correspond to the mean. If forecast density is symmetric, I would recommend out-of-sample MAE.

Andrey Davydenko

Thank you for your response

Pooria Behnam Hope it was helpful. Let me recommend our new detailed framework for forecast evaluation and visualization:

Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...