Dear all,
There are several performance metrics to assess the accuracy of a predictive model such as MAE, MSE, MAPE, r2, R, R2, RMSE, RRMSE, PI, d, E. Is there a combined index?
Thank you in advance.
Sadjad
Article Assessing the accuracy of predictive models for numerical da...
I guess it also depends of the your problem. Would be it possible to give more informations about your problem that you are trying to solve?
Best,
Vinicius
Dear you can use the
Root Mean Square Error RMSE to measures how much error there is between two data sets. In other words, it compares a predicted value and an observed or known value. It's also known as Root Mean Square Deviation and it is one of the most common measures in predictions.
see this:
http://statweb.stanford.edu/~susan/courses/s60/split/node60.html
https://gisgeography.com/root-mean-square-error-rmse-gis/
There are different metrics because it depends on your problem and what is important to you. For example, squared-error metrics (MSE) tend to penalise large errors heavily, while absolute-error metrics (MAE) are more robust to single-outlier-error type scenarios.
In energy modelling CV(RMSE) and Normalised Mean Bias Error are used to compare between different data sets with different magnitudes. I also see MAPE a lot. Stay away from R2.
Dear Friends,
Thank you for your comments. I know the common performance metrics. I would like to find a mixed criterion which includes the most important errors together.
Best,
You can't have it all. That is why there isn't "one metric to rule them all". You could construct your own one by taking something like 0.5*MSE+0.5*MAE, but you'll have a hard time defending it. Some metrics are also incommensurable, e.g. bias and variance metrics. That's why it is called the "bias-variance tradeoff", and why the two are often represented on orthogonal axes.
Thank you. MAPE, R and RMSE (or RRMSE) are very important I think. To compare the accuracy of two methods (not assessing their accuracy separately), It will be good if we can find a combined metric.
It depends on the loss function you used to produce optimal forecasts. If you optimized your algorithms for a symmetric linear loss, then use MAE, if for a symmetric quadratic loss, then use MSE.
If you have one time series, use out-of-sample MAE or MSE/RMSE. If you have many time series, I recommend the AvgRelMAE or AvgRelMSE/AvgRelRMSE as described in this chapter:
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
MAPE is not recommended, see the above chapter for the discussion of the problems of MAPE/MdAPE/sMAPE/etc.
Dear Sadjad Gharehbaghi ,
the section 5 of the following reference tried to give the greats sides and shortcomes of some performances indicators.
Despotovic, M., Nedic, V., Despotovic, D., & Cvetanovic, S., Evaluation of empirical models for predicting monthly mean horizontal diffuse solar radiation, Renewable and Sustainable Energy Reviews, Volume 56, April 2016, Pages 246-260, ISSN 1364-0321, http://dx.doi.org/10.1016/j.rser.2015.11.058.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics