I have modeled four non-seasonal exponential models using time series data on pork production over the period 1978-2012. data from 2013-2017 were hold-out for model validations. For simple exponential smoothing, stationary-R =—0.018; MAPE= 17.262; normalized BIC = 13.292. For Holt’s linear, the values were 0.540, 20.360 and 13.361, respectively. For Brown’s linear, they were 0.473, 20.317 and 13.405, respectively. For Damped trend, the values were 0.07, 16.119 and 16.979, respectively. Based on these data, what should I select as the best model to predict pork production?
The measures on the training set are not really suitable as a basis for model selection. This is because in the training set it is always possible to overfit. You could get the model with the excellent fit on the training set but very poor prediction capability on the test set that you hold out for the model validation.
Information criteria like AIC ( Akaike Information Criterion) or BIC (Bayesian Information Criterion) penalize the model complexity, accordingly. Therefore, they generally are better suitable for model selection.
A more appropriate is to assess measures of fit such as MAPE or MASE on a test set. This way you reduce overfitting because the models are fit on the training set but validated on the test set.
Thus, the model's performance in the test set is the best guide to its ability to predict the future. The caveat here is the test set (in your case only 4 years 2013-2017) is usually a much smaller sample of data than the training period (1978-2012). Hence, it is possible that a model may do unusually well or poorly in the test set merely because of being lucky or unlucky. Unless you have enough data to hold out a large and representative test set, it is probably better to interpret the validation set statistics in a more qualitative way: do they have a "red flag" on the possible unreliability of statistics in the test period, or not?
Thank you very much for your comments. I want to clarify these matters. First, is it okay to select the model with the lowest forecasting error (e.g. MAPE, MAE) in the validation period as the best model? However, in most occasions, the forecasting Errors in the validation period was almost the same among the four candidate models. In such circumstances, what would be the second criterion to select the best model for forecasting? Normalized BIC, MAPE or stationary R-square?
Dilan Dissanayake
If the forecasting error in the validation period is almost the same for all models then the second criterion would be the simplest (computationally and structurally) model. Quantitatively I would recommend BIC because it generally penalizes the model's complexity while MAPE and R-square do not.
Alexander Kolker, Thank you very much for your comments and they are really valuable.
- Badges
- Science topic
- Similar topics
- Statistics
- Time Series