I have 245 daily stock returns with some explanatory variables. I will compare the ARIMA and ARIMAX models with some basic machine learning techniques. The training periods will be 30, 60, 90, 120 and 150 days. Forecasting horizons will be 1, 2, 3, 4 and 5 days. I will use the sliding windows for these training periods. In the R program auto.arima will help me to find the best model for ARIMA and ARIMAX for each period. For example, if I have a 30 days training period, I will have 215 regression results by using a sliding window. The problem is that when the training period shifts, auto.arima can find different ARIMA models in each shift. I would like to evaluate the models based on error measures such as MASE, MAPE, MAE, RMSE etc. How I can combine the results to evaluate the models based on error measures. In 215 regressions, there are several different ARIMA models. There are only two issues that are the same for all 215 regressions. One is the training size and second is the forecasting horizon. In each regression, picking the model with minimum error, then counting them until 215.th regression will not provide a statistical or mathematical knowledge. Do you have any suggestions to compare the models as a whole?
Best Regards,
Let me recommend this chapter on error metrics:
Chapter Forecast Error Measures: Critical Review and Practical Recommendations
I would recommend the use of the AvgRelMAE measure.
If you have only 1 time series, AvgRelMAE will become RelMAE.
Also, you need effective data structures to store your forecasts.
Here I recommend the TSTS and FTS data schemas from this paper:
Conference Paper Data Schemas for Forecasting (with Examples in R)
Basically, you can use different sliding windows and compare alternative approaches using the data schemas described in the above paper and the metric described in the above chapter.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression