This is a good reference giving comprehensive overview of forecasting:

otexts.org/fpp2/

Our most recent article published in the Information Sciences shows an extensive evaluation regards statistical and machine learning algorithms for time series prediction. We also discuss the best conditions for the use of each model. Please, see the reference: Article Evaluation of statistical and machine learning models for ti...

Hi Fouad,

I would say that one helpful way to go includes two stages. First, one should evaluate out-of-sample the predictive accuracy of different candidate forecasting methods. As a second step you need a way to aggregate the best performing methods.

The basic ideas here are the following two: whatever theoretical metric you could use to evaluate forecasting methods might not be useful in real life applications. That´s why you will find the recommendation of out-of-sample evaluations, ie you mimic a real life situation, estimate your models with half of your data and then generate forecasts for the other half. Finally you evaluate with Mean Squared Prediction Error which method performed better in that portion of the sample.

The other important idea is that the good performance of a forecasting method in a given sample does not need to be the same in another sample. That is the reason why people decide to combine forecasts coming from different methods. There is a search for robustness in this practice. Typical combination methods include simple average of the forecasts, or the median of a group of the best performing methods in the out-of-sample portion.

Hope it helps!

I use two objective functions: accuracy and parsimony.

For accuracy I use RMSE, MAPE and Theil's U Statistic on out-of-sample data. RMSE is useful both for comparing models and as a loss function for hyperparameter optimization. MAPE is useful to communicate the model accuracy with non-experts. U Statistic shows how better is your model when compared with a simple naïve model.

The parsimony, in the other hand, is measured as the number of parameters of the model, and parsimonious models are those that use less parameters.

With these measures you can compare different methods and optimize the selection, chosing the most accurate and parsimonious model.

I almost forgot! You can also use the Aikake Information Criterion (AIC) or the Bayesian Information Criterion (BIC).

Ayman Amin , Thanks for the link. Rob is my favorite and I think this ref is so helpful to those who want to learn the concept of relating prediction performance to time series characteristics. The concept is know as the meta-learning. I'm more in the need of some information theoretic measures like entropy or the Hurst exponent that describe time series data from different angles.

Try the following link:

Article Rule induction for forecasting method selection: Meta-learni...

Dear Vinicious,

I read your paper. Very nice paper. I like the idea of categorizing prediction methods into parametric and non-parametric. A very wise choice as well as the performance metric you used in your paper.

Dear Pablo Pincheira ,

Thanks for this precise explanation. Though, I've focused on a bit different aspect. At the moment, I am trying to find out if the right prediction method can be picked based on the given time series. For example, is there a particular type of time series on which neural networks outperform the others? It might not be so clear from the abstract level but there might be some explanations. So I'm looking for new metrics and tests.

Dear Petrônio Cândido de Lima e Silva ,

Let's put it this way, rather than the performance metrics (which you have precisely explained about here), metrics that relate the model's accuracy to the given time series is what I am looking for.

I think that the set of useful metrics are symetrical absolute percentage error (SMAPE), relative root mean squared error (RRMSE), mean absolute error (MAE) and others. If you are looking for model stability, standard deviation of errors could be interesting. Alongside, if you are looking for for model complexity, use informations criteria like AIC, AICc or BIC could be useful, once they use the number of parameters to compute the criterion.

Few techniques that should be considered while finding the best model:

1) Data processing technique (i.e. sliding window) and data transformation technique (i.e. normalisation)

2) Comparison of different algorithms, i.e. ARIMA VS RNN.

3) Comparison of varying forecast strategies, i.e. MIMO VS DIRMO.

4) Considering the model evaluation procedure (Cross-validation is suggested) and metrics (MAE, MSE, RMSE, MAPE, SMAPE, etc.).

5) More information can be obtained from related research papers.

Over decades, many methods have been proposed for time series forecasting. However, it has been proven that none of them is universally valid for every task/ application and even within the same application, models performance may be subject to significant changes over time partially due to the changes in the time series itself (so-called concept drifts). In this case, one reasonable solution is to go for dynamic ensemble methods constructions.

In a recent work of mine, I have devised a self-adaptive ensemble framework for time series forecasting which decides automatically which models to include to perform the prediction task at a given point of time and adapt the selection following the changes in performances of the candidate models.

To have more detailed view of the method, I recommend to read my last paper:

Conference Paper A Drift-based Dynamic Ensemble Members Selection using Clust...