I am familiar with LSTM neural networks which work well for energy demand forecasting. However, I would like to compare its performance with the two other machine learning methods. What are the best options to choose?
Thanks
Hi Amirreza,
The state-of-the-art in time series forecasting associated with traditional statistical and machine learning methods is one of the key elements of our latest paper published in Information Sciences: https://www.researchgate.net/publication/330742498_Evaluation_of_statistical_and_machine_learning_models_for_time_series_prediction_Identifying_the_state-of-the-art_and_the_best_conditions_for_the_use_of_each_model
From a systematic review of the last decade, we expose the benefits and limitations of 11 traditional predictors (MA, SES, HES, AHW, MHW, ARIMA, SARIMA, MLP, LSTM, SVM, and kNN-TSPI) by running them on 95 datasets from synthetic and real domains. We report many lessons learned and recommendations concerning the advantages, drawbacks, and the best conditions for the use of each model.
We also performed experiments with SRN and RNN to compare them with MLP. RNN obtained a lower predictive performance than SRN and MLP. In contrast, SRN and MLP achieved comparable results, since we did not find statistically significant differences between their performances. As MLP is a more straightforward approach and has faster execution time, we prefer to keep only the results of this neural network in the paper.
Hi Amirreza,
The state-of-the-art in time series forecasting associated with traditional statistical and machine learning methods is one of the key elements of our latest paper published in Information Sciences: https://www.researchgate.net/publication/330742498_Evaluation_of_statistical_and_machine_learning_models_for_time_series_prediction_Identifying_the_state-of-the-art_and_the_best_conditions_for_the_use_of_each_model
From a systematic review of the last decade, we expose the benefits and limitations of 11 traditional predictors (MA, SES, HES, AHW, MHW, ARIMA, SARIMA, MLP, LSTM, SVM, and kNN-TSPI) by running them on 95 datasets from synthetic and real domains. We report many lessons learned and recommendations concerning the advantages, drawbacks, and the best conditions for the use of each model.
We also performed experiments with SRN and RNN to compare them with MLP. RNN obtained a lower predictive performance than SRN and MLP. In contrast, SRN and MLP achieved comparable results, since we did not find statistically significant differences between their performances. As MLP is a more straightforward approach and has faster execution time, we prefer to keep only the results of this neural network in the paper.
Hi Antonio Rafael Sabino Parmezan
Thank you very much for kind explanations.
Actually Figure 40 of your paper is a to the point answer ! From this figure I conclude that I should go for the KNN and SVM (Because SARIMA is an stochastic method, while we are focused on machine learning methods).
You have proposed preferred methods based on the type of time series, which is a great job.
To make sure, is water and energy demand in a building, which are related to the occupant behavior,classified as an stochastic time series?
And also, once the KNN-TSPI is proposed as a good approach, could I conclude that simple KNN is also outperforming the other methods?
Thank you in advance
Hi Amirreza Heidari,
Yes, we can classify it as a stochastic time series.
There are no guarantees that a simple kNN will outperform other methods since it is variant to distortions in temporal data. For instance, a generic kNN is quite sensitive to k value, which must be determined based on data rather than a fixed given number. Choosing an appropriate similarity measure, together with a proper parameter estimation technique, are crucial decisions here.
Regards
Hi all,
Your work is mainly focused on "univariate" time series forecasting. Do you have any similar paper or comparison for "Multivariate" time series forecasting?
Hi Maryam Farzad
I can recommend these papers:
Preprint Learnings from Kaggle's Forecasting Competitions
Preprint The M5 Accuracy competition: Results, findings and conclusions
Many different TS models for univariate/multivariate data are described in these papers.
One important issue here is which metrics to use for the comparison. Let me recommend our framework: Conference Paper Data Formats and Visual Tools for Forecast Evaluation in Cyb...
Generally, due to the poor choice of error metrics, the results of existing competitions are flawed, for illustrations and recommendations on improved schemes for accuracy evaluation I recommend this chapter: Chapter Forecast Error Measures: Critical Review and Practical Recommendations
- Badges
- Science topic