I'm currently working on a topic which aims at designing deep learning model to do short-term forecasting of time series. The data is collected from telecom equipments which is non-stationary and highly stochastic. When I try to compare the performance of different models as LSTM, Transformer and the others, there is a problem of the result.
In general, the time series is quite difficult to forecast, and if I check MAE and MSE, the difference of different models are very small. For example, the MSE of LSTM is 0.282 +/- 0.14. If I use Transformer, the MSE could be 0.273 +/- 0.12. Even though Transformer has the lower mean value of MSE, but by considering the standard deviation of MSE is quite large, the result is not statistically significant. In this case, how can I evaluate the performance of different models by considering the performance improvement is always much lower than the standard deviation of the loss.
Dear Shuyang Li
These articles might be an asset, have a look:
1. https://res.mdpi.com/d_attachment/data/data-06-00055/article_deploy/data-06-00055.pdf
2. https://towardsdatascience.com/an-overview-of-time-series-forecasting-models-a2fa7a358fcb
3. https://machinelearningmastery.com/xgboost-for-time-series-forecasting/
4. Article Prediction of hierarchical time series using structured regu...
5. Article Evaluation of stochastic and artificial intelligence models ...
Kind Regards
Qamar Ul Islam
Dear Zaher Mundher Yaseen and Qamar Ul Islam,
Thanks for the papers, I will check them : )
Based on traditional statistics, all data are not suppose to be stationary.
If you have a stationarity test, please do and make sure you make it stationary before modelling.
Dear Vikas Ramachandra and Olawale Abimbola,
I did not run tests for non stationarity, and I will do the tests further. But as I know about the data, the data was collected from base stations, and it has strong seasonality and trend due to the behavior of mobile users. In general, I think they are non stationarity.
Hello
A way to deal when you have several models that have skill, acceptable MSE or MAE, but high variance, is to use all your models in an ensemble. One of the most basic form of model ensemble is to perform the average of the outputs of all your models.
The most diverse your models are the best chance to effectively decrease the final variance of your combined model ensemble. One way to think about model diversity is the kind of model, LSTM, CNN... and the other way is to evaluate the correlation of the outputs of the different models. Two models that have highly correlated outputs means that both miss or hit in similar points and the contribution to the ensemble is poorer than if the outputs are not correlated; "the models are able to correct each other". The last point to highlight regarding ensembling is that normally the most models you have better for the final performance.
I hope I have shed some light on the matter.
Regards.
Dear Jose Manuel Vericat
Thanks for your reply, yes, we have tried the ensemble model which is slightly better than using a single model. For now, we want to create a new model which can greatly outperform the baselines. In this case, it's essential to know which architecture/mechanism is more beneficial for this task. If the variance is high, it's meaningless to compare the performance of different models by considering the difference between their results is not statistically significant.
Se tiene que hacer un análisis comparativo de los modelos para obtener el mejor y con aprendizaje profundo mientras más lo usen mejores respuestas tendrán y si puede ir mejorando sus algoritmos y por ende los modelos mejor hasta lograr lo que se quiere
There is always the "old school" answer of Statistical Process Control . . .
- Badges
- Science topic