When comparing two time series forecasting models, there are many statistics, e.g., R2, RMSE, Nash efficiency, etc. However, they are all for deterministic models, when considering the uncertainty of each model, what are the statistics that are available?
Dear Yang,
The answer depends whether the forecasting models you are comparing include means to represent uncertainty.
For instance, a Kalman Filter can estimate a predictive Gaussian distribution in timeseries forecasting applications. Comparing the uncertainty of two predictive models embedded into such filters is then a matter of computing e.g. the total variance of the resulting predictive distributions and see which model reaches lower variance.
Other class of forecasters that explicitly model uncertainty are interval-based ones. In this case, the prediction output is a lower and upper bound of some real-valued quantity and uncertainty is generally assumed to be represented by a uniform distribution over such intervals. Again, you can measure the quantities such as the lenght of the intervals to see which from two models are more confident.
Now, if the forecasting models do not include means to represent uncertainty and can only output, say, a real value representing their best guess for an unknown future quantity, then measuring uncertainty can be challenging.
The most obvious way would be to look at the series of historical prediction errors and compute the variance of the errors. If it is true that there is to some extent an implicit trade-off between accuracy and uncertainty in your application, then you can expect that a more confident forecasting model would achieve a lower predictive error variance and vice-versa, regardless of the error mean.
Those are my intuitive observations and unfortunately I cannot help much more than this without looking at the timeseries forecasting literature to see how it is done.
I hope those can be valid starting points, though.
Best,
Carlos.
Dear Yang,
The answer depends whether the forecasting models you are comparing include means to represent uncertainty.
For instance, a Kalman Filter can estimate a predictive Gaussian distribution in timeseries forecasting applications. Comparing the uncertainty of two predictive models embedded into such filters is then a matter of computing e.g. the total variance of the resulting predictive distributions and see which model reaches lower variance.
Other class of forecasters that explicitly model uncertainty are interval-based ones. In this case, the prediction output is a lower and upper bound of some real-valued quantity and uncertainty is generally assumed to be represented by a uniform distribution over such intervals. Again, you can measure the quantities such as the lenght of the intervals to see which from two models are more confident.
Now, if the forecasting models do not include means to represent uncertainty and can only output, say, a real value representing their best guess for an unknown future quantity, then measuring uncertainty can be challenging.
The most obvious way would be to look at the series of historical prediction errors and compute the variance of the errors. If it is true that there is to some extent an implicit trade-off between accuracy and uncertainty in your application, then you can expect that a more confident forecasting model would achieve a lower predictive error variance and vice-versa, regardless of the error mean.
Those are my intuitive observations and unfortunately I cannot help much more than this without looking at the timeseries forecasting literature to see how it is done.
I hope those can be valid starting points, though.
Best,
Carlos.
FPE might be investigated for what you have in mind. However the AIC criterion (and the theory behind) might be a better starting point, given its asymptotic properties (asympt. efficiency) and the so called "concistency of the criteria". In essence, a model with a lower rmse is likely to inject more uncertainty than a simpler model with a worse rmse. AIC should be able to detect such a situation. Model complexity is certainly something you want to look at in order to assess the degree of uncertainty. Finally, you might corroborate your analysis by using ad hoc distances designed for the used model. I hope it helps.
One way to do it is outlined in this patent: "Forecast model quality index for computer storage capacity planning". I am not sure who owns it now, though; I think BofA.
https://www.google.com/patents/US7788127?dq=ininventor:%22Alexander+Gilgur%22&hl=en&sa=X&ei=_0AsVJPUB4bdoATF0YD4Aw&ved=0CCYQ6AEwAQ
If I understand well, you are interested to know whether the model error depicts realistic or physically plausible weather variations? Still it is unclear what aspect of the weather model you'd like to evaluate? Suppose that you had a set of geographically distributed precipitation observations (e.g., hourly or daily) then you might collocate them with both models in time and space. You then could evaluate whether the distribution (PDF) of gauged rain at the set of locations of the gauges and at the given time accumulation is better represented by one model or the other. Of course, a similar approach may be followed for other poorly determined model variables, depending on your objectives.
Dear Yang pan,
You should read the article I wrote on Forecast evaluation using Stata and pay particular attention to the different techniques identified as well as the Diebold-Mariano test. If you use Stata, the program I wrote will do the forecast evaluation for you. This paper is avaiilable on Research Gate as well as in the proceedings of the 2010 Stata U.K. users group conference in London.
I hope this helps,
Bob Yaffee
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis