MODELS. Assume that your two models are F(x) and G(x). The question is how to verify the statistical difference between F(X) and G(X). These models are called predictive functions or models. The data that you have, called observation set: F(Xi): fx1, fx2, ..., fxn) and G(Xi): gx1, gx2, ..., gxn), may not be put into the place of x in the F(x) and G(x). With the Xs substituted, you now have the predicted value of F(x) and G(x). Obtain the descriptive statistics of both models: mean, SD. Then test for their significant difference:

(1) IF nf = ng or sample sample size

Td = d^ / (S / sqrt(n))

d^ = (Xfi - Xgi); S = standard deviation of d^ series, and n = common sample sizes where nf = ng..

(2) IF nf ≠ ng or different sample size

Z = A / B

A = (X^f - X^g)- ( µf - µg)

B = σ (sqrt((1 / nf) + (1/ng))

REFERENCE: Gopal Kanji, 100 Statistical tests

https://www.pdfdrive.com/100-statistical-tests-e97177.html

You'd need to provide details on sample size, parameter space (parameterization, dimension, identification) and the validation strategy you have followed to obtain those estimates of measure of fit to get more meaningful answers. For example, the distributions of test statistics generally depend on degrees of freedom.

There are also strategies to decide between competing descriptions of your data that do not rely on any knowledge about these quantities, see below.

For a very general strategy, have a look at the Diebold-Mariano test which can work even if you have no little knowledge about the models you are comparing. For example when you are trying to decide between a panel of experts and an econometric model.

https://www.tandfonline.com/doi/abs/10.1080/07350015.2014.983236

There are many suitable statistics that can be used as a log-scoring rule/DM-type test, it is important to clarify for yourself what you value from an economic (I presume) perspective. For example, choosing the most likely model (maximizing expected likelihood) will most likely result in picking a different model than choosing the model that attains the smallest absolute/squared errors (which could also be defined locally, for example within a range of outcomes that are of high importance).

If you are able to calculate the AIC and are interested in a general decision criterion, you can use that too. This is essentially relying on an in-sample estimate of expected likelihood. If you read the Diebold paper above, you can see that there can be good reasons to favor such an approach over (pseudo) out-of-sample approaches. Key reasons are that deciding based on holdout results is highly dependent on the characteristic of that holdout (for example you may favor a linear model if the dynamics in the holdout are linear even though the observed data follows various regime-dynamics), and that you will still need to rely on various assumptions to take cross-validation results at face value.

Article Comments on Testing Economic Theories and the Use of Model S...

Good luck,

Hi, a critical question es whether your question is about models or about forecasts.

It seems to me that you are more interested in the accuracy of the forecasts of those models. If I am correct, a good thing to do is to evaluate those forecasts out-of-sample. Traditional and old papers discussing how to do so are the papers by Diebold and Mariano (1995): "Comparing Predictive Accuracy" and chapter 2 in the first handobook of economic forecasting. It is written by Kenneth West. The chapter is called Forecast Evaluation. Here is the link

https://www.ssc.wisc.edu/~kwest/publications/2000/Forecast%20Evaluation.pdf

Hope it helps!!

If I understand your question correctly then my answer would be both of models are almost equivalent. If you are predicting water demand in litre, then there is no significant difference between these two models. However, if you are predicting water demand at larger scale then it might be interesting depending on the scale (e.g., gigalitre).

This might be helpful: https://towardsdatascience.com/art-of-choosing-metrics-in-supervised-models-part-1-f960ae46902e