The best methodology now used, especially for small samples, is the Leave-one-out Cross validation (LOOCV).

BEST

The two best sources I know of are:

https://www.amazon.com/Clinical-Prediction-Models-Development-Validation/dp/3030163989/ref=sr_1_fkmr0_1?keywords=Steyerburg+clinical+prediction+models&qid=1576169474&s=books&sr=1-1-fkmr0

and the attached notes. Best wishes, David Booth

1-st of all, the fundamental assumption of any forecasting technique (implicit or explicit) is that time series represents a stable pattern that can be identified and then extended into the future. If a pattern of the past data-points (swine population) is not statistically stable, then no meaningful future prediction (forecasting) is possible regardless of the sophistication of the forecasting technique.

2-nd, you use data of annual swine population from 1937 to 2017.

It seems reasonable to assume that too ‘old’ data-points do not practically affect (correlated to) the most recent data-points, let alone the future data.

The data-points that are strongly correlated to the newer ones can be used for making the forecast. The data-points that are weakly correlated to the newer ones (or not correlated at all) should not be included for forecasting; otherwise, the forecast will likely be skewed. By including 1937 and following population data in the training set, do you believe that, say, 1937 population might affect 2018 population??!

Thus, the use of a large number of the available past population data-points worth of about 80 (!!) years is not needed for making a meaningful forecast; in fact, the large number of the past data-points can be detrimental to producing a reasonably accurate forecast.

The maximum number of ‘steps back to the past’, at which the older data-points are still strongly enough correlated to the newer data-points can be estimated using an autocorrelation function (ACF) of a time series .

It is usually accepted that a strong enough correlation coefficients start with 0.6 or greater. Therefore it can be assumed that the number of the past data-points used for the purpose of forecasting (a cut-off value of a time-lag) corresponds to the correlation coefficient of about 0.6 of the autocorrelation function of the census time series (other cut-off metrics also exist).

3-rd, if you want to forecast the last 3 years data 2018-2020, then your validation set should be about this size, about 3-4 last data points, i.e. 2014-2017. These data points must not be seen by the training set which should be from cut-off correlation of the ACF up to 2013. (Notice that you used training set from 1937 to 2017 to validate 2012-2017- this is not correct, your training set should not have overlapped with your validation set).

Now the model that best forecast 2014-2017 should be used for forecasting the future 2018-2020. This is your best bet...... The mean forecasting error % can be defined as the mean absolute difference between the actual and predicted population divided by the actual known population.

Dear Dr Alexander, thank you very much for your informative answer and it helped me a lot. As you said, the test set has to be started from somewhere in the past and ended up in 2011 (validation set is from 2012-2017). In this time series, there are 81 points. Hence, I have selected 80 lags to see the auto-correlation in order to workout the minimum number of past-data-points required to make a meaningful forecast. The ACF correlogram is attached. I shall be much thankful if you could kindly help me to workout the maximum number of steps-back-to-the-past for the said purpose. Please note that I am going to make forecasting using exponential smoothing methods.

Dilan Dissanayake Your ACF correlogram indicates that your data points are rather weakly correlated to each other. ACF data decay fast with time lag. Most oscillating ACF data points are within the confidence limits band. Therefore I would use and try not more than 4 or 5 latest data points. Including the older data points in the training set will not help to make a more accurate forecast for 2018-2020 because they seem do not influence the population in 2018-2020..

In my work forecasting patient census I used recursive digital filtering forecasting technique- see attached file along with examples of time series, their ACF, and forecasting errors.

Now, if you use a simple exponential smoothing, the unknown parameter can be estimated by minimizing the SSE [sum of the squared errors] within your training set. In double exponential smoothing you add one more smoothing factor to control the decay of the influence of the change in trend called beta (b). Double Exponential Smoothing with an additive trend is classically referred to as Holt’s linear trend model: Additive Trend: Double Exponential Smoothing with a linear trend. Multiplicative Trend: Double Exponential Smoothing with an exponential trend.

Triple Exponential Smoothing is an extension of Exponential Smoothing that explicitly adds seasonality to the univariate time series.

This method is sometimes called Holt-Winters Exponential Smoothing.

It includes one more new parameter called gamma (g) that controls the influence on the seasonal component. Triple exponential smoothing is the most advanced variation of exponential smoothing and through configuration, it can also converts into double and single exponential smoothing models. I am not sure which exponential method you are going to use.

I would start with the simplest one-parameter exponential smoothing taking into account that the number of the past data points in this case will be low.

I am really so grateful for this informative answer. I hope I can do it now.

Forecasting theory affirms that exponential smoothing cannot provide good forecasts for time series with oscillating ACF.

�