Kostas -

As in any time series, something unknown could change and cause a data discontinuity at any point.

Once you decide on a method, you could go back a ways in the series and see how well you would have interpolated the following periods. This will show you what would have happened if you had done this then, if you only use data prior to that check point in developing your forecast. But other things can happen at any time.

Cheers - Jim

PS - I don't know exactly what you are doing or what's best, but you might check to see if Stata does "exponential smoothing." There are about three types of that, I think.

PSS - Oh yeah. Since you are interpolation, not just extrapolating, you could try the above forward, and then again, going backward, to the same point for each instance. This cuts down on the chance of surprise events. You might average your results. If forward and backward to a given missing case gives you very different results, you should ask what has changed to determine which might be better. But you might just average the results and make a note of it. - Just a suggestion. Maybe someone else has experience in this.

Do you know why they are missing? Is it "kids who miss the exam day" missing? Kids who left education, or is the data aggregated at a higher level?

Stata's -ipolate- command can do this for you.

*begin stata code

clear all

set obs 5

gen year = 2000 + _n - 1

gen pop = 100 in 1

replace pop = 150 in 5

ipolate pop year, gen(pop_interpolated)

*end stata code

INRE: 'limitations of interpolating missing data' - the interpolation isn't producing more 'data'. The limitations are the basics of sample size and sampling rate - 20 years of 5 year interval data is much different than fifty years worth, and especially a hundred years. The variation range, valence, fluctuations of the measured data relative to the magnitude is also a basic aspect. If all ten decades worth showed only a one to two percent only upward direction might make interpolation reasonable, if it is a sawtooth plus and minus 25% between each doesn't make interpolating very feasible. the specific 'limitations' are set by your data ( Article Quality measures for multisource statistics

Dear Kostas Simoglou.

The problem of interpolating missing data is related to the resolution of the series of the original information in terms of accuracy and time. Different results can be obtained with a high resolution of the time series combined with a low accuracy of the evaluation of the thematic data, and vice versa with a low resolution of the time series and a high accuracy of the thematic data.

Data interpolation between points is usually carried out either by function polynomials based on the general regularity of the process, or by a set of sinusoids (spectral analysis). The general limitation of spectral analysis is defined as follows: as a result of spectral analysis, harmonics whose period is greater than the length of the time interval or less than twice the minimum interval cannot be determined. In other words, the local extremes between the reference points cannot be determined by the interpolation method. The study of the possibilities of spectral analysis of geospatial data in time and space can be viewed in the work Pobedinsky G. G. Boundary conditions for the discreteness of geospatial data / / Proceedings of the congress "Great Rivers '2012". Volume 1. Nizhny Novgorod, NNGASU, 2013, p. 402-405. http://www.nngasu.ru/cooperation/2012-tom1.pdf.

Sincerely, G. G. Pobedinsky

Dear Kostas Simoglou

I think it depends on the data and the method you selected for interpolation. If your data have systematic errors, at first you need to model the sources of errors (they should have functions for doing this) to rectify the data then use the interpolation methods for the missing data. Without using such models, the interpolated data will have a systematic bias from the original signal.

For interpolation, you can use polynomial functions, spectral analysis and spline interpolation for example. polynomials are general functions that we are not allowed to use them every time. Using such functions basically depends on the problem and the model of interpolation. In spectral analysis you may miss some frequencies (signals which their period is greater than the time interval of data or less than twice the minimum interval in data) that may be in the original signals but retrieving such frequencies is impossible with these samples. Spline interpolation functions are generally better than polynomials. In this interpolation you divide the data into groups of 3, 4 or ... samples. For example, the first 4 samples of data set makes group 1, the second 4 samples makes group 2 and so on. Then fit a polynomial function to each group. In this method the error of interpolation is less than the general polynomials. because you don't use all the data for interpolation of a simple model. Once, I wrote the following code in MATLAB for spline interpolation. If you run the program you will see the model error decreases by increasing the number of samples.

clc;

clear all;

close all;

format long g;

n = input('Enter number of sampling points(>=2): ');

syms x xn xs;

x = -3:0.001:3;

y = sinc(sinh(x));

plot(x,y,'b')

title('initial function is sinc(sinh(x))');

hold on

x3 = linspace(x(1,1),x(1,end),n);

f = zeros(length(x3),1);

for i = 1:length(x3)

f(i,1) = sinc(sinh(x3(1,i)));

end

A = zeros((length(f) - 1)*4,(length(f) - 1)*4);

L = zeros((length(f) - 1)*4,1);

for i = 1:length(f)-1

L(2*i-1:2*i,1) = [f(i,1);f(i+1,1)];

A(2*i-1:2*i,4*i-3:4*i) = [1, x3(1,i), x3(1,i)^2, x3(1,i)^3;

1, x3(1,i+1), x3(1,i+1)^2, x3(1,i+1)^3];

end

for i = 1:length(f)-2

A((length(f)-1)*2+i,4*i-3:4*i+4) = [0 1 2*x3(1,i+1) 3*x3(1,i+1)^2 0 -1 -2*x3(1,i+1) -3*x3(1,i+1)^2];

end

for i = 1:length(f)-2

A((length(f)-1)*2+(length(f)-2)+i,4*i-3:4*i+4) = [0 0 2 6*x3(1,1+i) 0 0 -2 -6*x3(1,1+i)];

end

% Now I want to consider normal conditions

A((length(f)-1)*2+(length(f)-2)*2+1,1:4) = [0 0 2 6*x3(1,1)];

A((length(f)-1)*2+(length(f)-2)*2+2,(length(f)-2)*4+1:(length(f)-1)*4) = [0 0 2 6*x3(1,end)];

unknown_parameters = A^(-1)*L;

for i=1:length(f)-1

xn = x3(1,i):0.001:x3(1,1+i);

pn = unknown_parameters(4*i-3)+unknown_parameters(4*i-2).*xn+unknown_parameters(4*i-1).*xn.^2+unknown_parameters(4*i).*xn.^3;

plot(xn,pn,'r');

hold on

end

% second approach

A2 = zeros(length(f)+2,length(f)+2);

L2 = zeros(length(f)+2,1);

for i = 1:length(f)

L2(i,1) = f(i,1);

end

for i = 1:length(f)

A2(i,1:2) = [1 x3(1,i)];

end

for j = 3:length(f)+2

A2(length(f)+1,j) = 1;

A2(length(f)+2,j) = x3(1,j-2);

end

k = 3;

for i = 2:length(f)

for j = 3:k

A2(i,j) = (x3(1,i)-x3(1,j-2))^3;

end

k = k + 1;

end

unknown_parameters2 = (A2^(-1))*L2;

figure

for i = 1:length(f)-1

xs = x3(1,i):0.001:x3(1,i+1);

ps = unknown_parameters2(1,1)+unknown_parameters2(2,1).*xs;

for j = 3:i+2

ps = ps + unknown_parameters2(j,1)*(xs-x3(1,j-2)).^3;

end

plot(xs,ps,'black')

hold on

end

title('spline function')

best regards.