Hello Javed - 

Perhaps stepwise regression is still not telling you enough because of complicated variable interactions, so that you are really not using the 'best' subset of regressors.  If subject matter theory encourages you to alter the set of regressors to use, perhaps you should give those alternative models a try.  I'm just guessing here, but maybe you just need to experiment with other models (but noting that if you try enough of them, you may substantially increase your risk of spurious results.) 

Also, it sounds to me as if you may be using too many regressor variables.  Perhaps results would settle down with fewer.  Perhaps the larger set did well because it covered various contingencies, perhaps where the data should have been catagorized, but if you reduce the number of variables, you may need to reduce substantially to reach a fairly generally needed subset.  -  Perhaps the full set of regressors may be an overfit itself, and not tested enough to know it. 

Still, if you have plenty of test and/or validation data, you might try various models, plotting y on the y-axis and predicted-y on the x-axis, with various models plotted on the same scatterplot graph, and see how they compare. 

I'm not sure that this applies to your work, with which I am only vaguely aware, but perhaps it may help you develop an idea.

Cheers - Jim

Maybe you should split your data in half over the whole periods1951 to 2005 and not by time period.

Jos has a point, but note that if you consider that this could mean that the best regressors change over time, then you have a more complicated problem. 

Hi Javed,

the simplest answers to this is that the predictors you are using are either not robust and so the model is 'overfitted' to the earlier data.  Thi is a very common problem with any empirical model.  Alternatively, the data may contain non-stationary features such as trends. 

Assuming there is little autocorrelation you might deal with some non-stationarity by taking alternate years as your training data (I assume this is a version of what Jos is suggesting above) and the remaining years as validation.

Can you give a little more information on what you are trying to predict and what predictors you are using? It's very important to use a few key drivers of variability that well founded physical relationships to avoid overfitting.

Some simple but relatively successful attempts at this include these papers:

http://onlinelibrary.wiley.com/doi/10.1002/joc.2314/full

http://onlinelibrary.wiley.com/doi/10.1002/grl.50169/abstract

Best

Adam

In addition to the above . .....I don't know if these methods work for climate/atmospheric models and problems you are encountering, but in aggregate social and economic data, the same problems are often encountered.

One issue is the number of variables you may be using and the fact that they are correlated with one another.  You can get unstable estimates in this case due to factors such as multicollinearity, and in time series data, autocorrelation, and non-normally distributed variables.

One way to limit these effects if they are due to multicollinearity is, if there is an option in the statistical package you are using to set the tolerance limit for the variables.  The tolerance limit is a measure of the variance inflation factor. If this is available, check to see what the default tolerance limit is.  For example, in some programs the default  is tolerance = 0.1, which is a VIF of 10.  That may be too high, allowing highly co-linear variables to be included, creating unstable estimates.

In the case of multicollinearity, you don't have unique identifiers (the independent variables are related).  Another way to solve that problem -- if it makes sense theoretically -- is to create factors representing the pooled effect of related variables.

Another option is trying to standardize the predictors.  One way to do this is mean subtraction, which is sometimes preferred because the interpretation of the output is the same as if the variables were not standardized. You can assess whether that works by creating an estimation equation with nonstandardized variables, then run the same equation with the standardized variables and compare the outputs. What you should see, if this works, is a reduction in the standard errors of the coefficients.

Hi Javed,

using empirical orthogonal functions (eofs) of large scale fields to do downscaling needs to be done with just a few predictor eofs. 

First of all, decide how many eofs it is reasonable to use to represent your large scale variables such as geopotential height: you can do this using North's rule of thumb (you can easily find this on the web) to determine where you should truncate your eofs - this is normally about 3 for most large scale atmospheric fields, so if you are using many more than that you will likely be over-fitting the early data and therefore getting worse results in the later period. 

Once you have a limited number of eofs for geopotential height, be careful to ensure that you are not over specifying by also including too many eofs of other variables.  Rainfall for example and geopotential height are correlated due to the effects of latent heating (associated with rainfall) which will increase the geopotential height.  You can do this second step by simply checking that your principal component timeseries between one variable and another are not significantly correlated.

Reducing the number of predictors should hopefully lead to a more robust downscaling relationship.

Best regards

Adam

Before using stepwise for validation and prediction you should know that you are using TIME SERIES DATA as compared to cross section and I believe you have to check Durbin- Watson ,  Unit root , Co integration , VEC and VAR models   etc   otherwise, your findings will be  misleading

https://towardsdatascience.com/stopping-stepwise-why-stepwise-selection-is-bad-and-what-you-should-use-instead-90818b3f52df