You are likely to be suffering from the Texas sharp shooter problem - that is drawing the target on the barn door after firing the gun at it. It is highly likely that you have fitted the model to the data too well - modelling idiosyncrasies in the data you have that will not generalize out of that specific sample; this is known as over-fitting. One way to tackle this is random cross validation - fitting the data multiple times to a sample of your data and seeing how it predicts the non-included holdout sample. 

see

http://robjhyndman.com/hyndsight/crossvalidation/

and

http://scikit-learn.org/stable/modules/cross_validation.html

Subset selection itself is not stable as Breiman showed in the 1990s. See his paper, as well as Austin and Tu's.Both do stability studies.. Both are cited and reviewed in the attached. Best wishes.

Thank you professor Booth.

You are likely to be suffering from the Texas sharp shooter problem - that is drawing the target on the barn door after firing the gun at it. It is highly likely that you have fitted the model to the data too well - modelling idiosyncrasies in the data you have that will not generalize out of that specific sample; this is known as over-fitting. One way to tackle this is random cross validation - fitting the data multiple times to a sample of your data and seeing how it predicts the non-included holdout sample. 

see

http://robjhyndman.com/hyndsight/crossvalidation/

and

http://scikit-learn.org/stable/modules/cross_validation.html

@ Drs. Kelvyn and Booth. In fact, I'm doing the cross-validation (10 folds or LOOCV) to select the final model with comparing the models performance (MSE and adjusted R squared), but I did croos-validation only one time. I'll do that multiple times though. (I'm doing stepwise regression, best subset selection, robust regressing, regularized regression, principal component regression, and partial least square).

These are the steps that I'm following:

On the other hand, I was thinking whether or not the coefficients are always stable in different models with different number of variables. In this case I may be able to use this fact as another feedback for variable selection beside stepwise or cross-validation.

Thanks for your feedback.

First, I'd forget stepwise and all subsets regression because neither are stable. See the above paper.  If I were you I would just try lasso or elastic net regression. See the links below for code. If you have categorical variables, you will need group lasso. See the above paper for references. You can cross-validate these as described in Harrell Jr, Frank, Regression Modeling Strategies, Springer. Harrell developed R code for all of these cross-validations and the book tells you about them. I agree with Kelvyn that you are probably overfitting currently. Best wishes and I hope this helps a little. BTW I'd only use robust regression to look for outliers in the data. If you have some remove them and see if you can find their story separately. Then continue with the model building.  Also don't forget to look at graphs.

http://www.amazon.com/Everyone-Advanced-Analytics-Graphics-Addison-Wesley/dp/0321888030/ref=sr_1_1?s=books&ie=UTF8&qid=1458095283&sr=1-1&keywords=jared+lander

http://www4.stat.ncsu.edu/~boos/var.select/lasso.adaptive.html

Thank you Dr. Booth. It would be appreciated if you could kindly let me know your advice about the following questions.

@ Dr. Jones, It would be appreciated if you could kindly explain me how come I'm suffering from the Texas sharp shooter problem. Thank you.

Hi Amir,

1. Roughly overfitting means that your prediction equation fits the training data almost perfectly but does not predict the test data very well.  See Harrell, Frank, Regression Modeling Strategies, Springer for details. They are in the cross-validation sections I believe.

2. Measurement error isn't the only source of outliers. In general, you get an outlier if it comes from a different population than the majority of the data elements or is just one of the low probability points of the distribution of the rest of the data.. Since it probably belongs to a different distribution than the one you are modeling, it should be treated separately, which is essentially what robust regression does.

3. No I don't. That's why I made the suggestion from number 2.

Hope that I've helped a little. Feel free to contact me if you think I can help.. Best wishes.

Dear Amir, have you tried to create the betas using the matrix definition:

b=(X'X)-1X'y  

b=solve(t(X)%*%X))%*%t(X)%*%y  #in R

just to check the determinant of matrix (X'X)?

Re Texas Sharpshooter - ie over-fitting

This is a gut feeling based on experience- you have a model with 15 estimated parameters and while nature is complex, I do not think it is generally that complex. And therefore I think you are likely to be modeling un-replicable noise and not consistent signal. To be convinced I would need strong well developed theory about why each should be included and backed up by the empirical work.

If you plot up the R squared of the cross validated hold-out sample  against the number of splits(that is model complexity) you often get a U shaped plot and there is a point beyond which (the bottom of the U)  where increasing the number of terms (ie complexity)  results in a worse prediction to the hold out sample - this suggests you are modeling noise. My experience informs me that this is usually less than 15 parameters.    

I think you are trying to imitate what is done in glmnet package in R. There are different coefficient profile plots in that package which are similar in spirit I guess what you want to do. They are for penalized regression. I am not sure if you can exploit that package to get profile plots you want.

Note: I did not read all the previous comments above.

Dear All

i have problem 150 regression models for 110 kind of industry and i am going to use F-Test for coefficient stability . This test is known as the F-test for coefficient stability. An important maintained assumption in this analysis is that the error variance of ut in the two-sub-samples is equal. If this is not the case then the pooled regression equation will be heteroscedastic and the test will be invalid. the question is that which package in R will suitable for this kind of test?

Farhad,

Can you explain a bit about your regression models? How did you get these 150 competing regression models? What are you trying to measure, what is the dependent variable? Is industry one of your variable? How many observations do you have?