Hello my friend,

It is important to formulate a statistical analysis plan prior to performing the statistical analyses if you will not have access to the data again. Typically backwards selection comes up with a more resilient and inclusive but most researchers (will all of them, just not all of them do it) suggest to not rely on either selection method as these selection methods are based merely on statistical criterion and have no logical significance attached to them, Instead it can be better to only include the variables in which there is a theoretical connection with the depended variables.

Regardless, the reason for the difference is because of the algorithms used. For forward selection there are of course no variables included when the process starts and with backwards every variable is included. Then as you most likely at aware, each variable is added or subtracted based off of the changes to the other variables regarding their statistical significance followed by the addition of the next variable in which the same process happens. Starting with no vs. all variables  will most likely result in different end models.

In trying to pick the variables, lets randomly say you testing two different models for predicting erectile dysfunction: in the first model one (in which backwards selection was used) of the variables kept is testosterone with a p-value of 0.043 thus you consider it statistically significant and another variable obesity get left out due to a p-value of 0.059. However, in the forwards selection model testosterone gets the boot (p-value = 0.061) where as obesity just barely squeaks by a p-value of 0.049 and several other variables (astrological sign, p=0.042; and hair color, p=0.049) get added.

I am sure this all seems very convoluted... however the solution is very clear, both obesity and testosterone are known from prior research to play a role in the development of erectile dysfunction however, astrological sign and hair color do not play a role. Therefore, the example I have provided shows the importance of deciding the variables on more than the p-value which is what forward and backward selection does.

To check the difference in the models you would need to run both of them and look at the Akaike's Information Criterion (AIC) test or any of the other tests that attempt to aid in model selection. The preferred model would be the one with the lowest value.  I am sorry for the long winded example and then the brief blip on model selection statistics but I need to be going.

Wishing you the very best ,

Logan Netzer 

Feel free to ask further questions if I was too convoluted

Hello my friend,

It is important to formulate a statistical analysis plan prior to performing the statistical analyses if you will not have access to the data again. Typically backwards selection comes up with a more resilient and inclusive but most researchers (will all of them, just not all of them do it) suggest to not rely on either selection method as these selection methods are based merely on statistical criterion and have no logical significance attached to them, Instead it can be better to only include the variables in which there is a theoretical connection with the depended variables.

Regardless, the reason for the difference is because of the algorithms used. For forward selection there are of course no variables included when the process starts and with backwards every variable is included. Then as you most likely at aware, each variable is added or subtracted based off of the changes to the other variables regarding their statistical significance followed by the addition of the next variable in which the same process happens. Starting with no vs. all variables  will most likely result in different end models.

In trying to pick the variables, lets randomly say you testing two different models for predicting erectile dysfunction: in the first model one (in which backwards selection was used) of the variables kept is testosterone with a p-value of 0.043 thus you consider it statistically significant and another variable obesity get left out due to a p-value of 0.059. However, in the forwards selection model testosterone gets the boot (p-value = 0.061) where as obesity just barely squeaks by a p-value of 0.049 and several other variables (astrological sign, p=0.042; and hair color, p=0.049) get added.

I am sure this all seems very convoluted... however the solution is very clear, both obesity and testosterone are known from prior research to play a role in the development of erectile dysfunction however, astrological sign and hair color do not play a role. Therefore, the example I have provided shows the importance of deciding the variables on more than the p-value which is what forward and backward selection does.

To check the difference in the models you would need to run both of them and look at the Akaike's Information Criterion (AIC) test or any of the other tests that attempt to aid in model selection. The preferred model would be the one with the lowest value.  I am sorry for the long winded example and then the brief blip on model selection statistics but I need to be going.

Wishing you the very best ,

Logan Netzer 

Feel free to ask further questions if I was too convoluted

Thanks Logan this is so helpful. 

I had worked on many models & have used all the model selection techniques - forward, backward & stepwise and well versed how these models work however was not aware that forward and backward may give us some different predictor variables. 

I have used AIC, AUC, ROC & KS statistics to get the model fit. 

In my case i still have the opportunity to include one var at a time and then measure AIC or Bayesian information criterion because i only have around 10 predictor variables. 

But even if in case i select the forward selection in my case which has lower AIC value how would the predict function work as the backward has one extra variable which is not with forward selection. 

I hope i am able to make my case clear. 

Thanks again for your valuable help. 

In the end, you have to use some judgement as to what is the best model.  You could test interaction terms, and if necessary performing separate analyses.  E.g. if gender has a lot of significant interaction terms, then you might want to conduct separate analyses for men and women.