Hi everyone,
I conducted a 4-fold cross validation using a multivariate binary logistic regression with backward stepwise elimination.
The problem is that for each training sample (225 participants out of 300), some different predictors were found statistically significant.
for example:
1st training sample: predictors 1,2,3,4 and 5 were sig.
2nd training sample: predictors 2,3,5,6,7 were sig.
3rd training sample: predictors 1,3,4,7,8 were sig.
and so on...
My question is: How do I chose the final model?
Thank you!
Oh gosh where to start. First you need at least 10 fold CV ( see Steyerburg below link)
https://www.amazon.com/Clinical-Prediction-Models-Development-Validation/dp/038777243X/ref=sr_1_fkmr0_1?keywords=steyerburg+ewout&qid=1555955826&s=books&sr=1-1-fkmr0
second stepwise is horrible in this situation-VERY unstable-see the attached paper
third usually the final model is chosen using BIC or AIC-see Steyerburg again.
Sorry I have to be so negative. Good luck, David Booth
Thank you David for replying.
I will read your suggestions.
1. I mentioned the low N (=300) because I thought 10-fold CV was too much, as each testing sample would be only 30 participants. Don't you think 30 is too low for testing? That's why I chose 4-fold.
2. I know stepwise is not perfect, but I came across several published papers that used this procedure.
3. AIC gave the same models (for all 4 folds) as stepwise with backward elimination. So, which model should I chose. What am I missing?
I would appreciate your thoughts.
Eran, Hi. 1. Most of these things are stability issues. In simulations, bacKward stepwise picks almost all possible models. This means a stepwise run tells you nothing 2. In addition to steyerberg you might want to look at Harrells course notes attached. These 2 guys are tops in this area.3. you might also take a look at another paper of ours looking at BIC and AIC. My advice is to redo your study and get something you can really have faith in. Best wishes, David