I am working on building a binary logistic regression model in SPSS. I have a range of variables at my disposal to include in the model. Playing around with combinations of variables I get variations in the level of significance of some of these.
My question is, how does one go about determining which predictors to include in their model?
I thought you run the model once with all predictors, remove those that are not significant, and then rerun. But if removing only some of those predictors makes others significant, I am left wondering if I am taking the wrong approach to all this.
Is there a science to this, or is it more of an art?
Hello Tyron. Algorithmic variable selection methods tend to produce over-fitted models. Mike Babyak's article (first link below) gives a nice overview of the issue.
The second link has an "author checklist" from the Biostats Dept at Vanderbilt University. Scroll down to "Multivariable Modeling Problems", and note the comments on stepwise selection and absence of non-significant explanatory variables in a model.
HTH.
http://www.cs.vu.nl/~eliens/sg/local/theory/overfitting.pdf
http://biostat.mc.vanderbilt.edu/wiki/Main/ManuscriptChecklist
Hi Tyron,
Keeping all predictors significant is a very conservative methods which does not allow for co-operation/supporting of some less significant predictors. I would suggest using other technique for model selection. There are several different criteria allowing for model comparison (before and after predictor correction) in case of binary logistic regression. The likelihood ratio test is one of the possibilities. Other are Bayesian Information Criterion or series of Akaike Information Criteria - depending on what models are you comparing.
The procedure of model building mentioned by you is named backward selection and is recommended only in case of large sample size (to be more precise - in case of ratio between sample size and number of predictors being good enough, allowing for proper estimation of all coefficients). If your data does not fulfill that requirement the forward selection schema should be your choice.
And your last question - it happens that some of the predictors become significant after removing the other - it means they are somehow related.
Joanna
Hello Tyron. Algorithmic variable selection methods tend to produce over-fitted models. Mike Babyak's article (first link below) gives a nice overview of the issue.
The second link has an "author checklist" from the Biostats Dept at Vanderbilt University. Scroll down to "Multivariable Modeling Problems", and note the comments on stepwise selection and absence of non-significant explanatory variables in a model.
HTH.
http://www.cs.vu.nl/~eliens/sg/local/theory/overfitting.pdf
http://biostat.mc.vanderbilt.edu/wiki/Main/ManuscriptChecklist
If "But if removing only some of those predictors makes others significant, " occurs, It menas the multi-collinearity among the predictors. Do PCA analysis for continuous factors to see if there are multicolinearity among the independent factors, if some eigen-value is near zero, it is. Do contingency table analyses for categorical factors and the categorized continuous factors. if the distribution of the cell counts with many zeros, it means they are correlated.
In such a case: you may drop one of them, or define a new factor (transformation), or PCA regression, etc.
If all predictors are independent, you can use backward or stepwise, or AIC/BIC to select the 'best' model. The smaller AIC or BIC, the better.
Actually most of this is out of date. See the attachments below for more up to date information.. Best, D. Booth
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package