A multiple Binary Logistic Regression produced a model with 100 IVs, of which 30 are statistically significant predictors for my dichotomous DV of interest.
If I take those 30 IVs and add them to a second model (same DV of course), then a 5-6 more stop being significant anymore.
(1) Is this manual approach similar to backward elimination and (2) Should I report the ORs of the first or the second model?
Dimitrios,
Yes, this appears to be somewhat parallel with what is usually labeled a "backward" selection, except in that approach, the exclusion criterion is reexamined after each variable is omitted from the model, as whatever "contribution" index you might use (here p-values), will change as you omit a variable.
Variables that are no longer significant in your second model is evidence of the problem with this method, but as well, all stepwise methods. More can be spelled out, but the difficulty is that if such a method is used, one is considering univariate indices for a multivariate problem.
Therefore in my arena, such methods are frowned upon. As far as I can tell, more has been written discouraging this practice in OLS regression and discriminant analysis. The theoretical reasons, are, however, identical for LR, but if there are others who know of similar treatment in LR, please add it to this thread.
There are others, but a couple of citations that you might find interesting are:
Huberty, C. J (1989). Problems with stepwise methods: Better alternatives. In B. Thompson (Ed.) Advances in social science methodology (Vol. 1) (pp. 43-70). Greenwich, CT: JAI Press.
Thompson, B. (1995). Stepwise regression and discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4), 525-534.
As well, if classification accuracy is the criterion of interest, this practice is condemned in both editions of Carl Huberty’s Discriminant Analysis books.
I wrote a similar answer to a similar question on RG today, which brings to mind the exact tile of another citation:
Thompson, B. (2018). Why won't stepwise methods die?, Measurement and Evaluation in Counseling and Development, 21(4), 146-148.
On the other hand, if the goal is classification accuracy, one can often obtain greater accuracy with a reduced set of variables, in fact, this is almost always the case – see:
Article Achieving Accurate Prediction Models: Less is Almost Always More
The point herein, is that this is not the way to do it; the function of the model is multivariate, but by making univariate decisions, you miss the big picture. The alternative of considering the cross-validated performance of all 2p-1 combinations of variables, in your case is painful as such algorithms are not available in standard statistical packages, leading to special purpose programs, as in: (however unless you reduce the number of predictors, this is not only painful, but impossible)
Article Selecting Predictor Variables in Two-Group Classification Problems
I suggest first trying to reduce the number of predictors from the enormous "100" that you mention. It would be surprising if an predictor dimensionality of 100 is actually necessary. Consider reduction by expertise, and perhaps PCA or a common factor method, as well PLS.
My best to your research,
John
Dimitrios,
Yes, this appears to be somewhat parallel with what is usually labeled a "backward" selection, except in that approach, the exclusion criterion is reexamined after each variable is omitted from the model, as whatever "contribution" index you might use (here p-values), will change as you omit a variable.
Variables that are no longer significant in your second model is evidence of the problem with this method, but as well, all stepwise methods. More can be spelled out, but the difficulty is that if such a method is used, one is considering univariate indices for a multivariate problem.
Therefore in my arena, such methods are frowned upon. As far as I can tell, more has been written discouraging this practice in OLS regression and discriminant analysis. The theoretical reasons, are, however, identical for LR, but if there are others who know of similar treatment in LR, please add it to this thread.
There are others, but a couple of citations that you might find interesting are:
Huberty, C. J (1989). Problems with stepwise methods: Better alternatives. In B. Thompson (Ed.) Advances in social science methodology (Vol. 1) (pp. 43-70). Greenwich, CT: JAI Press.
Thompson, B. (1995). Stepwise regression and discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4), 525-534.
As well, if classification accuracy is the criterion of interest, this practice is condemned in both editions of Carl Huberty’s Discriminant Analysis books.
I wrote a similar answer to a similar question on RG today, which brings to mind the exact tile of another citation:
Thompson, B. (2018). Why won't stepwise methods die?, Measurement and Evaluation in Counseling and Development, 21(4), 146-148.
On the other hand, if the goal is classification accuracy, one can often obtain greater accuracy with a reduced set of variables, in fact, this is almost always the case – see:
Article Achieving Accurate Prediction Models: Less is Almost Always More
The point herein, is that this is not the way to do it; the function of the model is multivariate, but by making univariate decisions, you miss the big picture. The alternative of considering the cross-validated performance of all 2p-1 combinations of variables, in your case is painful as such algorithms are not available in standard statistical packages, leading to special purpose programs, as in: (however unless you reduce the number of predictors, this is not only painful, but impossible)
Article Selecting Predictor Variables in Two-Group Classification Problems
I suggest first trying to reduce the number of predictors from the enormous "100" that you mention. It would be surprising if an predictor dimensionality of 100 is actually necessary. Consider reduction by expertise, and perhaps PCA or a common factor method, as well PLS.
My best to your research,
John
It's a pity that such exceptional answers can only be recommended once...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression