I am trying to run several decision tree models (CHAID, C&RT, QUEST), but I have learned that some researchers have applied logistic regression model first in order to select risk factors. So, once they have significantly factors they used them to perform a decision tree.
Is it the best way to do that? Do I really need to do this before a decision tree? If so, what are the advantages?
Thank you in advance.
To me it does not make sense to use logistic regression to screen variables for inclusion in tree analysis. Logistic regression is a parametric method, assuming a linear (and therefore monotonic) relationship between the log-odds and the continuous independent variables, while three analyses are much more flexible. I would do both (and for that matter neural networks which I have no experience of) and select the most useful model, even combine the models that make sense into an overall model.
One problem with decision trees is the complexity of learning a model with respect to the number of features (factors). For instance, in general, the run time cost to construct a balanced binary tree is O(n_{samples}n_{features}\log(n_{samples})) (see http://scikit-learn.org/stable/modules/tree.html)
Some people try to reduce the number of factors by previously running feature selection to identify which of those features/factors are more "informational" and only learn the rules from them; logistic regression would be a form of feature selection before learning the decision tree. There are other options, though, you can perform dimensionality reduction as well (PCA, ICA) or use other techniques for feature selection; but they are not obligatory to learn your decision tree model.
Interesting question. There have been several studies that compared the various approaches and the jury is still out as to the best modeling approach. I am attaching a link to an article by Peter Austin who tested various data mining approaches to logistic regression and found that logistic regression performed better than the others given the data. I wouldn't argue that this is always the case, but it certainly suggests that you may not need to go to heroic efforts with your data.
http://works.bepress.com/peter_austin/96/
Thank you very much Denis, Ariel and Ibrahim.
I am concerned that other options are justifiable but using logistic regression models first does not seem to me quite right for several reasons. The problem comes when I learn that some researchers have done it, and there are others that have done the other way around as you, Ibrahim, say. This is what disturbs me.
I will take a look at the article by Peter Austin (thanks again Ariel). Any further comments will be greatly appreciated.
Rogelio
I'd suggest another, perhaps less computation-efficient, strategy. I'd start off with a random forest (e.g., cforest of the R package party) including all predictors. Then I would use conditional variable permutation in that forest (using varimp of the party package) to see which predictors are doing most of the predictive work. Then I would cut out the non-informative ones (those that have a variable importance of 0) and run a regression with these. This regression can be fine-tuned with the MuMIn package/ or with stepAIC (if its a linear regression), which basically recombines all possible configurations of predictors and calculates AIC for these combinations. Finally, I would report on the best model that comes out of that procedure and run a conditional inference tree for that combination of predictors.
Typically, you would do about 3 models, maybe, the logistic regression, decision trees, and a neural network model.
Then you would compare the three models based on the Misclassification Matrix, the Lift, the AIC, the model accuracy, the factors that relate and seem more reasonable according to domain knowledge, the MAPE or MAE, and maybe some other factors.
Depending on the results of the model (the measures above) and the business understanding and value to the user, the best model is chosen.
The logistic regression model is sometimes preferred because it takes typically quicker to run. Secondly, it is easier to formulate. Thirdly, it not only predicts but also provides a probability of the likelihood of the prediction.
To me it does not make sense to use logistic regression to screen variables for inclusion in tree analysis. Logistic regression is a parametric method, assuming a linear (and therefore monotonic) relationship between the log-odds and the continuous independent variables, while three analyses are much more flexible. I would do both (and for that matter neural networks which I have no experience of) and select the most useful model, even combine the models that make sense into an overall model.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis