Hello colleagues,
I am planning to run a decision tree model in order to identify profiles that are associated with the binary success variable in a hackathlon. Unfortunately, my colleagues who ran the hackathlon have collected up to 200 predictors--way too many to include them in a tree model (where I would guess only 4-5 are useful, correct?).
My hunch would be to first run a logistic LASSO to identify these highly important predictors that then go into the tree model.
I am aware that a variable may turn out as relevant in the tree model that have no main relationship that could become salient in the LASSO but as this is a exploratory study and I have to come up with a practical approach, I would neverthess assume that the LASSO can identify the relevant ones.
A more finegrained issue is whether there are problems to first run the LASSO on the training data (10fold CV) and then run the tree on the same training data. At the end, both have to face the test set, where the dues are paid :) I have only begun to run machine learning models for the case I don't see the obvious :)
I would appreciate any comments or alternative recommendations
All the best,
Holger
You might have a look at the attached paper. Best wishes David Booth
On rereading your question I could have been a little more clear. Gradient Boosting and lasso are equivalent in the sense that both should give the same predictors on the same data. That's why we used gradient Boosting here to confirm our lasso results. I don't know what regression tree method you are thinking about but you might want to keep this result in mind. Best wishes David Booth
I am not very familiar with LASSO, but I know a way if you want to reduce number of features or find more useful features / predictors for your work.
1. Feature Selection Algorithms:
Algorithms like information gain, chi-square test and correlation matrix help understand which features are more correlated and by how much with the class that is to be predicted. From that list, you can take the top most 4/5 or more or less feature/predictor for use.
2. Dimension Reduction Algorithms:
There are algorithms like PCA that help reduce dimensionality thus reduce the number of features. You can use it to get more resourceful features. Unlike feature selection however, you will not be using raw features from your data, but processed ones.
In my opinion you may first use feature selection and dimension reduction algorithms to find out which features hold most promises, then take only those features and train the model. At least, that is how I perform tests.
Hello Tashreef,
I thought about at but the problem is that I intend to end up with interpretable profiles of person characteristics that are (in a nonlinearly interactive fashion) associated with the success of participating in the hackathlon. PCA creates composites that are difficult to interpret and, more importantly , difficult to use in practice (e.g., for identifying students that will likely be successful).
But I will probably give it a try :)
Best,
Holger
Holger Steinmetz thank you for your response and I understand your requirements.
I think you can give feature selection techniques a run then, to see which features are more strongly related to the class determination. After seeing your requirements, I think it will be more appropriate to try them. I personally have experience of using selection technique and reduced the number of features by 50% and got better outcome from my models.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size