Dear Colleagues I hope all is well. I have a classification project, the event outcome is imbalanced (25% of the subjects having the event) The data has almost 90 variables that become 250 after doing one hot encoding I tried oversampling technique and the accuracy, sensitivity are really very well in the oversampled training data (just excellent, with accuracy up to 95%). However, it is very bad, on the validation set (almost 25%). This happens when using Logistic regression, random Forest, decision tree, XGBOOST, gradient boost and Bagging. I wonder if this would be related to the big number of features (250 features?) Shall I do Recursive feature elimination Random Forest before running all these models on the oversampled data? Would this make a difference? Or Recursive feature elimination Random Forest is only used at the end to get a simplified prediction model?
Hey Hatem,
You could make the feature selection, but I believe it won't significantly affect the model performance. I recommend measuring the F1 and roc auc scores too for the validation dataset. Oversampling occurs at what stage?
Hi Hatem,
this is really a strange situation and I am interested to know the cause of this problem once you solve it. Coming back to your question, I would recommend the BorutaShap method (based on Boruta algorithm and Shap values) to select the most relevant features/predictors for your classification task. It is available in Python and can also be easily adapted to your application. I would also suggest doing k-fold cross-validation to be sure it is not a matter of random effect in your data or parameters. Good luck!
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification