Hello,
I am running a logistic regression, random forest, and support vector machine models to predict whether a loan will default. My data is highly imbalanced, which I have addressed using resampling techniques (Random Over Sampler). I am getting unusually high ROC-AUC (very close to 1) scores that do not match the results of similar studies. I have already addressed variables with high correlations. I am at a slight loss of what to try next, and was looking for some guidance.
Update: I removed some variables and there were significant improvements in the Logistic Regression and LinearSVC models. However, the Random Forest is still returning scores equal to 1 for all metrics (accuracy, precision, recall, AUC, and F1). Any suggestions to address the Random Forest?
Thanks!
You've addressed data imbalance using Random Over Sampler, but the unusually high ROC-AUC scores might be due to issues with evaluation, data leakage, or model complexity. Consider cross-validation, different evaluation metrics (precision, recall, F1-score), hyperparameter tuning, alternative resampling techniques (SMOTE, ADASYN), ensemble methods, and consulting domain experts. Check for outliers, data quality, and external validation. Simplifying the model and exploring different resampling methods could also help.
Hi Maya,
What part of the data do you use to train the models and what part do you use to compute the ROC AUC?
Maya Dmello
ROC plot can be deceiving in getting a clear picture of model predictability, particularly for imbalanced data. I got same issue in one of my research projects [1]. Moreover, previous studies have stated the visual interpretability of ROC plots might mislead in drawing reliable conclusions on the classifcation performance [2]. The PR plot can overcome this shortcoming of the ROC plot. Instead of making a tradeoff between the true positive rate and the false positive rate as in the ROC plot, the PR plot considers the fraction of the actual positives over all positive predictions.
[1] X. Zhang, M.Z. Akber, W. Zheng, Predicting the slump of industrially produced concrete using machine learning: A multiclass classification approach, Journal of Building Engineering. 58 (2022) 104997. https://doi.org/10.1016/j.jobe.2022.104997.
[2] [1] T. Saito, M. Rehmsmeier, The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLoS One. 10 (2015) e0118432. https://doi.org/10.1371/journal.pone.0118432.
@maya thanks for clarifying: your last statement is confusing to me because you have to make sure that at ch step of data splitting only thr left-out-set of data is used for prediction (eg you split data so that 80% of data is used to train the model and the remaining 20% is used for prediction so the model *should not* be trained and tested on the same data (as per your last statement)
In general you might want to replicate the process 100 times or more so that you have different validation sets (inner cross validation loops with 80% data (for instance) and outer cross validation with the remaining 20%) so that you can obtain the sampling distribution of AucROC/AucPR accounting for both sampling error/model building/monte carlo variation
Making sure that the splits are stratified according to the outcome
You don't specify how imbalanced your class (or how big your sample is) is but generally speaking Precision-Recall curves are more appropriate with class imbalance
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude