In the case of imbalanced classified data, oversampling is a standard technique to avoid the learner to be biased toward the most represented class. In the case of cross-validation, we have two choices: 1) perform oversampling before executing cross-validation; 2) perform oversampling during cross-validation, i.e. for each fold, oversampling is performed before training, and this process is repeated for each fold. Case 1) is more efficient but model selection is done on the basis of the average performance on data including artificial data; case 2) is less performant but includes oversampling as a part of the model selection process. Which approach is most suitable?
In Case 1), as the entire original data is oversampled and the cross-validation is performed afterwards, similar examples (or exact replicas, depending on the oversampling algorithm applied) may appear in both the training and test partitions. Therefore, Case 1) seems more “efficient” because we are over-estimating the classification performance. This is known as the issue of “overoptimism”, introduced by a faulty CV design.
Case 2) is the correct way of handling imbalanced data, although proper care should be taken when choosing an oversampling algorithm (e.g. choosing algorithms that create exact replicas of original examples may lead to overfitting).
The following paper may be useful, as it compares Case 1) versus Case 2) (i.e. CV before oversampling and CV during oversampling) in what concerns the obtained performance results and associated data complexity measures in both the training and test partitions.
Article Cross-Validation for Imbalanced Datasets: Avoiding Overoptim...
Furthermore, the issues of overoptimism and overfitting are explained in detail and 15 well-known oversampling algorithms are compared across a benchmark of 86 publicly-available datasets.
Hopefully this helps future researchers in the field.
Best regards!
Many works prefer oversampling during cross-validation. I attach a link where you can find an explanation for this choice:
http://www.marcoaltini.com/blog/dealing-with-imbalanced-data-undersampling-oversampling-and-proper-cross-validation
I think that it is better to avoid oversampling of (under)sampling. CV is still (under)sampling. In this way oversampling takes into account more information.
What about undersampling of the majority class? You can still perform ensemble learning afterwards (see the enclosed publication - Fig 1).
Article Integrating microRNA target predictions for the discovery of...
Oversampling with (SMOTE) can be done before training. However there is a catch. SMOTE reduces the majority class as well hence there is a chance of loosing valuable information.
My approach is as follows: ( this has been tremendously successful in most of the work I have done).
I tend to impute only minority class using SMOTE and then add that back to the original data. This helps to improve the prior probability of the minority class and during posterior estimation using any machine learning model helps to get a better decision boundary with confidence.
I call it "Targeted SMOTE" as I take only the minority class imputation from SMOTE and add it back to the original data.
Now you can divide this set for training and testing and CV as well , it would work just fine.
Try to check the performance of this model on a holdout sample data without SMOTE which you must keep aside before SMOTE to do an acid test. I am sure you will find fantastic results.
In Case 1), as the entire original data is oversampled and the cross-validation is performed afterwards, similar examples (or exact replicas, depending on the oversampling algorithm applied) may appear in both the training and test partitions. Therefore, Case 1) seems more “efficient” because we are over-estimating the classification performance. This is known as the issue of “overoptimism”, introduced by a faulty CV design.
Case 2) is the correct way of handling imbalanced data, although proper care should be taken when choosing an oversampling algorithm (e.g. choosing algorithms that create exact replicas of original examples may lead to overfitting).
The following paper may be useful, as it compares Case 1) versus Case 2) (i.e. CV before oversampling and CV during oversampling) in what concerns the obtained performance results and associated data complexity measures in both the training and test partitions.
Article Cross-Validation for Imbalanced Datasets: Avoiding Overoptim...
Furthermore, the issues of overoptimism and overfitting are explained in detail and 15 well-known oversampling algorithms are compared across a benchmark of 86 publicly-available datasets.
Hopefully this helps future researchers in the field.
Best regards!
Kumarjit Pathak Please suggest some references using this approach. Thanks
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling