The minority class is over-sampled by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors. Depending upon the amount of over-sampling required, neighbors from the k nearest neighbors are randomly chosen. For instance, if the amount of over-sampling needed is 200%, only two neighbors from the five nearest neighbors are chosen and one sample is generated in the direction of each. Synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbor. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features. This approach effectively forces the decision region of the minority class to become more general. The synthetic examples cause the classifier to create larger and less specific decision regions. More general regions are now learned for the minority class samples rather than those being subsumed by the majority class samples around them. The effect is that decision trees generalize better.

HTH.

Dr. Samer Sarsam

Dear Dr Samer Sarsam , Thank you for the explanation. It was real helpful

First, you are fully right - directly using any (not only SMOTE) method of imbalanced data set handling, you really change your data set. To correct this situation, you can use simple approach : to use SMOTE only for Training subsets and Validation subsets don't change.

Dear Dr Sergey Porotsky thank you for the explanation. Can you kindly elaborate more.

First, please, read attached article, which compare accuracy of different methods under different imbalanced Data Sets. The methods are Random Undersampling, Random Oversampling, Cluster-based Oversampling, SMOTE, etc. One of the important conclusions, that SMOTE hasn't better accuracy, you really can use more simple method !

Second, question for you : How do you perform Cross-Validation and Test Data Set Evaluation ? There ate two possibilities:

1. To use SMOTE (or some other method - e.,g., Random Oversampling) for FULL Learning Data Set. In this case during Cross-Validation each time both current Training DS and current Validation DS will be essentially differ from original. So, obtained CV estimations will not be correct.

2. During Cross-Validation for Validation Set to use part of the original (without SMOTE using) data set. In this case your CV estimations will be correct.

By generating similar examples to the existing minority points, SMOTE creates larger and less specific decision boundaries that increase the generalization capabilities of classifiers, therefore increasing their performance.

However, SMOTE has some associated issues, such as the problem of overgeneralisation (the new synthetic examples may be generated in overlapping areas) and also the possibility of augmenting noisy regions (since no distinction between different types of minority examples is performed). Despite this, it seems that its ability to generate larger decision boundaries is still a major strength, even with its susceptibilities.

The following paper provides a detailed discussion on the intrinsic characteristics and behaviour of 15 different oversampling methods, comparing their inner procedure and how they are able to address the data complexity and improve the classification results, also highlighting their main advantages and disadvantages. The oversampling algorithms are compared across a large benchmark of 86 publicly-available datasets.

Article Cross-Validation for Imbalanced Datasets: Avoiding Overoptim...