Handling imbalanced datasets in machine learning is a difficult challenge, and can include topics such as payment fraud, diagnosing cancer or disease, and even cybersecurity attacks. How to improve massively imbalanced datasets in machine learning with synthetic data?
https://imbalanced-learn.readthedocs.io/en/stable/
I have used SMOTE to synthesize data.
Dewan Azmal Hossain ,
As Shahir Asfahan indicated, SMOTE is one possibility.
There are many recent variations of such a technique.
Moreover, you could also try undersampling techniques, such as RUSBoost.
I have successfully employed it in recent works for imbalanced classification:
Article Ensemble learning by means of a multi-objective optimization...
Conference Paper Monitoring of drinking-water quality by means of a multi-obj...
Finally, you could also combine oversampling and undersampling techniques.
A good overview of imbalanced learning can be found here:
Article Learning from imbalanced data: Open challenges and future directions
Handling imbalanced datasets in machine learning is a difficult challenge, and can include topics such as payment fraud, diagnosing cancer or disease, and even cyber security attacks. What all of these have in common are that only a very small percentage of the overall transactions are actually fraud, and those are the ones that we really care about detecting.
In this post, we will boost accuracy on a popular Kaggle fraud dataset by training a generative synthetic data model to create additional fraudulent records. Uniquely, this model will incorporate features from both fraudulent records and their nearest neighbors, which are labeled as non-fraudulent but are close enough to the fraudulent records to be a little “shady”.
See link below
https://towardsdatascience.com/improving-massively-imbalanced-datasets-in-machine-learning-with-synthetic-data-7dd3d856bbdf
The common way to improve Balancing Datasets is SMOTE.
There are a number of approaches to addressing class imbalance and increase sensitivity to the minority class:
- synthesis of new minority class instances
- over-sampling of minority class
- under-sampling of the majority class
- combination of under- and over-sampling
- adjust the cost function to make misclassification of the minority instances more important that misclassification of majority instances
I suggest you follow https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html
https://www.aaai.org/Papers/Workshops/2000/WS-00-05/WS00-05-001.pdf
I also used SMOTE in one of my works. You can try that technique.
Jason Brownlee provided a step-by-step framework for imbalanced classification projects on his website:
https://machinelearningmastery.com/framework-for-imbalanced-classification-projects/
- Badges
- Science topic