I have a dataset to use for training machine learning models.There are 4 classes in the data. Class A has 645 instances, Class B, C and D have 287 , 9 and 3 instances respectively.
For the test data A,B,C and D have 150,85,1 and 2 instances respectively.
I was thinking of using LOOCV for the training and testing . Is it worth it ?
I found this post on tackling imbalanced data in machine learning quite useful: "8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset" by Jason Brownlee (please see link):
https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/
In any case, if you want to find out by (stratified) cross validation whether the classifier performs well on all classes and not only on the frequent ones, accuracy would not be the appropriate measure but rather macro-averaged F1.
I found this post on tackling imbalanced data in machine learning quite useful: "8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset" by Jason Brownlee (please see link):
https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/
In any case, if you want to find out by (stratified) cross validation whether the classifier performs well on all classes and not only on the frequent ones, accuracy would not be the appropriate measure but rather macro-averaged F1.
Dear Vincent Appiah,
You can use 10-fold cross validation technique to solve the problem.
You can take help from the following link:
https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/
http://www.chioka.in/class-imbalance-problem/
Hi Vincent,
I would say you will need to perform data augmentation on your under-represented classes. This depends on your data and it is essentially trying to generate more data instances by relying only on the training data you have.
LOOCV is good for undersampling. That is, taking less samples from the majority classes.
For oversampling and for an explanation of dealing with imbalanced data please refer to this article: https://www.marcoaltini.com/blog/dealing-with-imbalanced-data-undersampling-oversampling-and-proper-cross-validation
Finally you should also find out if classes C and D are outliers or just under-represented.
Hi,
We used SMOTE to balance the classes in one of our problems. And validating your outcome is very challenging. Have a good measure of Precision, Recall and F1score.
Good luck!
Thanks,
Vikram
- Badges
- Science topic