Hi,
I am training my models to classify 10 classes. Some of the classes are easy for my models and achieve accuracy greater than 95% but some classes have very hard patterns and accuracy stops at 83% which sucks overall accuracy of my models. I am struggling to find any way which could deal with hard pattern to improve overall accuracy.
How could I achive better accuracy on hard examples?
Thanks in advance!
I would like to add to Bazhdar Noureddin Mohammed answer by suggesting the following
1. Your dataset may have class imbalance. If I recall correctly from Dr. Ng's course, this could cause your models to become biased during training time. A bit more details about your project may help the community to better understand why you are having this biasing issue.
2. I strongly recommend auditing this course Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization. Don't worry if most of it is on Deep Learning, the intuitions are mostly same for classical Machine Learning techniques
3. Find one or two survey papers that is closely related to the problem your are solving. This is arguably the easiest way find out what tools and techniques other researchers are using to solve the same or similar class of problem that you are solving
4. Try to visualize the dataset and perform Principle Component Analysis. It might turn out the feature vectors you are using may have less correlation to certain classes which you are having a hard time in training.
5. Consider using a simple DNN but be advised, DNNs perform better when they have a lot of data (this is my personal opinion but I read something similar in literature of DNN)
I hope these information helps you out. Good luck
you can improve your model accuracy by:
1- add more data
2- treat wisely with missing and outlier values
3- use a proper feature selection
4- choose classifcation techniques based on your data
I would like to add to Bazhdar Noureddin Mohammed answer by suggesting the following
1. Your dataset may have class imbalance. If I recall correctly from Dr. Ng's course, this could cause your models to become biased during training time. A bit more details about your project may help the community to better understand why you are having this biasing issue.
2. I strongly recommend auditing this course Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization. Don't worry if most of it is on Deep Learning, the intuitions are mostly same for classical Machine Learning techniques
3. Find one or two survey papers that is closely related to the problem your are solving. This is arguably the easiest way find out what tools and techniques other researchers are using to solve the same or similar class of problem that you are solving
4. Try to visualize the dataset and perform Principle Component Analysis. It might turn out the feature vectors you are using may have less correlation to certain classes which you are having a hard time in training.
5. Consider using a simple DNN but be advised, DNNs perform better when they have a lot of data (this is my personal opinion but I read something similar in literature of DNN)
I hope these information helps you out. Good luck
The features might not be discriminative enough. Change the features so that they better reflect the class boundaries and feed again to the classifier. Also cross-validation can be used to remove the bias during selection of training-testing datasets.
Thank you all for useful suggestions. I am using pretrained-imagenet CNNs with 60K training and 10K testing samples. Would it be good to apply 10 K-Fold Cross validation?
Some actually useful suggestions for deep learning training:
- At least in Keras, you can assign class weights before training with e.g. normal cross-entropy loss.
- If class weights don't work, use focal loss. That is exactly for imbalanced classes. It penalizes easy samples during training to learn the hard samples better.
firstly check your data size and data balance, as imbalanced data cause a bias towards some of the classes. However, evaluating a model with imbalanced data using accuracy only is not enough. Accuracy for imbalanced data could trick you about the model over all performance. A better way to to evaluate your model performance with imbalanced data is to use F1-score metric that comprises both recall and precision. F1-score will give you more meaningful performance of model trained with imbalanced data of all classes.
you can also use techniques like data augmentation and parameters fine-tuning to achieve your target performance metrics.
I would also recommend you to various pre-trained CNNs for your application from shallow CNNs to deeper CNNs
Besides what other respected researchers mentioned, I would recommend Transfer learning for the same domain.
check out this paper for more details.
Article Towards a Better Understanding of Transfer Learning for Medi...
Agree with the suggestion of Azmyin Md Kamal and Laith sabah Alzubaidi , I also recommend,
1. Visual the hard class input features. Try to assess why the model is misclassifying? Strong recommendation: use of LIME (explainable machine learning).
2. In addition to data balancing, feature engineering, and hyper tunning. I suggest using the R square as a performance metric. Select the best report R square model and then used that model to expand the knowledge with uncertainty based active learning mechanism especially on hard classes.
- best of luck.
- Badges
- Science topic