The problem is that NN implicitly learn the prior distribution of the data, if you are using the posterior distribution (approximated by cross-entropy or MSE) as the training loss.

So the "proper" way would be to use a different loss function for training (which is however probably difficult to implement). You can try a logarithmic loss (e.g. mean_squared_logarithmic_error in Keras), but I am not sure whether it helps.

Alternatively, you can try to increase the number of samples of your small class, to make your dataset more balanced. The best way would be recording new samples of this class. If this is not possible, then you can try to increase the number of samples with augmentation, or simple repetition. You mentioned that you already tried augmentation, but did you augment the data to a degree that the classes were nearly balanced in the dataset?

Or you can search for "imbalanced classification" in Google, perhaps something useful comes out.

Regards,

Michael

Thank you very much Michael for your suggestions. I would definitely try with logarithmic loss. I did not make the class balance, as augmentation was performed on all classes equally, I will try that one out too.

Regards

Shihav

The other way you should try are generating synthetic data and transfer learning.