Accuracy score might be uninformative here: 92% score can be achieved without predicting last two classes at all. Probably it is not the algorithm you want (imagine you have 4 groups of patients, 70% healthy, 22% with simple cold, 3% having pneumonia and 3% having asthma; you should care more about two last classes no matter they are minor). I suggest you to measure macro-averaged f1-score (it calculates f1-score for each of classes and then takes the mean value over all classes), it is a more adequate characteristic of performance.

Simple classification is sensitive to lack of balance. You can, instead, use any learning machine running in regression mode to get an estimate of the probability of group membership. This is strictly more informative than simple classification, and importantly, regression is not sensitive to lack of balance.

You may wish to explore algorithms with sampling strategies aimed at unbalanced data such as SMOTE or RUSboost.

Dear Prof. Sabeur,

In our experience, overfitting with learning machines is an artifact of inelegant and unrefined coding. Nathalie Japkowicz and Mohak Shah wrote about multiple, simple solutions to the balance problem in their excellent book: Evaluating Learning Machines (2011). See especially Section 5.6. As mentioned in my earlier note, there isn't any balance problem when doing machine learning for regression estimates. 

Following Prof Sabeur's answer, you can also try to make a one-against-all evaluation (using ROC score for instance) for each group and see if your models are as good with group 1 than group 3 and 4...

This is quite fast to implement and you will have another clue about any over-fitting issue due to imbalanced data.

Maybe you can make additional patterns in the rare classes with the adding very small noise to original patterns. You can also use rare patterns twice. Border pairs method is very goog for avoiding of the overfitting.

Article Border Pairs Method—constructive MLP learning classification algorithm

Salman,

This is an expected behavior.

Generally, cross-validation method can be used to avoid overfitting problem, so its result is much reliable. On the other hand, I'd like to highlight that having balanced data doesn't mean having better model accuracy (even under the evaluation of cross-validation) or performance, but it may help building a very good prediction model that can be efficiently generalized later on. 

HTH.

Samer