I am using Qualitative_Bankruptcy( link: https://archive.ics.uci.edu/ml/datasets/Qualitative_Bankruptcy ) dataset which is fully categorical, first I have encoded dataset to be able to train it with models. Then I get X - feature columns, and y - target column. I am downloading the screenshots from the source code written by me to be more understandable. Thanks in advance
It is not clear to me if you have over trained, but it may be that your classification problem is easy -- of very low uncertainty and ambiguity, perhaps like discriminating between mature humming, mature sharks, and mature goats. Almost any plausible feature set (outer layer or tooth sample for instance) will give you near "perfect" results. In signal-classification terms, the discriminating signal-to-noise ratio is high, in which case any plausible classifier algorithm should work well.
But note that a finite sample size N for testing a classifier never allows you to claim 100 % performance. You have to use statistical analysis based on your test sample size N to estimate the confidence limits on performance.. Presumably the upper bound of X % confidence limits on a high-performance classifier with 100 % correct classification on a sample size of N will be 100 % correct classification. But but the lower confidence limit will perhaps be around
(1- (X%/100))^(1/N) X 100 %
correct classification. I'm estimating wildly here, but you get the point. Observation of perfect performance against a finite test set (your case) does not mean it is a perfect classifier. The probability of error remains unexplored by the small sample size. In some critical real-world problems that small probability is very important.
Ronald Kessel thanks for your support , as you mentioned this classification problem is easy , it is very small dataset , may be that's the reason of getting 99-100% accuracy.
Anar Sultani ,
I would recommend you to use k-fold or leave-one-out cross validation for small datasets (https://scikit-learn.org/stable/modules/cross_validation.html).
Hold-out validation would be best suited for problems with more samples.
Rahman Peimankar, Victor Henrique Alves Ribeiro, thank you for all your suggestions ad help, but I have tried and also it shows that almost all folds' accuracies are near to 99. I would like to know is there any possibility that encoding categorical variables with label encoder are wrong or true. I mean, I am not sure that I have done the encoding process properly.
Anar Sultani ,
The data set you posted seems easily distinguishable in a visual manner (lots of Ps in class NB and lots of Ns in class B). Perhaps it is an easy problem. Try using a logistic regression model, it may achieve similar results.
Regarding the encoding process, I would suggest using OneHotEncoter instead of LabelEncoder for the variables.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training