If we have multiple classifiers and we need to know which one is under-fitting, and which one is overfitting based on performance factors (classification accuracy, and model complexity)
Are there any method to select the dominate classifier (optimal fitting) that balance between the above-mentioned two factors?
Hi Hayder,
- A significant difference between your train and test accuracy is the sign of over-fitting.
- To understand under-fitting you need a reference model that fits better on the same dataset.
If you want to compare 2 classifiers, you need to compare their test set performances. According to the problem you may want to use the following metrics:
- Accuracy
- Precision
- Recall
Plus, you may try to combine multiple models in order to increase the overall performance, its called ensemble learning.
Good luck!
Thank you Murat Cihan Sorkun for your explanation, If you have any reference discuss the overfitting and the under-fitting problem in the experimental result, it can help a lot.
To answer your question, we need to understand two things:
Why do we need a classifier? This is because the final trained model is deployed somewhere for solving some real-world problem/application.
And why do we test the classifier? To check the generalization capability of the classifier. Test accuracy gives a rough estimate of the generalization capability of the classifier when used in real-world application.
So if your test accuracy is good, then you can expect that your classifier will perform well when deployed to solve real-world application. Thus test accuracy is important. Moreover, as pointed out by other people, there is a trade off under-fitting and over-fitting. Over-fitting reduces the training error but decreases test accuracy, on the other hand, under-fitting increases training error but might increase the test accuracy. We need to strike a balance between this while keeping the focus on test accuracy.
Useful link below:
https://medium.com/greyatom/what-is-underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6803a989c76
I hope; I made sense.
Thank you Salah Mortada for your explanation. However, I would like to draw a figure that compares different methods that used in the experiment result and clearly show the overfitting, under-fitting methods. If you have an example of such an experimental result will be grateful.
It's my pleasure Hayder Naser Khraibet Al-Behadili
I attached here two links:
Article Correct machine learning on protein sequences: A peer-review...
https://towardsdatascience.com/overfitting-vs-underfitting-a-complete-example-d05dd7e19765
Wish to you all the best
I recommend you that you compare Confusion Matrix, Kappa Statistic (KS),
Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). Because these values give information about true classification without overfitting and underfitting.
The classification performance is very reliable because Kappa values are very close to the value 1 as given in our a study.
Thanks Fahriye Gemci for your suggestion, regarding to the above-mentioned evaluation criteria's how we can decided the comparing classifiers are under-fit or overfit the data instances.
As Fahriye Gemci is refering to the Confusion Matrix go one step further and use Receiver Object Characteristic (ROC)-curve to visualize, your performance. Be aware that not only do classifiers have different tendencies to over-fitting but also your performance metrics show different properties in relation to your data/problem structure (excellent paper on the topic is http://atour.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf):
- binary vs. multiple classes
- invariant against class distribution or changes in the confusion matrix
- noise in the true class distribution
- heavily scewed class membership distribution (e.g. positive class has magnitudes less cases in which accuracy does not help you much as a metric
If you consider this in your metric selection you will get your best performing classifier(s) regarding your problem.
For the second part of under-/ overfitting, there are things indicating this. For example, in SVM a high number of support vectors indicates overfitting. In Decision Trees it is the number of splitts, the depth, and information gain that indicates overfitting.
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining