I am dealing with Image Classification problem and I am using SVM classifier for the classification. I have setup an experiment that consists of two level classification. I assume that I first pass the test image through the top level classifier, if the classification confidence of top level classifier is above some threshold its ok, but if it is lower than the threshold, the test image is feed to lower level classifier. Is it possible to calculate the classification confidence in terms of percentage? And how do I achieve this?
As you may know, SVMs works by determining a separating hyper-plane between the two classes. For a new test sample, it classifies it as class 1 or 2 according to which classes it is closest to. The decision_function(X) gives you the distance of the sample to the hyper-plane, or the distance to each of the two classes. This distance is a value that can be too large or too small according to where your sample is located from that hyper-plane or from the tow classes.
Now, the confidence score (in terms of this distance measure) is the relative distance. For example, if sample S1 has a distance 80 to Class 1 and distance 120 to Class 2, then it has (100-(80/200))%=60% confidence to be in Class 1 and 40% confidence to be in Class 2.
If you want direct confidence score, you can use function (predict_proba) in sklearn.svm.SVC directly. It will give you the probability of prediction of the test sample to be in each class. You can use these probabilities as confidence scores when you decide on your classification threshold.
Some SVM implementations allow you to retrieve the classification confidence (which they use to assign the prediction class) instead or along with the predicted classes; and some implementations do not. To the best of my knowledge, Matlab does not give you confidence scores, but LibSVM and Sklearn do.
Thank you for your answer. I am using Sklean LinearSVC and it has a method 'decision_function(X)' which predicts confidence scores for samples. There is one value per class and these are not in terms of percentage. How do I interpret these values in terms of percentage? What exactly these values are?
As you may know, SVMs works by determining a separating hyper-plane between the two classes. For a new test sample, it classifies it as class 1 or 2 according to which classes it is closest to. The decision_function(X) gives you the distance of the sample to the hyper-plane, or the distance to each of the two classes. This distance is a value that can be too large or too small according to where your sample is located from that hyper-plane or from the tow classes.
Now, the confidence score (in terms of this distance measure) is the relative distance. For example, if sample S1 has a distance 80 to Class 1 and distance 120 to Class 2, then it has (100-(80/200))%=60% confidence to be in Class 1 and 40% confidence to be in Class 2.
If you want direct confidence score, you can use function (predict_proba) in sklearn.svm.SVC directly. It will give you the probability of prediction of the test sample to be in each class. You can use these probabilities as confidence scores when you decide on your classification threshold.
Use the Entropy measure on your class probabilities. Check it out.
http://www.umiacs.umd.edu/~zhuolin/Publications/OnlineDDL.pdf
SVMlight gives you the confidence level of the classification. The value is positive for class 1 or negative for class -1, but it does give you the confidence rather than just the classification.
http://svmlight.joachims.org/
Thank you all. I used the method predict_proba of sklearn. The predict_proba(x) method predicts probabilities for each class. In my project there are 300 classes and when I feed test image to the classifier it gives probabilities p1, p2, ..., p300 for classes c1,c2,...c300. The probabilities are such that p1+p2+...+p300 = 1. Would you mind telling how do I manipulate these probabilities to calculate confidence for a particular class.
try libsvm, it calculates the probability estimation using logistic regression.
- Badges
- Science topic