I am implementing a two-step classification algorithm. The idea is to identify examples for which classifier is not certain in prediction in the first step and to delay decision about membership for that example till the second step of classification. Of course, I can look at absolute value of prediction. Are there other possibilities?
SVM classifiers can give you decision values instead of simple class labels. The decision value for a data instance is the (signed) distance to the separating hyperplane in the SVM model. Labels are assigned based on the sign of this decision value.
The absolute value of the decision value (e.g. the distance) can be regarded as a measure of confidence, larger being better. This is the only measure of confidence you have when using SVM models.
You can empirically assess the model's precision as a function of the decision threshold to choose a reasonable value in the first step of your classification scheme (this is also how ROC curves are calculated). The default decision threshold for SVM classifiers is 0 (e.g. labels are assigned based on the sign of the decision value).
Hi, you can get some probability estimates as well, cascading some additional processing to SVM. There are some implementations already available in libraries such as LibSVM:
http://www.csie.ntu.edu.tw/~cjlin/libsvm/
by running it you directly get the output with the associated classification probability for each class.
A description of how it works can be found in section 8, here: http://140.112.30.28/~cjlin/papers/libsvm.pdf
Hope this may help!
I agree with the proposal of Marc Claesen
in addition you can test your data directly with the Matlab function
Regards
Hi,
you can use a competence measure of your classifier for your test data. This competence measure is calculated by a potential-cost function which uses the distance between your training data, used to train your classifier, and the new test data. Thus, you can train a first classifier as a competence voter which "knows" the valid data set. Then you can train another classifier or an ensemble of classifiers for your problem at hand. The first classifier decides which one of the second classifiers has the best competence and uses that for classification.
See: Combining Pattern Classifiers: Methods and Algorithms (Chapter 6)
Ludmila I. Kuncheva
ISBN: 978-0-471-21078-8
Bests,
Peter
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training