Hi all,
Currently, I am doing a research in classifying RBCs into four groups.
After extracting best features, I trained a pattern recognition Neural Network and performance is evaluated by 10-fold cross validation check. Miss-classification rate is around 6%.
In another experiment, I tried to use SVM model to classify one group and three other groups so called one-versus-rest classifiers. Then again repeated same strategy for the 3-group class until there is no need for another binary classification. Performance is evaluated by the same technique applied in the Neural Network strategy. Miss-classification rate for the first binary SVM classifier is zero and for the second and third binary SVM classifier is 2.1% and 3.2%. Which strategy is better and why?
If multi stage SVM is outperforming NN strategy, how can I evaluate the miss-classification rate?
Cheers,
What are RBCs?
The first thing that i always ask with multiclass problems is the class distribution. If the classes are unbalanced accuracy isn't a great metric for performance evaluation - try something like f-measure instead.
There's a commonly stated 'no free lunch' theorem that, loosely says, no optimisation algorithm is always better than another. So, for an arbitrary dataset like I will assume yours is, there isn't an answer to your question :-)
Generally, these questions are either answered empirically, or you may be able to specify a sensible model apriori by understanding the data and the problem.
If each of your examples belongs to only one of the four classes, your problem is a multi-class one, and there are many ways to solve it. SVMs can also be non-linear if using a kernel. As Niall already pointed out, there is no a "best" method for all the problems, and you need to look for understanding of the dataset, or to make empirical comparisons.
The F1 measure is a good metric of performance in multi-class problems. There are two versions of this: macro-F1 and micro-F1. However, since you have a single-label multi-class problem, the micro-F1 is the same as the accuracy. The macro-F1 measures the performance per class, and in that way, it compensates for class imbalance. This is computed as:
F1i = 2(Pi)(Ri)/(Pi+Ri) --> Macro F1 for class i
Pi = TPi/(TPi + FPi) --> Precision for class i
Ri = TPi/(TPi + FNi) --> Recall for class i
TPi = Number of True Positive classifications for class i (positive examples classified correctly)
FPi = Number of False Positive classifications for class i (negative examples classified as positive)
FNi = Number of False Negative classification for class i (positive examples classified as negative)
After computing the different F1 for each one of the four classes, you can do a simple average to estimate the general performance of the model.
In addition, you can run a statistical test between the results obtained from two different models to see if such results are significant. One good test in classification is the McNemar’s one.
Article Minimizer of the Reconstruction Error for multi-class docume...
Conference Paper Automatic Labeling of Forums Using Bloom’s Taxonomy
- Badges
- Science topic