By higher discrimination, we mean higher AUC and specificity. The supervised SVM was trained, tested with a 10 fold cross validation. In my opinion, I would tend to go with the lower results obtained from the supervised SVM - What is your take?
Is there much difference in complexity of the two models? Perhaps one is too simple (underfitting) or / and one is too complex (overfitted)?
Was any form of validation used in generating the unsupervised model? I would advise an independent validation set not used in cross validation to test both models and allow you to directly compare their performance on independent data.
True independent test sets are critical in evaluating performance of discrimination/classification algorithms as the risk of overfitting is enormous.
James, No form of validation was used for the unsupervised SVM. The independent test set is a great idea - however, data is limited. And yes, there is probably difference in complexity - supervised model was validated by 10 CV and bootstrapping.
Paul, thanks for the references...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package