I have severely optimized SVM meta-parameters on the Iris classification dataset. For this I used 1000 times random cross validation and I took the best parameters to run the algorithm again. After that I observed that the accuracy on the test set was higher (around 1.3% error) compared to the training set (around 3.3% error). The difference is very significant. I do not have such results on any other dataset I tested on. How can this be explained? Ok, the parameters are optimized with the accuracy of random test sets, but how can a model do better on the test data than on the training data?
I split the dataset 1000 times in 90% training and 10% test data. So the standard error is very small. All samples are therefore 900 times in the training set and 100 times in the test set on average.
This might happen due to the Pinlaize parameter used with the SVM. The seperator hyperplane allows some training samples to override some samples (allow to exceed it) which cause the the (3.3% error). And while you take the test set significanlty smaller than the train set such a result is likely to be obtained... However the result you present show that your data is well clustered around the classes.
You can over-fit the hyper-parameters just as you can over-fit a model.
The Iris data set is so tiny that this isn't all that surprising a result.
True, I thought about this possibility. But how can there exist a model trained on training set with optimal hyperparameters so that its test error is even lower? What kind of geometry would such models have? It must have to do with wrongly labeled data in Iris.
We made a study on this phenomenon and this resulted in the attached publication
Conference Paper An Analysis on Better Testing than Training Performances on ...
I think this might have happened due to the size of your test set which I believe is small. Why not trying a 10-fold cross validation and see the results?
- Badges
- Science topic