When trying to fine tune the SVM classification model using the grid parameter optimization, i found many values of Cs and gamma with different numbers of support vectors having 100% cross validation and training accuracy. Having understood how to choose my best support vectors, im wondering what part gamma value has to play in this. Do i just need to consider the number of my support vectors only in deciding the best C value to use or gamma value too should be put into consideration now that i have many choices with 100% CV and training accuracy.
Although having 100% accuracy in a grid search is not uncommon, it's important to understand why that is the case.
In your case, I assume, you are actually fitting with the whole data-set in each cross-validation fold, and then testing on a subset of your data-set.
In this case the accuracy should approach 100% if you built a good model. However, note that at this point you don't really know if you are over-fitting (or under-fitting).
I would suggest to validate each cross-validation fold using a better 'metric', or in this case just doing proper test/train splitting folds.
Concerning the gamma value in the SVM, gamma says how far the 'reach' of each training example is (http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html), but can be just thought of as a regularization parameter. The higher the gamma, the more local the reach, and you have to watch out that your model keeps a general behavior since it is prone to adjust too much to the training examples.
Although having 100% accuracy in a grid search is not uncommon, it's important to understand why that is the case.
In your case, I assume, you are actually fitting with the whole data-set in each cross-validation fold, and then testing on a subset of your data-set.
In this case the accuracy should approach 100% if you built a good model. However, note that at this point you don't really know if you are over-fitting (or under-fitting).
I would suggest to validate each cross-validation fold using a better 'metric', or in this case just doing proper test/train splitting folds.
Concerning the gamma value in the SVM, gamma says how far the 'reach' of each training example is (http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html), but can be just thought of as a regularization parameter. The higher the gamma, the more local the reach, and you have to watch out that your model keeps a general behavior since it is prone to adjust too much to the training examples.
Thank you. This is a great explanation. Can you tell me more what you meant by "reach" and the statement "the more "local the reach"as per gamma value?
Thank you. This is a great explanation. Can you tell me more what you meant by "reach" and the statement "the more local the reach" as per gamma value?
It is best understood by the plot in the link I sent you before, i.e. "Intuitively, the gamma parameter defines how far the influence of a single training example reaches, with low values meaning ‘far’ and high values meaning ‘close"
As stated by Mr Filos, large values of gamma indicate that distant data have more chance for being similar and measure of similarity is not so strict but when you have small values for gamma for data x to be similar to data y they must be fairly close.
Simple summary from above:
- gamma value is a scale of space distance
- it controls what is 'close' and what is 'far'
- let imagine points in 2D with circles around points with some diameter. Points in the circle are 'close', points out are 'far'. The gamma value set the diameter of the circles (non-linear, negative dependency on the gamma value).
- of course everithing is smooth in SVM, and the kernel value is a continuous value of the 'distance' between points
Just to elaborate on the excellent answers above: the SVM model is provided by a weighted sum of the kernel values evaluated over your support vectors. If you use a Gaussian kernel, each kernel function evaluates the distance of your testing point from a support vector using a Gaussian-shaped function. The "gamma" parameter in this context is akin to the variance of the Gaussian, i.e., high gamma (if gamma is in the numerator of the exponent) means that the Guassian has a very small variance, low gamma means that the Gaussian has a very large variance. In the extremes, for gamma that goes to 0 all points are considered and you get a sort of "global average", while if gamma goes to infinity you obtain a rule which is similar to 1-NN, since a single point will contribute to your prediction.
Thanks for the question and excellent discussion. As a traditional lab-based scientist new to pattern recognition and SVMs, this was incredibly valuable! To run SVMs I use the e1071 package in R. One feature to appreciate with this package (and I assume others) is to run a tuning matrix prior to the analysis. The SVM tuning will suggest best gamma and C (cost) coefficients for the subsequent analysis.
- Badges
- Science topic
- Similar topics
- Engineering
- Electrical Engineering
- Signal Processing
- Image Processing