I know that in data mining it is very important as data is processed before putting it in a classifier, I wonder if it also has a relationship with the type of kernel chosen, thanks
Normalizing features before classification is indeed important.
I recently read two articles that found similar conclusions: when dealing with Linear classifiers (like linear SVMs) the L2 norm performs very well.
I do not know which are proper normalizations when using different kernels.
You can find attached the two publications I mentioned.
http://www.robots.ox.ac.uk/~vedaldi/assets/pubs/chatfield11devil.pdf
http://www.robots.ox.ac.uk/~vgg/rg/papers/peronnin_etal_ECCV10.pdf
Which kernels go well with what normalization depends, as usual, on the data at hand. However, if you use kernels such as the exponential (rbf) kernel, normalization can be very important indeed. Since the rbf kernel is defined as k(x,y)=exp(-gamma*||x-y||^2) or similar variants thereof, each dimension of the feature vector is treated equally, since every dimension contributes similarly to the euclidean norm ||x-y||^2 = sum_i (x_i-y_i)^2.
Thus, the degree of variation (in terms of the absolute size |x_i-y_i|) of features can have a huge impact upon the result. Just imagine the first feature being uninformative for the given problem, but highly variable. An informative feature with only little amount of variation might easily suppressed by the first.
In such cases, methods such dimension-wise re-weighting or principal component projections might tremendously improve the classification result.
Also note that such dimension-wise normalization is not that important in the linear kernel, as the learning algorithms implicitly scales the dimensions on its own.
A good practice especially before applying kernels such as the RBF is data standardization which includes data centering and scaling. This family of kernels assumes that the data is centered around zero. In general you can achieve this by substracting the mean of all features. In addition, the features can be scaled to have variance 1.
I probably miss somthing, but I'd rather say that the rbf kernel does not benefit from the mean centering step, since the mean is a constant offset to each feature. This offset is effectively cancelled out when computing differences, i.e. ||(x+m)-(y+m)||^2=||x-y||^2. Hence, no benefit, but no harm done either.
As pointed out before, however, scaling might be a sensible thing to do.
Indeed, centering is not needed for distance-based kernels which are translation invariant. Thanks to Michael Kemmler for correcting. I find centering meaninful for higher degree polynomial kernels. However, I am not sure that centering around zero is always the best choice. I think it depends on the data distribution, but probably the mean should still be close to zero.
Read the link
https://docs.google.com/a/koyauniversity.org/document/d/1x0A1nUz1WWtMCZb5oVzF0SVMY7a_58KQulqQVT8LaVA/edit#
- Badges
- Science method