I have two feature sets I evaluated using SVM on a two-class problem. The first set (the "good" one) provides acceptable results, while the second one is far behind. When I merge the two sets using early fusion, SVM provides me with a result that is somewhere in the middle.
I use the RBF kernel, optimal gamma is estimated using grid search on the test set. The optimal gamma values were found to be different for the three feature sets.
I am curious, what kind of classifier is able to get the maximal benefit from using more informative features? Is it possible to get the same accuracy for the combined set as for the "good" one?
> What kind of classifier is able to get the maximal benefit from using more informative features?
People usually use Multi-Kernel Learning (MKL) for fusion of multiple features instead of simple early fusion. The idea is to set different weights and/or different kernels to the different features. There are a lot of public tools on MKL-SVM in the web. I never tried this before by myself, so you have to find them by yourself. An example is the SHOGUN library.
> Is it possible to get the same accuracy for the combined set as for the "good" one?
In the field of image classification, many people have shown that MKL-SVM with multiple features outperfoms the best individual SVM with individual feature, even if the multiple features include non-informative ones. That is, it is possible to get even higher accuracy for the combined set than the "good" one in your case.
There are multiple views to this problem, depending on the nature of the non-informative features. Are they irrelevant? Or redundant? Or perhaps just very noisy? As indicated in previous answers, Random Forests are well suited to handling several types of these problems in a generic way due to the nature of the algorithm: In each node a random subset of the features is evaluated and the split is made based on the criterion (e.g., entropy or Gini gain) selecting the best feature from this subset. As the results of different trees are combined more informative features are used in more node splits and the overall model compensates for a few wrong selections due to noise, chance, etc. However, Random Forests are not a panacea and for really doing as good as possible you need to understand what is the reason that the bad features are bad so that you can tackle it also outside of the classification algorithm, at least to some extent.
If you suspect co-linearity between features then doing PCA and keeping the needed ones is highly advised. If there are other non-linear types of redundancy auto-encoders will help create a more compact representation but are more computationally expensive.
If you suspect that there is significant noise something like MNF components might improve the performance on those features - kernel MNF coupled with regularization is particularly effective in this scenario.
If the features are just irrelevant to the problem then you can use feature selection schemes. However, if your set of features is too large then these may not be computationally feasible in which case you can pass the problem to the classification algorithm. Random Forests will do reasonably well here, provided you have enough training samples.
- Badges
- Science topic