In machine learning research, many researchers have defined the stability of feature selection algorithms as the robustness of the ”feature preferences” they produce to training set perturbations.
Need some emphasis on real time scenarios when exactly the situation which causes perturbation to the training data set?
.
the point is not that your training set will change in time !
the point is that your training set is a random sample from some distribution and, as such, it does not faithfully describe the totality of the distribution : some aspects of the distribution will be poorly described, some aspects of the sample will be "irregularities" resulting from the random sampling process which do not belong to the distribution
therefore, you cannot fully trust your training sample : if you had drawn (or were given) another training sample, feature selection results might have been different
.
what you expect from "good" feature selection algorithms is that they are both efficient (find significant features) and stable (that is do not depend on sampling "irregularities") ; you need both properties since it is pretty easy to be just stable : for instance, report "no informative variable" is very stable but definitely not efficient !
.
if your selection algorithm is both efficient and stable, then you can be confident that things will be ok when you will deploy your model on new data (which can be considered as a new random sample from the distribution) : as your feature selection does not depend on the "irregularities" of your training sample, your feature selection process has selected features that should be relevant also for your new sample (deployment sample).
.
Selecting features from data set with low noise may not be the best option to noisy data for instance. Other examples is when in biological data some features are selected from only a single specie or an individual. A robust feature set may be as invariant to noise/individual and environmental effects as possible.
- Badges
- Science topic