When measuring the variable importance, Olden(2015) proposed a new R2-based measure of relative variable importance after revealing methodological weaknesses of PSW and PCW.
Its basic steps is to:
Is there any need to train again after permutation? Or, Could I just make use of the original model weights to predict direcly under the modified datasets ?
P.S. I found the latter condition displaying in vi_permute function of vip R package.
.
good question !
if you do not retrain, you are studying the sensitivity of your trained model to each variable and can conclude for the importance of the variables conditionally to your trained model
this is not necessarily this notion of "variable importance" you want to quantify : you may want to quantify a notion of variable importance for your problem, irrespective of the model you are using, or rather inside a model class but irrespective of a specifically trained model
a first step to get rid of the dependence on the trained model is then to retrain for each permutation
.
- Badges
- Science topic