We're looking to use random forests to understand the importance of environmental variables in predicting species abundance and richness. I'm struggling a little bit with understanding a few different things, for example, why one would choose OOB over randomCV. Below are the points where I am looking for help.
1. With a relatively small dataset (several hundred points) with “many” predictors (20) what is better, random cross-validation or OOB. Mtry for both always ends the same (2).
2. When looking at variable importance scores, what is “better”, mean decrease in accuracy or, Mean decrease in GINI node impurity?
3. Should we be tuning ntree or is a default of 500 ok?
4. Should we be transforming the response variables to have a more normal distribution?
Thanks!
Any resources for a layperson would also be appreciated.
As a note, I'm using the r package caret to create the models and calculate the variable importance.
Please watch this
https://www.youtube.com/watch?v=dJclNIN-TPo
It may be of a help to understand random forest in R Programming
Good Wishes!!!
Andrew J. Fairbairn
I’m going to make this a detailed response to make it self-sufficient, please be patient and read through the entire thing.
a) First of all, let’s consider what the Out Of Bag metrics are. When training a Random Forest, we create bootstrapped datasets to train each Classification And Regression Tree separately, so as to make them as decorrelated as possible. Each bootstrapped dataset is as “big” as the original, but was created by sampling with replacement from the original dataset. Thus, there will be samples that are repeated (or sampled multiple times) in every Bootstrapped Dataset. Conversely, there are samples in the original dataset that are never sampled in a particular bootstrapped dataset. What you'll find is when you do this Bootstrapping on an average about 30-40% of the samples in your original data set will not end up in a specific bootstrap dataset. These samples that don't end up in the bootstrap dataset or don't end up in the bag are referred to as out of bag samples.
As these Out Of Bag samples weren't used to train that specific decision tree, we can use them as a surrogate validation set. So, we select a particular sample and find all the trees in the random forest where it was an Out Of Bag sample. We calculate the accuracy of these trees in predicting this OOB sample. We repeat this procedure for all the samples to get an averaged OOB score.
Now, as you can see, the OOB score is slightly unbalanced as for any one sample, we only considered trees where it was an OOB sample. So, it may not always be as good as a “proper” validation score. However, in cases where the learning dataset is small, it is a good idea and an established practice to use the OOB score as a validation metric.
b) First of all, I assume that you have a classification problem, or else your question may not be perfectly stated.
Using the mean decrease in accuracy versus the Mean decrease in Gini node impurity are not better…but different.
The mean decrease in accuracy metric calculates how much precision the model gains on an average by using a specific feature.
On the other hand, the Gini score calculates how homogeneous the split becomes on an average if we use a specific feature to split on. If a feature is informative, it tends to split mixed labeled nodes into pure single class nodes.
The Gini impurity is a local attribute (as it looks at every split in every tree), but the Accuracy scores are more global (as it permutes over the final random forest). In that sense, I think you might be looking for the Accuracy scores as your parameter of choice.
c) You should always vary this hyper parameter and select the best value. There is no way to know if ntree=500 is enough, or too few.
d) In most cases, that would be a good idea if you were using a gradient based learning approach or making assumptions about the distributions of the features.
However, decision trees (and thus, random forests) do not use gradient based learning and don’t make assumptions about the distributions of the features.
Thus, there is no explicit need to do such transformations.
Viel Gluck!
Aashwin Mishra Thank you for the very detailed and informative answer! It helps a lot. Cheers!
- Badges
- Science topic