The task involves predicting a binary outcome in a small data set (sample sizes of 20-70) using many (>100) variables as potential predictors. The main problem is that the number of predictors is much larger than the sample size, and there is limited/no knowledge of which predictors may be more important than other. Therefore it is very easy to "overfit" the data - i.e. to produce models which seemingly describe the data at hand very well, but in fact include spurious predictor variables. I tried using an ensemble classification method called randomGLM (see http://labs.genetics.ucla.edu/horvath/htdocs/RGLM/#tutorials) which seeks to improve on AICc-based GLM selection using the "bagging" approach taken from random forests. I checked results by K-fold cross-validation and ROC curves. The results seemingly look good - e.g. a GLM which contains only those variables which were used in >=30 out of 100 "bags" produced a ROC curve AUC of 87%. However, I challenged these results with the following test: several "noise" variables (formulas using random numbers from the Gaussian and other distributions) were added to the data, and the random GLM procedure was run again. This was repeated several times with different random values for the noise variables. The noise variables actually attained non-negligible importance - i.e. they "competed" fairly strongly with the real experimental variables and were sometimes selected in as many as 30-50% of the random "bags". To "filter out" these nonsense variables, I tried discarding all variables whose correlation coefficient was not statistically significantly different from zero (with Bonferroni correction for multiple variables) and run randomGLM on the retained variables only. This works (I checked it with simulated data), but is of course very conservative on real data - almost all variables are discarded, and resulting classification is poor. What would be a better way to eliminate noise variables when using ensemble prediction methods like randomGLM in R? Thank you in advance for your interest and comments!
Dear Craig,
The two problems you mention, overfitting and interpretation of random forests, are partly mythological and mostly, now, historical.
About overfitting. It's almost always caused by letting the trees continue splitting to purity. There are analytical results showing that doing this is provably inconsistent. We exactly no idea as to why this is a default option in random forest.
Solution, don't let any terminal nodes in a single tree contain less than some fixed small size, say around 5% of the sample size. As expected, the current best analytical results show that the terminal node size must grow, slowly, with the sample size. But we've found that just using some small fraction, like 5%, works fine, as shown by intensive replications and cross-validation and permutation testing.
More on overfitting. Use a crowd machine or a regression collective over a large number of differently tuned machines, of any type. This way we avoid having to name a winning scheme for any data set, and get asymptotically optimal results. The larger point here is that tuning parameters of any kind, or application of any collection of machines, such as SVMs with different kernels, is no longer necessary. Researcher brainpower is better directed at other problems.
About the black box aspects of a model-free learning machine. We use two solutions. First, use the machine as a signal detector and as a refinement scheme to declare good predictors, while driving out noise. Recurrency, mentioned in an earlier note, seem to work well for both problems. Then--with some much smaller feature list in hand--send it to any small, interpretable parametric model of the researcher's choice. And, of course, compare the two predictions made by the original machine and the small model. They are routinely quite close.
Another method for opening up the black box: use a risk machine. That is, let random forest, or any other machine, run as a probability machine and get the log odds for each feature. Similarly you can get risk effects, risk differences. These all compare well with a logistic regression model, in the sense that the individual risk machine estimates for each parameter are agreement with that given by the logistic model. The two estimates of the individual parameter can be shown to be asymptotically equal. And importantly, using a risk machine we don't have to initially declare any parametric model, linear or logistic, over a set of features.
We have peer-reviewed publications for everything above.
Jim
Dear Prof. Shuryak,
You could consider applying the method of recurrency. This uses random forest, or any learning machine, to calculate the probability that a feature has no predictive capacity.
The method is described in joint work with Emily Holzinger, myself and several others. This appeared in the BMC Proceedings; paper attached. In this project we also described other methods and comprehensively compared the results.
The recurrency scheme can also be used to declare some features more predictive than others, by estimating the probability that a feature is predictive. Doing this we avoid having to generate any linear ranking over the features. Ranking schemes of any type are easy to defeat, as shown by Condorcet (1785), and are all versions of the voter preference problem. There is a huge literature on this subject, and it's generally acknowledged that there is no provably optimal solution, only plausible ones.
Moreover, recurrency can be used to declare some pairs or sets of features as jointly strongly predictive, while any elements of the set might be only weakly so. For this, co-recurrency is deployed.
Applying this scheme to several million SNPs, for example, remains a computational challenge. But for smaller feature sets, this is a version of entanglement mapping, that is, model-free interaction detection.
Let me know if you have any questions about any of these methods.
Jim
Thank you very much, Aliaksei and James! I will read the literature you suggested with interest, and would be grateful for additional suggestions from anyone interetsed!
I add with the reminder that there is a big difference between modeling for prediction and modeling for statistical inference. Ensemble models such as random forests are great predictors, but as you have pointed out can overfit the data and are difficult to explain. Getting at the heart of your question, I have seen publications where random noise is included as a variable (as you have done), and that is then used as a benchmark for inference. Those explanatory factors with lower ranking than your random variable are interpreted as noise parameters (but of course they contribute to the prediction accuracy in a non-meaningful way), and those predictors with explanatory power (or inclusion probabilities or whatever) greater than the random benchmark are the topic for discussion and interpretation.
Update: I've attached an example for you.
Article Prediction of fishing effort distributions using boosted reg...
Thank you Craig, this is very instructive! Can you please provide references w)hich use random noise as benchmarks?
Dear Craig,
The two problems you mention, overfitting and interpretation of random forests, are partly mythological and mostly, now, historical.
About overfitting. It's almost always caused by letting the trees continue splitting to purity. There are analytical results showing that doing this is provably inconsistent. We exactly no idea as to why this is a default option in random forest.
Solution, don't let any terminal nodes in a single tree contain less than some fixed small size, say around 5% of the sample size. As expected, the current best analytical results show that the terminal node size must grow, slowly, with the sample size. But we've found that just using some small fraction, like 5%, works fine, as shown by intensive replications and cross-validation and permutation testing.
More on overfitting. Use a crowd machine or a regression collective over a large number of differently tuned machines, of any type. This way we avoid having to name a winning scheme for any data set, and get asymptotically optimal results. The larger point here is that tuning parameters of any kind, or application of any collection of machines, such as SVMs with different kernels, is no longer necessary. Researcher brainpower is better directed at other problems.
About the black box aspects of a model-free learning machine. We use two solutions. First, use the machine as a signal detector and as a refinement scheme to declare good predictors, while driving out noise. Recurrency, mentioned in an earlier note, seem to work well for both problems. Then--with some much smaller feature list in hand--send it to any small, interpretable parametric model of the researcher's choice. And, of course, compare the two predictions made by the original machine and the small model. They are routinely quite close.
Another method for opening up the black box: use a risk machine. That is, let random forest, or any other machine, run as a probability machine and get the log odds for each feature. Similarly you can get risk effects, risk differences. These all compare well with a logistic regression model, in the sense that the individual risk machine estimates for each parameter are agreement with that given by the logistic model. The two estimates of the individual parameter can be shown to be asymptotically equal. And importantly, using a risk machine we don't have to initially declare any parametric model, linear or logistic, over a set of features.
We have peer-reviewed publications for everything above.
Jim
Dear JIm,
Thank you for an interesting and informative reply. Can you suggest some references for these methods?
Igor
Dear Igor,
Send me your email address and I will send you the list of accepted papers.
Jim
Dear Jim,
As you suggested, I sent you an email and look forward to your publications!
Igor
You might find it useful to look into the Boruta algorithm for feature selection by Miron B. Kursa, Witold R. Rudnicki, which is also available in R. It provides a way of identifying irrelevant variables by introducing noisy variables (similarly to what you have been trying to do).
- Badges
- Science topic