3. Random Forest algorithm will handle correlated variables without issues.

You typically obtain 'variable importance' estimates showing you the relative importance of features (e.g. which variable is the best predictor). But due care is needed when interpreting that 'variable importance' values, as they are not designed to automatically answer "which is the best predictor", though they are often ranked/sorted in the correct order, from most important to the least important.

Interpreting the variable importance - especially in case when multiple heavily-correlated variables exist - is not trivial, but there are already numerous studies on the subject.

You can use RF in the caret package to perform regression problem and variable importance.

Some below R code which may help you to produce the results:

library(caret)

TrainY=trainingData[[1]]  #the 1st column is the response continuous variable

TrainX=trainingData[,-1]

TestY=testingData[[1]]

TestX=testingData[,-1]

indx

Thank you, Thanh-Tung Nguyen for the detailed script. I wonder if random forest in caret or other R package  can understand (1|id) "random variable" in the right hand side of the formula?

You mean: y~x1+x2+..+xn?

If yes, it is easy to understand when you use randomForest R package.

Hi Hanan,

As Adam said before, the interpretation of the variable importance measures (especially  Breiman's 'permutation importance measure') is not so simple. Some studies show that the importance decreases when the number of correlated variables and the level of correlation increase. So the first variable retained by the ranking computed by the random forests is not necessarily the most relevant if it belongs to a group of correlated variables. Actually I wrote a paper about it with some theoretical results (see attached files).

Article Classification with correlated features: Unreliability of fe...

Article Correlation and variable importance in random forests

I think you better off with the mixed modeling approach. As far as I know, the predictors used in Random Forest are fixed not random.

You might also want to read about all-relevant feature selection algorithms that are designed with exactly the purpose of determining all features relevant to a classification. Due to the complexities involved they use quite advanced, heuristic approaches to do the work.

I have used the Boruta R package for that; you can read the associated publications by Kursa & Rudnicki, maybe also some general info in addition to papers attached by Baptiste above (e.g. by Tuv et al).

Article The All Relevant Feature Selection using Random Forest

Article Feature Selection with Ensembles, Artificial Variables, and ...

Thank you all for your suggestions . I have tried both the Boruta package,  the caret

package and, party package  and they all have selected more or less the same important features. However,  varImp{party} throws the most important feature selected by Boruta and caret  to the end of the list. So, as Adam said, "it is no so simple" .

Hanan 

Just looking in to similar issues myself and came across the following which may (or may not) prove useful...

Sela, R. J., & Simonoff, J. S. (2012). RE-EM trees: a data mining approach for longitudinal and clustered data. Machine learning, 86(2), 169-207. (coincidence re: name)

Eo, S. H., & Cho, H. (2014). Tree-structured mixed-effects regression modeling for longitudinal data. Journal of Computational and Graphical Statistics, 23(3), 740-760.

Bürgin, R., & Ritschard, G. (2015). Tree-based varying coefficient regression for longitudinal ordinal responses. Computational Statistics & Data Analysis, 86, 65-80.

Another tree based approach...

Fokkema, M., Smits, N., Zeileis, A., Hothorn, T., & Kelderman, H. (2015). Detecting treatment-subgroup interactions in clustered data with generalized linear mixed-effects model trees (No. 2015-10).

install.packages("glmertree", repos="http://R-Forge.R-project.org")

-Steve