I am trying to perform RandomForest in R. I wonder if I need to define the response type as Binary (like we do in GLM/GAM). If we need to do so, how to do it. I could not find any option.
I think R perform this change automatically. However, the command you are looking for, if I understand correctly your question is : "as.factor()". For example : Y_new = as.factor(Y_old). Factors are a type of data which are heavily used in R along with vector, matrix, array, list, dataframes, etc...
In brief, consider that "factor" is the appropriate format for categorical variables, I suggest you to transform your dependent and independent variables which you consider as being categorical in factors before performing any analysis.
Note that you can easily shift from one type of data to the others. For example, from numeric to factor, factor to character, etc...
Important remark : If you want to transform a factor into a numeric vector, do not use directly "as.numeric(vec_example)" but rather "as.numeric(as.character(vec_example))" or more efficiently "as.numeric(levels(vec_example)[vec_example])". This is the only trap to avoid, any other command in this field being pretty intuitive.
in addition to Geoffrey´s answer you should keep in mind that your data type for y will affect the mode random forests is used (classification or regression):
"Y, the response vector. If a factor, classification is assumed, otherwise regression is
assumed. If omitted, randomForest will run in unsupervised mode."
You can find this here: https://cran.r-project.org/web/packages/randomForest/randomForest.pdf (Page 17)