Hi there,
I have over 1 million records, with a categorical response variable and a mix of categorical and continuous predictors. There are several missing values in the dataset, this is represented by a zero. This is just a trial; my final dataset will have at least 10 more predictors. I have attached a copy of what the data looks like...
Im using python to run random forest model to determine which variables that best predict y. I have 2 questions:
1.When I try to train the model using the training sets (y_pred=clf.predict(X_test), I usually get an error stating it cannot convert a str to a float. I have tried to use the LabelEncoder to help with transforming the values but it gives me a bad input shape error, I then tried to correct this with no luck. So, is a Random Forest appropriate when we have large datasets, several predictors and both continuous and categorical predictors? If so, any advice regarding data structure for RF?
2.In terms of missing values/NA’s, which are represented by a zero, im concerned about what RF does with these. If it takes a zero value, which is very close to other values such as (0.5) does the model then consider these as a value of zero or a missing value. There are many forums suggesting changing these NA’s to the median of that column or a value that is out of range so the model might exclude it. I’d prefer excluding NA’s rather than adding a value. Any suggestions on a way forward?
Thanks in advance!
Dear L. Perumal,
1) RF is accurate in this case. Like error said you probably have a string in any column in DataFrame, of course, string are not allowed as predictor value for a model, maybe try to use the pandas built-in function df.info() to find them.
2) In the case of NAN you can:
- filling gaps by zero will undoubtedly generate model noise!
- change NAN to median/mean it may be the most accurate option
- delete records with NAN using df.dropna() method (data loss can have an adverse impact on model training)
- train a new model to fill Nan - quite interesting idea
There is no best method, it is dependent on: data quality and how many NANs are in your data set. In my opinion, you need to try different ways to find the best one. I can not help more if i do not see code. Do you have repo in GitHub or GitLab ?
Best regards!
Hello L. Perumal,
1. Random Forest should do the trick. Generally Random Forest models can accommodate both categorical and numerical variables, but strings and characters will need to be standardized and factored before the model will run.
2. I definitely agree with Jan Bińkowski regarding missing values. It's best not to leave them as 0, those values will not be treated as missing, they will be regarded as a value of 0 for that variable. Some algorithms will allow you to leave the cell blank and run ignoring that particular cell, though there are better ways to approach this.
The mean/median value is a pretty safe approach for a data set of that size (assuming there aren't too many missing values), although there are also some pretty interesting random forest imputation methods that will fill in missing cells by using a minimal RF model based on the other available variables.
Something else to take a peek at if you really want to dive in, check out Boosted Regression Trees. Functionally they are similar to RF, but allow for much greater parameter tuning and are able to deal with missing values.
Good luck on your ML journey!
Jan Bińkowski Thank you for the info! I dont have a repo but when I do I can let you know, if you are still interested.
David Emde I think Boosted Regression Trees might be a good option. Thank you for assisting :)
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python