I am looking at a relatively large data set with numerical, nominal features. Unfortunately as is the case with such data, some of the entries are either missing or are invalid. Currently, I am following this approach
i. For Numerical Continuous Features : Find Mean/Median for each class and replace the invalid/missing value with the mean/median of that class.
ii. For Categorical/Nominal Data : Find Mode
I am not entirely sure what to do for test data. But I am thinking of following the approach suggested here (http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#missing1)
Could someone comment on my approach and suggest a better approach.
Thanks!
use the most probable value to fill in some missing values'...as suggested by Professor Jiawei Han in his book Data Mining: Concepts and Techniques...you document showed some of those...
you may also think on knn imputation using the Gower distance as implemented in the R package VIM. This distance accounts for numerical and categorical variables simultanly.
I once used a Cellular Neural Network to "find" the missing values and the results were nice, better than the results using the mean or a linear relation.
Hy,
You can use the regression model for missing data (climatology, hydrology). The regression can be make in Excel too very easy.
Hi Vinay
use multiple imputation or expected maximisation methods to impute missing values. You can do either of these via few clicks in SPSS.
This pre-edit document can be helpful.
https://www.researchgate.net/publication/262151892_Introduction_to_SPSS
Cheers.
Article Introduction to SPSS
Thank You All,
One question I have with the Regression method or for that matter any other method which predicts the values, wouldnt the use of such a method create a circular dependency when I later use the predicted data to train a model to predict labels?
Vinay
There would be no circular dependency because when you impute missing values into your dataset, a 'new' complete dataset will be generated. From there, you can proceed with regression or other stats of your choice.
- Badges
- Science topic