I partitioned my data into train and test sets. I did outlier detection and missing value imputation in my training set. Then, I built a model on this set. I am not sure what I should do about the missing values and outliers in my test set? For example, in case of missing values, should I ignore them in the test set? Impute them using the same algorithm? Fill them with a simple estimate, like mean (mean estimated from the test set or train set??)
I think, this thesis might help. http://www.cs.toronto.edu/~marlin/research/phd_thesis/marlin-phd-thesis.pdf
Firstly, "missing values" and "outliers" are two very different aspects which should be considered distinctly. In the case of outliers, I would say that it depends on your data modeling objective. maybe the outliers are precisely the values of interest (for example, fraud detection). In the case of missing values, if you have several input variables, maybe only one of the variables suffers from missing values. Then it depends if that variable is highly relevant to the data model. if not, maybe you could remove the variable altogether. Another criteria is what percentage of missing values is acceptable. Substituting the missing value for the average of the non missing (or the mode if categorical) is reasonable, up to a certain point. Overall, I would say that missing values are a data preprocessing step and not a modelling problem.
Going to agree with David's overall distinction drawn, but disagree with mean substitution. It really messes with the normality assumptions, as well as reducing association with other variables.
.
when you learn a model on a training set and deploy it on a test set, you implicitly assume that both sets are drawn from the same distribution P(X,Y), X being the explicative variables and Y the target variable
if you learn a model on a pre-processed training set (imputing missing values for instance), you should deploy it on a test set pre-processed in the same way
(which means in particular that you should not use your target when imputing on the training set : as you do not know the targets on the test set, you cannot impute on the test set if the imputation on the training set uses the target !)
.
For further reading, you can check out Chapter 5 (Data Quality) of my latest book, which considers two visions of data quality (as input to a data model): relevance and reliability.
"Commercial Data Mining, Processing, Analysis and Modeling for Predictive Analytics Projects", Morgan-Kaufmann 2014. 1st Edition
http://store.elsevier.com/Commercial-Data-Mining/David-Nettleton/isbn-9780124166028/
You can use machine learning method to predict missing data !
Article Machine Learning Based Missing Value Imputation Method for C...
- Badges
- Science topic
- Similar topics
- Computer Science
- Data Mining