I am working on a sales data for one of our customers and performing some exploratory analysis.
We have almost 100 million rows of data. As this is a sales data/ purchase data hence the gender is a very critical variable.
Of these 100 million almost 10 million rows do not have gender information.
Please suggest on techniques to apply on these missing variables as we cannot drop these variables due to high contribution.
Also i read about KNN algorithm can be used for categorical variables. I have not applied this technique before hence request to please advice on this technique or any alternate approach for the situation.
Regards.
If you need absolutely these data, you could run a PCA or build a PLS model using NIPALS algorithm (for exemple with SIMCA software). This algorithm does not need estimation of missing data and keep all observations. A postrieri, you can estimate missing values by inverse calculation via matrix of scores and loadings.
Just create a third category for Gender- 'missing.' Think on this outside of statistics: it makes sense to impute an account balance for someone with missing data; it does not make sense to arbitrarily assign a gender to someone.
If I understand correctly, you have 10% missing values of a binary covariate. In this case, I would recommend using the mice package in R which gives you the option to impute not only binary but all kinds of variables on different scales. Think of it as some kind of logistic regression which learns a model based on the fully available 90% of your rows and afterwards predicts the 10% missing values based on this model. It is much more sophisticated than that because it uses chain equations and takes into account the covariance of the imputed values to produce the most plausible imputations. The only issue might be the sample size as mice typically is not very fast. If this is a limiting factor, consider subsampling your data.
In your case, missing makes up of 10% of the total of the records, so you can use these missing as added category for the gender. If not, association rule is also alternative choice to make a guess about gender of customers based on existing variables.
kNN algorithm can be used to determine missing information as described. The use of kNN algorithm for such scenario results in classification problem. You will need to split the 90 million vectors where you do have full information into two groups based on gender information(Male and Female). These form training samples, then assign each of the 10 million vectors where you do not have gender information to either of the two group based using k nearest neighbor rule (kNN) as implemented in R package class. To know more about kNN, read about it here http://ieeexplore.ieee.org/document/1053964/
Alternative techniques include support vector machine, rank distribution classifier (see http://link.springer.com/chapter/10.1007%2F978-3-319-22404-6_15) among others
For each given question of interest, you might first use the 90 million
vectors where you do have full information to determine the extent to
which the missing data plays a critical role.
http://link.springer.com/chapter/10.1007%2F978-3-319-22404-6_15
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics