I'm working on a set of clinical data and am eager to know if there is any method that can help me to decide whether to keep a variable (feature) or discard it because of the percentage of its missing data. For example, is there any method that can lead us to this conclusion that because there is 50% of missing data in variable X, this variable should be discarded?
I have to note that I'm coding in MATLAB and am not using statistical tools such as SPSS.
Thank you all in advance.
No, it all depends on the structure of your data, how much correlation there is between the variable with many missing data and the other variables and what the reason is for the lack of observations on that variable. On the other hand, if the percentage missing is low, say less than 5%, nearly all strategies give the same answer to your statistical analysis. If in doubt, compare the results from different methods.
By the way note that some people mean by 5%, 5% of the total number of observations, while others mean 5% of the objects/subjects/entities have missing data. Quite a difference
My advice would be. (1) Perform a multiple imputation on your data set and do the analyses you have in mind. (2) Do the same analyses on the data set without the variable in question. Compare the two results and evaluate the difference using your knowledge of the field. (3) Depending on the type of analyses you have in mind there may be a (single imputation) expectation-maximum likelihood method which can carry out the estimation of the parameters and the imputation of the missing data at the same time. Again compare the outcomes and take an informed decision.
No, it all depends on the structure of your data, how much correlation there is between the variable with many missing data and the other variables and what the reason is for the lack of observations on that variable. On the other hand, if the percentage missing is low, say less than 5%, nearly all strategies give the same answer to your statistical analysis. If in doubt, compare the results from different methods.
By the way note that some people mean by 5%, 5% of the total number of observations, while others mean 5% of the objects/subjects/entities have missing data. Quite a difference
My advice would be. (1) Perform a multiple imputation on your data set and do the analyses you have in mind. (2) Do the same analyses on the data set without the variable in question. Compare the two results and evaluate the difference using your knowledge of the field. (3) Depending on the type of analyses you have in mind there may be a (single imputation) expectation-maximum likelihood method which can carry out the estimation of the parameters and the imputation of the missing data at the same time. Again compare the outcomes and take an informed decision.
My suggestion is similar with the the one proposed by Pieter. Perform a classification with an increasing number of samples. If you plot the SEN versus the number of instances you will find the point where the classification is degraded.
It is important to analyse patterns in missing data. There is a difference between data that are missing at random and data that form patterns.
For example, in one questionnaire I analysed, there were clusters of missing items. These items all reflected employment (for example loss of earnings, failure to be promoted, disrespect from colleagues). So they were missing where people were unemployed or had never worked.
Once you understand the processes generating missing data, you can then make decisions about imputation.
hi,In my opinion, to resolve the problem of incomplete data, you can use the relieff algorithm approach. In the following article for an introduction to these methods, has been brought
Good Luck.
- Badges
- Science topic