You need to get a random sample of your data. If there is some systematic bias in the missing data (for instance lab results for a gene assay that only applied to males), excluding rows with missing data will bias your results. (all females will be dropped out of your analysis).

You need to decide on how large of a subset of data you want to analyze. If I am reading this right, you have 203000 rows of data with 150 columns. If you are subsampling because the data matrix is too large for the memory on your computer or perhaps it is taking too long for complex processing, I would try and sample as large as possible. One possibility would be to select every 4th row for a dataset that is 25% of the original size.

Mind you you still may have some bias and you will need to run your descriptive statistics to make sure everything looks good, and if possible, make sure it follows the same distribution as the full set.

Do you want to sample the data by row only, by column only, or by both? You can use SAS to do this.

Thanks. I want to do it by rows. I use R. Any suggestion ?

if your matrix is a

apply( a , 2 , function(x) all(is.na(x)) )

get columns that all of arrays are NA

apply( a , 1 , function(x) all(is.na(x)) )

get rows that all of arrays are NA

so you can adjust !is.na matrix for example:

a

You need to get a random sample of your data. If there is some systematic bias in the missing data (for instance lab results for a gene assay that only applied to males), excluding rows with missing data will bias your results. (all females will be dropped out of your analysis).

You need to decide on how large of a subset of data you want to analyze. If I am reading this right, you have 203000 rows of data with 150 columns. If you are subsampling because the data matrix is too large for the memory on your computer or perhaps it is taking too long for complex processing, I would try and sample as large as possible. One possibility would be to select every 4th row for a dataset that is 25% of the original size.

Mind you you still may have some bias and you will need to run your descriptive statistics to make sure everything looks good, and if possible, make sure it follows the same distribution as the full set.

Thanks everybody for the suggestions.

Hello, Tridib.

I agree with Phil. You could do a systematic sampling with random start in the rows.

Rows with large number of missing values must be represented in the new sample to avoid a bias (you may define a threshold of missing values and count all rows over it: that percentage must be the same in the new sample)

Don´t forget to check the subsample with different statistics and compare results with the original dataset. It is very easy to bias the sample with these procedures.

Regards.

I suggest that you first filter your data to remove all rows containing NA.

Following that you may apply random sampling, bootstrapping or whatever sampling technique you like.

I suggest this because I think that the missing values are actually meaningful to you.

In case that the columns with missing values are not important to you, I suggest just to get rid of these columns before any other analysis.

Sometimes, it is the unexplained which answers the questions we have so be aware of that, but in most cases, you will be fine with filtering your datset and then perorming your data mining approach.

Due to recent events, I also would like to suggest to check the accuracy of your input ... can you actually be sure of the features you calculated or are they just 80% accurate.

I suggest resampling methods to ensure that your random draws are balanced out (i.e.take a few random samples without replacement; implying that no random samples can be identical). E.g. if your dataset is as follows (size 5):

1 - A - d - g

2 - D - d - l

3 - W - l - l

4 - X - s - m

5 - NA - NA - k

you could draw the following sampes of size 2:

Sample 1:

1 - A - d - g

3 - W - l - l

Sample 2:

3 - W - l - l

5 - NA - NA - k

Afterwards, you apply your analyses on each random sample separately, and combine the corresponding results to get final parameter estimates and SE.

In comment to author above ("I suggest that you first filter your data to remove all rows containing NA."): removing rows with missing values will introduce bias if missing does not occur completely at random (very likely). It is not recommended, and multiple imputation strategies are generally preferred.

I agree.

In some samples, however, n/a is unwanted or unexpected and then any such instances should be removed. In my opinion the tone set by Mr. Dutta implies that he does not expect missing values.

In any case, as Thomas suggests, you should try different sampling methods (random, leave one out, ...). Actually, it would also be good to do some cross validation (but I am sure you must be aware of this) and pair it with sampling to get different 'opinions' on the model you learn during your data mining approach.

.

@Thomas, thanks a lot for explaining it with example. I am wondering if you could give your opinion on what to do in the following case when, I noticed that out of 150 variables, the last 10% of the variables contains almost 90% NAs. Is it ok to delete those NAs or the variables ?

Thanks,

If 10% of your data contains 90% NA imputation will go along much uncertainty. I suspect deleting corresponding cases will ulyield similar results (particularly point estimates) although you may get a better grasp of uncertainty by performing multiple imputation.

Border Pairs Method (BPM) provides a quality selection of representative samples. For samples without missing values​​, this selection works very well and by the way even determines how difficult are the learning patterns.

Conference Paper Border Pairs Method – Constructive MLP Learning Classificati...

I suggest you to use reservoir sampling:

http://en.wikipedia.org/wiki/Reservoir_sampling

It is a very efficient algorithm (o(n)) to sample a very large set. The principle is simple and smart. You use a reservoir, which has the size of the wanted sample:K. It is initialized with the first K elements of your dataset of size N.

Then you read each element of index i>K of your dataset. At each step you replace a randome element j of the reservoir with a probability K/i. A the end you obtain a sample of size K, where each element of your dataset has a probability K/N of inclusion.