Although several approaches were developed for handling missing data, majority of them are only suitable to restore incomplete patterns which have random variation. When random variation exists beside shift, trend, systematic and cyclic behavior in which the order of data also contains valuable information, what is the best approach to handle missing data?
Perhaps you could find appropriate regressor data to 'predict' missing data, along with estimating the variance of the prediction error.
Thanks dr. James, Would you please explain more about the suggested method. I have a data stream and the task is online monitoring of the data. Due to the importance of each data I have to replace the missing data with suitable estimation of it. However, I do not know in advance whether the developing pattern in data is trend, shift or perhaps random. In this case, if I estimate the missing data with the mean of the historic data I made a big mistake if the pattern is non random.
Razieh,
I totally agree with Mr. Malone, single imputation methods are not recommended nowadays. I'd like to suggest the investigation of some machine learning methods for multiple imputations, we have a plenty of them.
To help the choice of the "best" imputation methods for your problem we should know more about your data. Is it composed only by numerical attributes? What are the missingness statistics (% of instances with Missing Values, % of MV in the whole dataset etc)?
Regards,
Fábio
Thanks dr. pat
How should I start learning about "model-based multiple-imputation"?
If I generate a model for shift, another for trend, one for cyclic,..... How should I decide which one is the best in a developing dataset?
Thanks Fabio, My dataset has numerical values with 20% missingness. The data is not necessarily random, I expect to see some shift, trend and cyclic behavior in certain ranges, but the dataset is still developing, so I do not know the pattern in advance. the key is to identify early even when some part is missing.
Razieh -
I basically agree with Patrick, though, as might be inferred from my latest response to your other recent question,
https://www.researchgate.net/post/How_do_you_differ_missing_data_with_ignorable_mechanism_and_missing_data_with_non-ignorable_mechanism#5615f1126307d9596e8b4569,
there are some uses for single imputation, if you are careful to pay attention to variance.
Cheers - Jim
PS - However, it sounds like you are short of information to do anything here. It sounds like all you have is a 'developing dataset,' and it will take some patience to obtain what you need.
Dr. Pat,
I have started reading the book you suggested and I found it very useful. Thank you!
There is a question in my mind that I could not find an answer to it, till now.
There are many variables that MAY influence the outcome which is gone missing. However, I have not recorded any variables but the outcome (main variable of interest). The problem arises when some of the values in outcome are missing. Can i impute missing values in the outcome from the observed values in the outcome, itself?
Razieh -
It seems that perhaps your application here is for a univariate time series for which you have no other data to do anything more complex. If your current value is "missing," then that sounds like you are really saying that you need a forecast. And because you have no clearly defined patterns established in your time series, the simplest format for "exponential smoothing" may be as good as you can do, at least for now. It will put the strongest influence on the most recent data that you do have.
Jim
Razieh,
From my experience, the best approach for missing data imputation is Random Imputation of Missing Data (RIMD).
This algorithm and other are included in the attached paper.
It is so easy to implement in R.
I can guide you about that if you prefer to use R.
Best,
Conference Paper Comparative Statistical Algorithms for Imputation of Missing...
depending on the data type and data distribution
sometimes the data is time series data or .....
I think the most common way is a regression
(depending on the sensitivity of data and article type )
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Statistical Software
- R Statistical Package