Hi Mario,

although I am not using SPSS I am encountering similar problems from time to time. However, I am not entirely sure about the note "Rubin's rule". Are you addressing the derivation of the overall standard error for the estimates?

The differences in your matched data sets actually reflect the uncertainty of the imputations and this effect should increase between imputation variance of estimates. I have to admit, this is not a theoretical rationale, but increasing the variance between imputations makes the results more conservative in terms of significance of predictors. I value this property in the presence of missing data.

Additionally, you might wanna check if this between imputation variance varies with the complexity of your PS model.

Mario, Andrea Lamont, in her not-yet-published dissertation research, encountered this. At my recommendation, she "hard-coded" the matching criterion to be consistent across imputations, though I can't cite literature for you. You might contact her.

As a side note, current recommendations are for well more than five imputations. Five will typically get you unbiased estimates, but you can reduce your standard errors / increase you precision and power substantially by using more. Paul Allison discussed this on his blog a couple of years ago.

Hope that helps,

Pat

https://www.researchgate.net/profile/Andrea_Lamont

http://www.statisticalhorizons.com/more-imputations

This is an interesting paper on the subject...

Mitra, R. and J. P. Reiter (2012). "A comparison of two methods of estimating propensity scores after multiple imputation." Stat Methods Med Res.

Abstract: In many observational studies, analysts estimate treatment effects using propensity scores, e.g. by matching or sub-classifying on the scores. When some values of the covariates are missing, analysts can use multiple imputation to fill in the missing data, estimate propensity scores based on the m completed datasets, and use the propensity scores to estimate treatment effects. We compare two approaches to implement this process. In the first, the analyst estimates the treatment effect using propensity score matching within each completed data set, and averages the m treatment effect estimates. In the second approach, the analyst averages the m propensity scores for each record across the completed datasets, and performs propensity score matching with these averaged scores to estimate the treatment effect. We compare properties of both methods via simulation studies using artificial and real data. The simulations suggest that the second method has greater potential to produce substantial bias reductions than the first, particularly when the missing values are predictive of treatment assignment.

https://stat.duke.edu/~jerry/Papers/smimr13.pdf

https://stat.duke.edu/~jerry/Papers/smimr13.pdf

Thank you all very much for your replies.

Ray, the Mitra & Reiter paper seems interesting, it mirrors the two approaches suggested by Bjoern and I'll read it carefully. Patrick, I take your point about using more than 5 imputations but it was time consuming enough with just 5 so I'm reluctant. Your idea of hard coding the matching so it is the same set of cases for each imputation would certainly get around this as I can just do the matching once and then let spss handle the multiple imputation. But if I use the first imputation to set the hard coding then it may not be the most suitable match for subsequent imputations. However I may not have correclty understood your idea of hard coding. Finally Adrian, by Rubin's rules I do mean the derivation of the overall standard error and I would still like someone to point me to some accessible introduction to these rules.

To give you some more info on my problem, once I've got my 5 matched data sets I run a simple t-test on mean LDL cholesterol. In each imputation the control group has significantly higher mean ( p ranges from 0.00003 to 0.037) but the overall pooled test is not significant (p = 0.057). I think that the reason is that while the mean for the treatment group does not vary very much across imputations (1.864 to 1.871) that for the control group varies more widely (1.930 to 1.989). The standard deviations also vary more for the control group. Now the variation in the treatment group is simply due to the imputation of missing values as they are the same cases each time (there are about 8% missing values for LDL cholesterol) . However the variation in the treatment group is partly due to missing values and partly due to there being different cases matched each time. I suspect that Rubin's rules can accurately take account of the variation due to missing values variation but not that due to the matching. I hope I've expressed this clearly and I look forward to your comments.

Mario,

I was actually suggesting a hard coding (stochastic or deterministic) based on an aggregate of the imputed values for the criterion, so this doesn't get around the time problem. This idea would also require far more than five impuations.

Pat

Hi Mario

Although MI is widely used in missing value analysis. It should also be noted that handling missing values requires more than just imputation. Preliminary assessment of the missing value pattern should be made to determine whether the pattern in missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR). Each pattern has requires slightly different approach.

As much as I praised MI method, I'm also keen on using Expected Maximisation (EM) approach which is also provided by SPSS. The best part of EM is that it runs Little's MCAR test (Little, 1988) to see if the missing values pattern is MCAR or not. 

See Little's wonderful discussion on missing value:

Roderick J. A. Little. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198-1202. doi: 10.2307/2290157

You can also find a section on handling missing values using SPSS from this pre-edit manuscript [link below]

Regards

Saiyidi

Article Introduction to SPSS