Dirk, I don't know how extensive is the magnitude of your missing data, but if it is not too extensive, I would consider a casewise deletion of all the subjects with missing data. If you still have a large dataset for running your factor analysis after that, I will proceed with factor analysis. However, if you have something like a third of your subjects with missing numbers, consider calculating doing mean substitution or the computation of expected estimates. After doing that, I recommend you running the factor analysis AND the IRT analysis based on the probability values. Compare both sets of data, if they match each other, then the estimated numbers/fillers can be considered dependable.

You are taking on something that is very challenging because of the amount of missing data and the pattern of missingness.  Also, the number of observations per test item will differ.  I imagine that some will have very low frequency.  I would recommend first creating a large person by item matrix and then sort the item dimension by the frequency of observations.  Then, plan on doing your analysis on the items that have a frequency greater then some number like 100.  I would prefer 200, but I don't know how large your data-set is.  Then, check the dimensionality on the smaller set of items.  Using a program that will handle the missing data is a necessity, but it is also possible to impute the missing values, but I would not impute the missing item responses with a unidimensional IRT model.  That will cause underestimation of the dimensionality.  When I have done this, I have computed the correlation matrix for the items and then imputed the correlations for missing item pairs.  The resulting correlation matrix may need to be smoothed to avoid negative eigenvalues.  Bock has a procedure for doing this.  Then, I prefer doing parallel analysis on the correlation matrix.  This all needs to be done cautiously because of all of the issues that are involved.

Thank you all for your answers. I have a very large itembank (> 200 items), however, all items do have >300 responses. But I still have a very large person-item matrix with a lot of missings.

In response to Prof. Böhnke and Reckase, I took on your suggestions. After some reading I found out that IRTPRO actually does EFA with FIML. Estimation took a while but results seemed about right.

I also used Mplus to do an EFA with FIML, however, with only 2 factors it took about 5hrs to run. It was nice to see that results from IRTPRO and Mplus where quite similar though. However, somehow it feels kind of weird to me that there is more missing data than observed data for most items: therefore I also conducted analyses (as Prof. Reckase suggested) on a subset with relatively larger shares of observed data and used the imputation method in Mplus (with the WLSMV estimator). Unidimensionality seemed to hold.

Prof. Reckase, when you say you "have computed the correlation matrix for the items and then imputed the correlations for missing item pairs", could you elaborate on this? How would you do this (e.g. using which program)? 

First, a comment.  An important number is the number of pairs of scores that go into the correlation between items.  I expect that some easy items have very low frequencies when paired with hard items.  Correlations are not very stable unless the number of paired observations is over 100.  That is where the smoothing of the correlation matrix is important.  I expect that there are some pairs of the 200 items that are not administered together so it is not possible to compute the correlation between the item scores.

The attached paper gives the details about how we approached this problem.

Good morning Prof Mark D. Reckase,

I hope you are available. I am working on some adaptive test CFA and in a situation which items don't have pairwise correlations at 0.5% of the correlation matrices; also 98/(42*42) pairwise counts are less than 100.

Would you please share how you imputed and smoothed the covariance matrix? Do you recommend any software? That will be enormously appreciated.

Regards,

Xiaoyu

Hello Dirk,

You can check out a study we did last year regarding imputing responses for datasets generated by a CAT: http://www.jattjournal.com/index.php/atp/article/view/158509

My opinion is that using theta/ability estimates to generate responses (assuming the unidimensionality holds) pushes the data to be unidimensional. So, you will have to use a different imputation approach by taking the theta into account as a predictor but not for directly calculating the probability of success in each item.

Thank you for sharing, Okan! It's a very advanced study.