I have a six-item psychometric scale each item having 6 answer categories. In the pilot testing I used EFA and CFA on the Pearson matrix on a sample n=511 and they supported a bifactorial structure with F1(3 items) F2(3 items) good loadings, no cross-loadings, good fit indices, factors correlated r=0,4. Now, in the validation phase (sample smaller n=200) I decided to us polychoric matrix (ordinal level data) and varied factor diagnostic (BIC, PA, Hull method) and extraction methods (ULS, ML, MWLS, Minimum Rank Factor Analysis) and many combination of them in FACTOR software. Whether I configured the test for 1 factor or two, hals of these method combination suggested 1 dimension and about half 2. Typically when I used PA it suggested 1. But if I used ULS and DWLS they suggested 2 when used with any of the three factor retention methods. Explained variance was about 50% in the 1 factor models and about 70% in the 2 factor models. The CFA done in R lavaan package supported two factors. Finally I did Mokken scaling. It really turned everything upside down. It seems to support a clear unidimensional scale with coef H=0,47. I checked this for the two possible subscales too an they also showed monotonicity with coefH 0,5 and 0,7 respectively. Now I have no idea whether my scale is on dimensional or two. Can you help please?
N=511 is adequate for IRT analysis and recommended as most advanced psychometric technique.
-Matt
Hello Attila,
So, the combination of different samples, different correlational methods, and different extraction methods yields different results; no surprise there!
1. With respect to variance accounted for, more factors works better than fewer factors. However, if you're after parsimony, fewer factors works better than more factors.
2. With respect to "acceptable" model-data fit, it is quite possible that both a one-factor AND a two-factor structure could work.
3. All other things equal, if I had the choice between deciding on "best evidence" between one sample of 511 cases vs. another of 200 cases, I'd opt for the 511 cases.
4. Mokken scaling is asking a different question altogether, but checks on dimensionality are important for the resultant scale to make sense.
Having said all that, my recommendation would be to focus on the approach that makes the most sense for the intended use of your measure(s), and stick with that.
Good luck with your work!
Have you considered a Confirmatory Factor Analysis where you can test the improvement in fit between a one-factor and a two-factor model?
Dimensionality determination techniques and factor revealing techniques differ. I think the findings of Mokken, EFA and CFA analysis do not contradict. Probably you may also get similar findings when you test a single-factor structure with EFA and CFA. Firstly, in this case, it is important to determine the number of dimensions. I also recommend using IRT techniques because of the high sample size.
- Badges
- Science topic
- Similar topics
- Developmental Psychology
- Attachment