Hello,
I am in a peculiar situation where I need help from experienced researchers.
I am creating a new scale (I know the procedure). After running an EFA (extraction method: principal axis factor (PAF), parallel analysis for identifying the number of factors and oblique rotation), I received a two-factor solution:
Sample: 168
Factor 1 (6 items with loadings between 0.5-0.9, alpha: 0.96, 4% variance)
Factor 2 (10 items with loadings between 0.6-0.8, alpha: 0.97 , 24% variance)
Then I went on and collected a new dataset and ran CFA. Model fit indices are in the attached image.
In your opinion, which solution is better? Or none of them are good?
Kind Regards,
Ali
Ali Farooq, I hope I understand your results and situation correctly. If so, I'd start by saying that I think your cutoff for the SRMR is a bit high. I prefer to use .05 to .08 as being acceptable, and < .05 as preferable.
I think that your cutoff values are OK for the other goodness-of-fit indices, but your results seem to fall short of being satisfactory according to those criteria - particularly the CFI and TLI.
I hate to say this, but I think that neither set of CFA results is good.
However, I'm also suspicious of CFA results when there are only two factors and more than about 6 items on each of them.
Apart from that, your two factors seem to account for percentages of variance that are too small. I like to see at least 50% of the variance accounted for in the retained factors - and up around 70% if possible.
So all up, things don't look good to me. Sorry, Ali . . .
I agree with Robert Trevethan. The fit of your CFA models does not look too good. You can examine standardized residuals and/or modification indices in CFA to explore which unmodeled effects may cause the overall misfit. For example, there may be cross-loadings for certain items that are not equal to zero in the population (those cross-loadings are implicitly constrained to zero in CFA, and that may cause misfit if they aren't zero in the population). Or certain items might share specific variance, causing error variables to correlate (by default, error variables are assumed to be uncorrelated in most standard CFA models). Such residual associations can also be detected by looking at standardized residuals and/or modification indices.
Christian Geiser, thanks for your endorsement. I have to confess that your knowledge of stats is obviously a lot more sophisticated than mine, but I have just noticed something I overlooked before:
Ali, you said that your six-item factor had an alpha of .96. For me, that raises the possibility that those six items are highly redundant - at least in the eyes of your participants. A 10-item factor with an alpha as high as .97 rings the same kind of alarm bells for me - i.e., the prospect of participant-perceived redundancy.
Might that be producing your unsatisfying results?
Incidentally, I do support you (Ali) in terms of what you did in your EFAs, e.g., parallel analysis followed by an oblique rotation.
Thank you, Robert Trevethan and Christian Geiser for sharing your opinion and guidance.
Indeed, CFA results are not encouraging (putting it softly for me :D ). I will check for cross-loadings in CFA and see if that makes any difference.
Robert Trevethan: On your point regarding variance. What can be the remedies for the problem? Is it because of participant-perceived redundancy or something else?
Further, just out of curiosity, I ran CFA using the same sample I used for EFA. I understand this is not suggested (running EFA and CFA using the same dataset). The results (attached) look better than the results I shared earlier.
Assuming things do get better after what has suggested by Christian, what could be the way forward?
- I should collect a new dataset for CFA?
OR
- I should go back one step before the EFA and check for the items that are redundant.
Thank you both, once again, for your time and valuable suggestions.
Ali Farooq, I think the way to avoid redundancy is to look very carefully at the construct you want to measure and then to ensure that the items you use capture all aspects of that construct without being repetitive.
With your data, I'd now look very carefully at the interitem correlations to see which items might be most related to each other. Then I'd delete one item from each pair of highly correlated items.
Apart from that, your new results are interesting. They certainly look better than your previous results. However, because they are different from the previous results, I think they demonstrate that it's dangerous to conduct analyses with one sample and regard the outcomes as definitive. Change to a sample that might have many similarities to an earlier sample, and the results change!
I wonder how often researchers acknowledge this kind of situation might exist. Instead of acknowledging it, I guess many researchers regard their findings as somehow "final". Sure, results can differ from sample to sample, but when samples are ostensibly very similar to each other, it's interesting that each yields different results. When are those differences trivial, and at which point are those differences "problematic"? That's a nice issue.
- Badges
- Science topic
- Similar topics
- Psychology
- Mental Processes
- Learning