Dear Andres

I would ask you if the instrument you used for the analysis is already exists or you developed it? Then did you conduct CFA and EFA for the same sample or two different samples? And what about your sample size and the number of variables (items) you have and what was the value of KMO and Bartlett's test? The GOF indices for the two factors were acceptable or not, therefore sometimes you might have acceptable GOF indices but other model have a better indices so here you must make a decision about the difference between the "acceptable " and the "better"

Hello,

I fully understand your concerns. However, is it newly developed scale? Then EFA, but if this is established measure then CFA. 

If it is well established and you have satisfactory fit indices then you do not need to reverse to EFA. 

Regards, Witold

Dear Andres,

I agree with Witold. Also, CFA is a more rigorous, and theoretically driven approach than EFA. I will follow CFA (that constrians the factor loading for each indicator to a specific factor) rather than EFA (that allows all the indicators to all potential factors). Even for EFA, you can try diffrent estimaton methods and different rotations.

Thank you for your recommendations. The instrument used is a translated and adapted version of an existing instrument. Hence, there is a solid theoretical base and CFA would probably be most appropriate to use. There is however indications from other studies that the three factors of the original instrument may be collapsed into two factors. But, the use of EFA vs CFA is not really the question here, I conducted both, mostly out of interest, and were slightly confused by the contradictory results.

Also, here is additional information that I should have included in the first post: both analyses were conducted on the same sample including around 5000 participants. The instrument contains 14 items and KMO=0,90, Bartlett's χ2=22324; df=91; p

Dear Andres

did you try to check the modification indices, maybe two of your factors have a covariates, so try to do it you also might have a non significant value of chi square during to less parameters. So check the third factor you have extracted use EFA how many variables were significantly loaded for it and check them qualitatively.

if you need any help Please just ask

The modification indices you might check for the three factors model use CFA

The three factor model will almost always produce a "better" fit simply because of its additional complexity, and with 5000 cases, that improvement will be statistically significant. The key question is whether the fit is enough better to merit retaining the third factor. Since the fit for the two-factor model is only marginal with some of your indices, I would go with the three factor model.

FYI, if you want a strong comparison between the EFA and CFA models, you should run the EFA which is most directly comparable to the statistical model in CFA, which should be Maximum Likelihood (presumably with oblique rotation, if you allowed the factors in the CFA to be correlation).

And the discussion is not entirely pointless, since will need to decide which of the two CFAs to use.