I too had same question. The answers to my question might be useful.

https://www.researchgate.net/post/Can_we_do_exploratory_and_confirmatory_factor_analysis_in_the_same_data_set2

If I understood your question correctly,  your idea is to test the stability (validity) of your PCA (performed in a "learning subsample") by splitting your sample and run a CFA (with fixed parameters) in the "validation subsample". So we then are in the field of resampling techniques and you could choose a lot of variants of resampling techniques to scrutinize model stability. Among these methods jackknifing and bootstrapping are most widely used. Please keep in mind, that boostrap methods assume random samples.

You could start here with reading: Efron, Bradley, and Gail Gong. "A leisurely look at the bootstrap, the jackknife, and cross-validation." The American Statistician 37.1 (1983): 36-48.

I would answer that you have done this exactly right. I understand that there is no consensus on this point, if for no other reason then there are many ways of using CFA.

In any case, I am presuming that you took the precise factor structure found by PCA and then tested it on the hold-out sample. If you simply carried the number of factors to CFA then it is "ok" but you may get more certainty from bootstrapping. This also assumes that the hold-out sample is large enough. 

Take care,

 If PCA is being used strictly as a data reduction method, what is the goal of the subsequent CFA? CFA is typically used to demonstrate good data-model fit of theoretical relations between variables. However, that doesn't seem to be what the research interest is here. The stated interest is in data reduction, so (in my opinion) it would seem that a PCA would perform this function by itself and that a CFA is not necessary because the theoretical relations between variables are not pertinent (otherwise EFA would be used from the beginning). 

Dear Daniel,

   PCA has limited reliability under many of the circumstances in which it is used. Particularly with less than 100 observations per intended factor and distributions that are less than multi-variate normal the size of the second and subsequent eigenvalues are not entirely trustworthy. This has led to a great interest in bootstrap and other approaches. The interest of CFA in this context is that if the same structure can be found in "independent" data then you have about as strong an argument as can be made. 

Cheers.

Thank you All! Bruce, I appreciate the informed feedback. Daniel: To be more specific, my first goal was data reduction. A lot of items purporting to measure several constructs; I first wanted to scale down those items (PCA), then validate a structural model using those items (CFA). I did indeed use bootstrapping for the CFA (but not for the PCA). Any other information or caveats is appreciated, and thank you all again!

Hi Mr,/Mrs. Wallace,

your idea confuses me a bit. On the one hand you understand that PCA is a mere item-sorting technique with no underlying model that tries to explain the correlations among the items whereas EFA/CFA applies the "common factor model" which is a theoretical model with many assumptions about "reality", "truth" and causality. Mixing a PCA with a subsequent factor model (may it be EFA or CFA) hence makes no sense.

On the other hand you want to do exactly this. 

IMHO, I try myself to avoid PCA in situations in which I want to reduce data but rather tend create sub-indices based on theoretical reasoning. When the sub-indices shall be later included in some regression or structural model, "causal homogeneity" may be (in my point of view) a reasonble "sorting technique". In this case those items are sorted together that can be expected to follow a similar causal mechanism with regard to the IV/DV.

Sorting items only based on some correlation between them (=PCA) is just one possibiltiy - and in my point of view a theoretically blind one. :)

Best,

--Holger

I completely agree with Holger. 

"Validating" a PCA with a CFA doesn't mean anything in my mind. Components and factors are completely different entities -- components try to maximize the total variance explained whereas factors try to maximize the shared variance between variables. You might end up with good data-model after fitting a CFA based on a PCA solution but that isn't evidence that the PCA is providing a good solution because the CFA model is a completely different model (components vs. factors) that is testing a different research interest. 

No way PCA and CFA is conducted using the same data. The Pilot Study data cannot be recycled and used as a field study data.

PCA is conducted to determine the dimensionality of items only. To assess how many items have acceptable factor loading etc and to gauge the Cronbach Alpha for the constructs. The data used was from Pilot Study.

Based on PCA results, you rearranged questionnaire for field study. The data from field study is meant for CFA and SEM.

Dear Zainudin,

there are still some misunderstanding about the differences between PCA and E/CFA models: A PCA creates an artificial composite of the items. Hence, if you change the input, you will gain a different results. The composites (not "factors") have no surplus meaning and no existence beyond the input (items).

Factor models (applied in EFA or CFA) assume a theoretical /causal model that posits that there is some unobserved/latent variable that causes the item responses and - hence- creates the correlation among the items.

Cronbachs alpha makes little sense when applied to PCA composites as it requires essential tau-equivalence (i.e., all items measure one underlying thing AND equal factor loadings). 

I really recommend the following readings:

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219.

Russel, D. W. (2002). In search for underlying dimensions - The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28(2), 1629–1646.

Cho, E., & Kim, S. (2014). Cronbach’s coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18(2), 207–230. doi:10.1177/1094428114555994

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120.

Best,

Holger

If the scale validity is already established in other setting or other language, then it is preferred to do the confirmatory factor analysis. If the result support the original instrument structure, then no need to do the Exploratory factor analysis. However if the CFA did not support the original tool validity, then you have to do EFA.