Usually if you are using factor analysis to refine a pool of items, you need to remove one item at a time and rerun the analysis after each change. Note that you should be removing items not only on the basis of factor loadings but also giving consideration to item wordings (in case the items you remove changes the meaning of the scale).

Note that many people recommend against conducting CFA on the same dataset you have examined with EFA.

Ben's advice is good.  The only time I run EFA is if I am developing a new scale.  If I am using an established scale that claims to have 4 factors I run CFA using R or Lisrel or AMOS preferably in that order..

As per other suggestions, you should remove items one at a time, using empirical information (item loading strength, cross-loadings, etc.) and rational decision-making (maybe of two items you are thinking of removing, one is very similarly worded to another item - if so, remove this item first as its wording is redundant). After removing one item, re-run the analysis on the remaining items as the loadings and other parameters will be different after you have removed an item.

If you have a large enough sample, consider splitting it, running EFAs and modifications on one half of the sample and at the end, run a CFA on the modified items in the second half.

Great advice David!  Reasoned approach to EFA is a good idea!

In case it was not clear from David's advice, I would like to stress that you should never run an EFA and CFA on the same data. Indeed splitting the data is the way to go, otherwise you come into the realm of double dipping and thus bad scientific practice!

As per my answer on another similar thread, the terms EFA and CFA are really philosophical (describe a mindset) more than methodological or mathematical.

If the OP has insufficient sample size to split and run a cross-validation of the modified model, there are still advantages and reasons to run a "CFA" (LISREL/AMOS-type analysis) after running model modifications.

Running a "CFA" at the end will enable a maximally restrictive analysis (doesn't allow cross-loadings and can allow specific interfactor correlations), which will provide valuable information not available via an "EFA". However, if the OP does this, it is important to recognise what you do have (a test of a more restrictive model, with helpful fit indices, based on a set of exploratory modifications), and also what you do NOT have (i.e., in this situation you should avoid describing the final CFA results as a "cross-validation").

The OP might even consider using "CFA" analysis methods (LISREL/AMOS) to conduct the exploratory portion, deleting items based on modification indices for cross-loadings and correlated errors, etc.) and then re-running after each item deletion. 

So my approach to this is:

1. CFA should be used to 'confirm' the posited factor structure

2. if this does not fit your data, randomly split your data set paying attention to the subject to item ratio

3. run the EFA on one sample. decide on the factor loadings and consider cross factor loadings.

4. Run a reliability analysis as well and consider dropping item-total loadings

I would also recommend that you run your factor analysis again. I would also suggest that you randomly split your sample run and EFA and then a CFA.

I propose to redo the analysis. For confirmatory analysis, fresh dataset will be better.

Dear folks, allow me to present a slightly different perspective. I don't think that pure replication of a factor model is useful to investigate if the structure is correct - may it be with a new sample or random split of the actual sample (cross validation)

a) Cross validation (CV)

The initial goal of CV was to investigate if "some results" may be a sign of idiosyncratic features of the actual sample and thus, a result of sampling error versus features of the population. However, by dividing the sample randomly into two parts you get two equivalent samples. Hence any differences between the sub samples result from failures to randomize. As the sampling error has already been occureed (because the overall sample was drawn), CV by random split is absolutely of no value.

b) Pure repetitions of tests of causal structure

A factor model is a causal model that posits that there is a latent variable causing a bunch of items. As anybody experienced with CFA/SEM knows, models can fit although they are wrong. There are two reasons for such cases:

1) The model only fits in the actual sample and is fitted to idiosyncratic features (see above). In this case, drawing a new sample will reveal that it is wrong (again CV is useless).

2) Far more important is a fit to the stable part of the covariances (which will occur far more often in IMHO). In these cases, drawing a new sample and simply repeating the wrong model will lead again to a fitting model.

Think of a two factor model which is causally misspecified and tested as a one factor model. This model will not fit and the modification indices will tell you that adding an error covariance will lead to a excellent model fit. Now take a new sample an re-test the modified (wrong) model with an error covariance, and this model will fit again and again and again.

Summary: 

a) cross validation by randomly splitting is useless and drawing a new sample just tests for sampling error and idiosyncratic features and

b) repeating a model test only tests for sampling error not if the model is systematically (across all possible samples) misspecified.

The better choice would be to incorporate something new in the re-tested model that serves as additional challenge for the model - e.g., instrumental variables, outcomes, similar but different variables and so on.

My 2 c....

Holger

You only need to do EFA for  Pilot Study data. Remember the number of respondents required is minimum 100. The purpose of EFA is to determine the dimensionality of your measuring items in order to collect data in the field study.

Once you obtain the field study data, you cannot re-run the EFA. Remember - no EFA for field study data. Otherwise you have to re collect data in field study. The field study data is meant for CFA (Confirmatory Factor Analysis) only.

Yes, you should rerun EFA as the parameters will change after removing one or more items. If you then want to confirm the final structure that resulted from EFA using CFA, you need an additional sample. A good idea is to randomly split your whole sample into two halves and use the first one to run EFA and the second one to confirm it with CFA. Of course you should have a large enough sample to do that.

Hi folks,

with all modesty and respectfulness, I would nonetheless disagree with some statements :)

1) Simply replicating a factor analysis (may it be EFA or CFA) does not provide strong evidence that the modelled structure is correct. Remember that a factor model is a causal model (as any other SEM) that posits the existence of common causes (which the factor(s) represent) that influence the response on certain indicators. 

If the model is systematically misspecified although it fits (or even more, if the model does not fit), this misspecification will work also in a further drawn sample, because the misspecification is systematic. A pure replication, however, provides evidence that the results are not only fit to the idiosyncratic characterisics of the sample. This may be some worth.

2) Cross-validation (randomly splitting the sample) has - please correct me if I miss a point - absolutely no value. By randomly splitting the sample you get two identical halfs - why should you expect differences? And how does a result in the second half provide evidence about the structure of hte factor model found in the first? Even testing the idiosyncratic aspect aforementioned does not work, as the sample has already been drawn and the sampling error has been occured. 

The conclusion is that results of E/CFA have to be replicated by adding things - like connection with other validating variables, instrumental variables or causal restrictions that the model must pass if the structure is valid. In this regard, I think that the masses of papers out there simply publishing results of factor analyses (e.g., in scale development) are not very helpful.

Having said that, beyond all factor modeling, it is sometimes very useful to expand the perspectives on the relationships between measured variables and latent variables a bit. Here is a nice paper that does that:

Hayduk, L. A. (2014). Seeing perfectly fitting factor models that are causally misspecified: Understanding that close-fitting models can be worse. Educational and Psychological Measurement, 1-22.

http://epm.sagepub.com/content/early/2014/04/06/0013164414527449.abstract

Best,

Holger

I noticed there are comments about deleting items one at a time and 'rerunning' it after doing an intitial EFA.

Can I clarify if people mean rerunning using an EFA or CFA?

If it were an EFA again, would I:

1) highlight any potential cross-loadings and factors with less than 3 loadings again? Or

2) would I continue to delete the items one at a time from the first EFA and make notes?

If I were to run a CFA, would I:

1) Note down all items that could be deleted from the EFA (after looking at cross-laodings, and items on factors which have less than 3 loaded), delete them, and then run it again through a CFA?

EFA procedure is to determine the latent structure of the items and to filter the poor loading items. CFA is to validate the measurement model of latent constructs. As for rerun and not depends on the researcher.

what about if I did the EFA and I found that I should delete two items. so if I delete these items in CFA I receive bad results but if I keep them I got good results.

so what should I do? delete them or keep them.

Hello Mamdouh,

can you be more specific? Optimally attach a figure of your EFA and CFA and a bit of context. And present more infos about model fit and so on.

Best,

--Holger

Hello Holger,

thank you for your response. My research in the area of creativity and innovation.

The attached file shows you the EFA rotated component matrix with cross-loading items(TC3) and (TC5), and two figures of CFA results. one with keeping the items and the other with deleting them.