I'm not aware of any problems or issues that would arise from having an "imbalanced" (unequal) number of indicators (items) per factor. I therefore do not think that this needs to be justified beyond saying that the scales consisted of x number of items. It is rarely the case that scales consist of exactly the same number of items. We would not "trim" items from some of the scales just to have the same number of indicators per factor. Having more (high quality) indicators per factor tends to be better than having fewer indicators.

What is more important is that all your items are high-quality indicators, meaning that they are unidimensional for a given factor and highly reliable (as indicated by high standardized factor loadings).

Hello Lily,

Did you start the CFA proposing three factors having 3, 3, and 8 items/variables, respectively? Or was it the case that you had equal numbers of variables for each factor at the outset?

If, as part of the CFA, you ended up jettisoning items/variables so as to arrive at a model which did demonstrate adequate model-data fit, it's possible that you have overfit the model to your data set. In that case, the real concern would be whether your final model holds with new data, and not the fact that factors have differing numbers of variables associated with them.

Good luck with your work.

Thank you very much for your response, Christian Geiser

Hello, David. I had equal number of items per factor at the beginning of the scale development.

Lily, the fact that only three items are available to measure a construct means that this construct is quite narrowly defined compared to a broadly defined construct measured by many items. I think that you should have developed more items than you thought would be needed to define the scales. Generally, experts recommend that the number of newly developed items should be three to four times the number of items that will ultimately be needed. The reason for writing more items than needed is that items do not always function as well as one would hope. As Bandalos (2018, p. 50) stated: “In fact, items have an annoying tendency to completely ignore the purpose for which they were developed and to function in ways that are utterly at odds with the scale developer’s intentions.” I guess that this is your problem and that’s why using a larger item pool would have been beneficial. If the newly developed scales do not accurately reflect the constructs of interest, any inferences made based on these measures are suspect.

When you found that the CFA with your intended assignments of the items to the factors resulted in a poor model fit, did you then run an EFA and used these results as the basis for running the CFA with modified assignments of the items? However, when you change the assignment of the items to the factors, the CFA loses its confirmatory character and becomes an exploratory analysis.

You should now have a critical look at the constructs measured with different items than originally intended. Did the constructs change their meanings? For scales consisting of only three items: Do they still measure the intended constructs, but only a narrow part of them? If yes, this would be a justification for using only three items to measure a construct.

Reference:

Bandalos, D. L. (2018). Measurement Theory and Applications for the Social Sciences. New York, NY: The Guilford Press.

HTH, Karin