Mean of variables of a factor is the most appropriate!

To use the mean of the variables, you first need to assess reliability, which is typically done thought coefficient alpha.

If your alpha is below .8, you should assess whether you have multiple factors (alpha assumes there just one underlying factor). To do that, I recommend using maximum likelihood for the extraction with an oblimin rotation (correlated factors).

Mohsin Iqbal Qazi David L Morgan Having Cronbach's alpha >= .8 is neither a necessary nor a sufficient condition for a single-factor model to be the "true" model underlying the items or for a mean score to be appropriate. A mean score is appropriate only when the variables are essentially tau equivalent in the sense of classical test theory. "Tau equivalent" means variables that reflect a single-factor model with equal factor loadings. The proper way to test this is by fitting a single factor confirmatory factor model with equal loadings and evaluate the model fit.

Cronbach's alpha only makes sense as a reliability measure for essentially tau-equivalent measures, that is, for variables that measure a single factor model with equal factor loadings. If you have a single factor, but unequal loadings, Cronbach's alpha may underestimate reliability. If your items measure more than one factor, then Cronbach's alpha for the full item set is completely meaningless. This is why the "Cronbach's alpha >=.8 rule" is misleading. Item sets for which alpha > .8 can still be multidimensional (reflect multiple factors). On the other hand, item sets for which alpha < .8 can be unidimensional (reflect a single factor).

Christian Geiser I believe we have debated this before. My standpoint is that when alpha is above .8, the only way to get two or more factors is if they are so highly correlated that they might as well be treated as one.

Hi Kujtim Hameli

this is not necessarily true. Computing the mean factor score from your items implies, as explained by Christian Geiser that all items load equally onto the factor. In reality, this assumption is often violated, but can be tested by evaluating the model fit of that factor in a CFA when all loadings are set to be equal (not fixed to a specific value!). If the data fit your model, then tau equivalence - I keep mixing up tau equivalence and congeneric equivalence, but I think it is correct - is achieved and you may proceed to use the mean factor score.

On the matter: https://www.researchgate.net/post/Whether_to_use_computed_variable_from_factor_analysis_in_SPSS_or_should_I_a_mean_of_five_questions_for_independent_sample_t-test_in_SPSS

Best

Marcel

David L Morgan Unfortunately, this is a very common myth. You can have alpha >= .9 and still the scale need not be unidimensional. Please see, for example:

Article Limitations of Coefficient Alpha as an Index of Test Unidime...

https://link.springer.com/content/pdf/10.1007/s11336-008-9101-0.pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.615.4053&rep=rep1&type=pdf

https://rameliaz.github.io/files/kuliah/matrikulasi-mapsi/mcneish.pdf

Hi Christian Geiser

your are a constant stream for excellent references, much obliged. I do not want to force a conversation on this matter, I am not a statistician, but I am literally reading this article at the moment:

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412.

The reasons I am mentioning this is that you could turn the argument of Cronbach's Alpha being unsuitable upside down: Alpha is only suitable if whatever the personality trait is you want to measure is truely unidimensional. That way, you "may avoid violating the equal-item variance [...] property of the essentially tau-equivalent model" (p. 402). Unfortunately, you substitute one bold claim (i. e. equal-item variance) with another (i. e. the trait in question is truely unidimensional).

This is merely my humble contribution to this discussion.

Best

Marcel

Edit: This is my reference list on alpha (and omega). Feel free to share it.

McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412–433.

Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha and omega to estimate internal consistency reliability. Anales de Psicología, 33(3), 755–782.

Goforth, C. (2015). Using and interpreting Cronbach’s Alpha. Zuletzt zugegriffen am 25.01.2022 unter http://data.library.virginia.edu/using-and-interpreting-cronbachs-alpha/.

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105(3), 399–412.

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53–55.

Yurdugül, H. (2008). Minimum sample size for Cronbach's Coefficient Alpha: A Monte-Carlo study. Hacettepe Üniversitesi Eğitim Fakültesi Dergisi (H. U. Journal of Education), 35, 397–405.

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440.

Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria. Organizational Research Methods, 9(2), 202–220.

Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28(12), 1629–1646.

Hulin, C., & Cudeck, R. (2001). Why conduct a factor analysis and then compute an alpha? Journal of Consumer Psychology, 10(1/2), 58–59.

Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8(4), 350–353.

Green, S. B., Lissitz, R. W., & Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37(4), 827–838.

Cronbach, L. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.

Nelson, L. Alpha and omega reliability: A working paper dated 12 October 2021. Zuletzt zugegriffen am 24.01.2022 unter https://www.lertap5.com/Documentation/AlphaOmegaSpotTheDifference.pdf.

Marcel Grieger I'm not familiar with the Dunn et al. (2014) paper that you cite but the essential tau-equivalence model of classical test theory (CTT) does not require or imply equal item variances or equal item residual (error) variances as appears to be claimed in the quote that you cited. The requirements/assumptions of the model of essential tau equivalence are:

1. essential tau equivalence (true score variables tau_i, tau_j for different items can differ only by an additive constant c_ij [i.e., intercept]: tau_i = tau_j + c_ij) --> this assumption implies equal (unstandardized) loadings, but not equal variances or equal error variances (it also does not imply equal standardized loadings because the item residual/error variances can differ)

2. uncorrelated error variables

Equal variances and residual (error) variances are required only by the stricter CTT models of parallel and essentially parallel variables. For those latter more restricted models, we would use the simpler Spearman-Brown formula for composite reliability rather than the more complicated Cronbach's alpha formula.

McDonald's omega is a composite reliability formula suitable for congeneric measures (measures with unequal unstandardized loadings). The congeneric model is the least restrictive model of CTT that allows items to have different loadings, intercepts, and error (residual) variances. It still implies unidimensionality (a single common factor).

I am not at all arguing that Cronbach's alpha is generally unsuitable or not useful. I think alpha, when properly applied, is very useful as a measure of composite reliability. My point is that it should be applied in the correct way. That is, when you can show that your test score variables are in line with the models of essential or strict tau equivalence, then alpha is the proper composite reliability index and quite meaningful/useful to estimate scale score reliability. In my experience, there are quite a few test score variables that satisfy the assumption of (essential) tau equivalence (and having essentially or strictly tau equivalent measures is also a desirable goal for test construction). In that situation, alpha is quite useful to determine composite reliability (the reliability of the sum or average of the test components).

However, it is incorrect to say that a high alpha value (e.g., >= .8) is indicative of a unidimensional scale. That is not necessarily the case. You can very easily generate (simulate) data from a multidimensional scale (i.e., a factor model with multiple distinct factors) and find alpha > .9 for the sum of all (multidimensional) items.

In other words, unidimensionality cannot be tested by computing alpha. Unidimensionality should be tested using confirmatory factor analysis or methods of item response theory. Note that the parallel, essentially parallel, tau equivalence, essential tau equivalence, and congeneric CTT models all imply unidimensionality (a single factor model), but different measures of composite reliability.

Christian Geiser I can only get the abstract for the article you referenced, but it appears to be about "item homogeneity," not multidimensionality, which is also the issue raised by Marcel Grieger. In this regard, alpha is treated as a lower bound for reliability because differential weighting could yield a high level.

My position is that if you look at the CFA equivalent to simply adding (or averaging) the items, it is very difficult to allow for either major differences in the loadings or multiple factors, once that lower bound reaches .8.

So, in theory, you can improve on alpha by either differential weighting of the items or using two or more dimensions, but in practice, once alpha gets high enough, neither of these approaches is likely to make a meaningful difference.

David L Morgan My position is that Cronbach's alpha only makes sense as an index of composite reliability under the assumptions of (1) unidimensionality (single factor model) and (2) equal item weighting (in summary, if essential tau equivalence holds). Therefore, it is unclear to me what you mean by "you can improve on alpha by either differential weighting of the items or using two or more dimensions." My position is that it would not make sense to compute or interpret alpha under either one of these conditions (differential weighting or multiple dimensions). I would use alpha only after finding that (essential) tau equivalence holds for my variables; in other words, if a single factor model with equal loadings fits the item set. In the other situations (differential weighting or multiple dimensions), I would consider alpha to be meaningless and/or potentially misleading as an index of reliability.

As I said, once alpha gets above.8, it is very difficult for either differential weighting or multi-factor solutions to improve on that.

Think of it in terms of a CFA, where you would be comparing the fit of a single factor solution with uniform loadings and an alpha of .8 to the best fitting model with either differential loadings or multiple factors. The fit from a model with an alpha of .8 is so tight that there is very little room for improvement with either of the best fitting alternative models.

Another way to say this is that once alpha reaches point .8, that means the average correlation among the items is so high that: a) all the weights have to be quite high, and b) multiple factors would be correlated near 1.0.

I suppose there might be a counter case if you reach .8 with a moderate average correlation and a multitude of items but I don't have any experience with the CFA model for that (I do survey research, so asking 20+ questions is not feasible).

Mohsin Iqbal Qazi did you get the answer or help from the above heated debate on what was asked :) Share the nature of your variables and your objectives of the study so that workable suggestions could be shared with you by some expert.

More importantly, can you share why you need to perform t-test? Good luck.

I agree with Muhammad Zia Aslam that the Cronbach's alpha discussion might have been somewhat distracting from Mohsin Iqbal Qazi 's original question, although I don't think it is necessarily heated. Nonetheless, this discussion is important because this is a source of frequent confusion.

So back to the original question: (When) should we use equal weighting (a simple, unweighted sum or mean score) versus a weighted factor score?

The psychometric answer to this question is very clear. When your variables are in line with the classical test theory (CTT) model of essential tau-equivalence (or, equivalently, for binary items the Rasch [1-parameter logistic] model of item response theory), you can and should use equal weighting. Essential tau equivalence implies that items measure a single common factor with equal unstandardized loadings and uncorrelated error variables. A testable implication of the model of essential tau equivalence is equal covariances of all variables (items) in the population. This implication can be tested by fitting a single-factor confirmatory factor analysis (CFA) model with equal loadings. If this model fits well, then equal weighting may be justified.

If a single factor model with equal unstandardized loadings does not fit, but a single-factor model with unequal unstandardized loadings (i.e., a congeneric model) is acceptable, then a weighted score makes more sense. If no single-factor model fits (i.e., when the items measure more than one factor), a single score (weighted or unweighted) may not be meaningful at all.

My point in my initial response to the original answer by David L Morgan was that Cronbach's alpha >= .8 is neither necessary nor sufficient to justify using a weighted score because the suggested cut-off value of .8 (or any cut-off for alpha) is in no way indicative of whether or not essential tau-equivalence holds. That is, you could have Cronbach's alpha < .8 for five (or more or fewer) items that are perfectly in line with the model of essential tau equivalence. In other words, even if you find alpha < .8, this does not necessarily mean that you shouldn't use equal weighting. It could be that your items are still essentially tau equivalent (making equal weights justifiable)--even when their composite reliability is < .8.

On the other hand, if you find alpha >= .8, this does not mean that equal weighting must necessarily be justified because the items may not be essentially tau-equivalent or even unidimensional.

The reason is that there is nothing in the computation/formula of alpha that "tests" the underlying assumption/implication of equal item covariances in the population implied by the model of essential tau equivalence. You can compute alpha (and it can be high or low) even when this assumption/implication is violated.