The recommended procedure for Cronbach's alpha is to add together all of the raw variables (assuming they are all scored on the same response format; if not, then normalize them with a Z-score).

There have been so many questions here about Cronbach's alpha that I have compiled a set of resources at:

https://www.researchgate.net/post/How_can_one_help_people_create_simple_scales_from_Likert-scored_items

Paul, Cronbach's alpha is used for scales consisting of several items. It does not make sense to multiply variables with regression coefficients.

Performing a CFA is the best solution. Of course you cannot compare the coefficients using different methods. So just go ahead with your CFA.

Regards,

Karin

David, Karin, Thank you very much. It is indeed possible to compute composite reliability in CFA. However, I still wonder if this outcome can be compared with the alphas of the original composite scales.

Paul, if you need to compare the reliability of this measurement instrument with the alphas originally reported, then you need to also calculate alpha. You may argue that this was done solely because of reasons of comparison, and then report omega if the assumptions of alpha (unidimensionality, tau-equivalence, i.e., equal factor loadings, and uncorrelated error terms) are violated. 

Regards, Karin

Thank you Karin, that was really helpful. I already made that comparison, but I realise that I will need to give the composite reliability as well, since the factor loadings are different indeed.

If the loadings are quite different, you might want to consider dropping items that have low loadings. This is common in Exploratory Factor Analysis, but if you jump straight to CFA, it often does not get considered.

In SPSS, and probably other programs, you can also examine this as an option within the Alpha (Reliabilities), where you ask for "Alpha if item deleted." In particular, when an item is only weakly correlated with the other measures in a scale, the Alpha can go up when you omit that item (because the average correlation goes up).

Of course, driving Alpha up is not necessarily the best path to follow, since it is by definition only a "lower bound" for the reliability. But it is also true that items with low loadings, by definition, do not make much contribution to the underlying unmeasured variable.

Thanks David. My primary concern is to assess the reliabilities with all items of the original scale  included. I did so at different levels of analysis,  I suspect higher level variance, which may inflate the alpha's. So MLM would be the best strategy to find that out.

I'm unclear about how you got from alpha to Multi-Level Modeling. Can you tell us more about the measurement model you are using?

That is quite a story. The point is that I am testing five scales (in one model) for four different levels of analysis (these are nested). Cronbachs Alpha seems all right for three scales, for all levels of analysis. For two of these three the composite reliabilities - as obtained by a CFA are just below .7 - so in fact not reliable. This is a bit surprising, as Cronbach is supposed to under-estimate under these conditions (covariance between factors). 

Are you saying the alpha for that scale was lower than the reliability that you got for the same set of items in the CFA? If so, one possible explanation is that most specifications of CFA include correlations among the sub-scales, while alpha either treats each sub-set of items separately or consolidates the full set of items.

But I'm still unclear what you mean by "nested." in Multi-level modeling, it is the respondents who are nested at different levels of analysis (such as students in schools). It sounds more like you are specifying a second-order factor analysis, where the sub-scales are treated as indicators of a single, higher order factor.

If that is indeed the specification of your model, then the lower loading for the one sub-scale does not really mean that it is unreliable; instead, it indicates that it doesn't share as much variance with the common underlying concept as the other sub-scales do.