Dear @Jaclyn What I could from your query is that you have developed scales for four different constructs and four of them are not leading to any higher order latent construct thereby meaning that they are different from each other thus not highly correlated. If I could get it correctly, then I would refrain from using composite score (variable) made of observed items for all four constructs rather treat each one of them as a distinct construct and accordingly making composite variables with their respective observed items. Wishing you best.

If you are constructing an instrument to measure "a psychological construct" there must be a higher order factor - based on the theory.

If it does not show in your data you need to reconsider different things:

- your operationalization (item wording might be biased and inaccurate)

- your scale construction sample (sample composition, sample size might be insufficient)

- your statistical analysis (exploratory or confirmatory factor analysis might be done inaccurately).

Hello Jaclyn,

beyond what my predecessors said: Of course you can use a multi-faceted composite for research or practice (it is done all the time when using "indexes"--for instance, in economy) but the ontological properties change. While measurement of anything implies the attribute you measure exists independently from your measurement procedure, the computation of indexes just refers to the mixing process of things into the index and has no surplus meaning. Practically, if you get some estimates involving the index, interpretation is ambiguous.

Here is some lit

Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155-174.

Edwards, J. R. (2001). Multidimensional constructs in organizational behavior research: Towards an integrative and analytical framework. Organizational Research Methods, 4(2), 144-192.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203-219.

Cadogan, J. W., Lee, N., & Chamberlain, L. (2013). Formative variables are unreal variables: Why the formative MIMIC model is invalid. AMS Review, 3(1), 38-49. doi:10.1007/s13162-013-0038-9

Guyon, H. (2018). The fallacy of the theoretical meaning of formative constructs. Frontiers in Psychology, 9(179). doi:10.3389/fpsyg.2018.00179

Beyond that and more concerning higher order factors, I would rather tend to start with measuring each factor as validly as possible and then try to estimate some potential higher-order factor. In this regard, there was some interesting criticisms on these (which I share)

Lee, N., & Cadogan, J. W. (2013). Problems with formative and higher-order reflective variables. Journal of Business Research, 66(2), 242-247. doi:10.1016/j.jbusres.2012.08.004

In this regard, please be aware that a higher-order factor is just one possible hypothesis that tries to explain the correlations between primary factors (while there are often others, for instance, a set of confounders having effects on the factors or factors affect each other). Hence, beware of following the factor-reflex. Hence, to add one thing to the list that Nicola provided, I would add "there is no higher-order factor" :)

HTH

Holger

This essay may be of interest for your research:

Preprint Item Response Theory and Its General Total Score

Thank you all for sharing your feedback! This is all helpful.