I have two working memory measures obtained from two psychometric tests. One assessed verbal WM while the other assessed spatial WM. As such, the correlation between them is moderate. Can I sum and average their z-scores together to generate a composite z-score - based on the theoretical concern that they are tapping into the same WM construct.
From a statistical view this is a theoretically interesting question. even though 0.30 is not a very strong correlation. Maybe some of your subjects unconsciously use images like mnemonics for recalling words. Since you have already started analysis, maybe you should be looking at scatter plots with verbal WM and spatial WM on the axes. You could ask how they cluster, etc. However, assuming you have two Z values for the same thing, as when you have run the same test more than once, Stouffer’s Z-score method of combining outcomes (Stouffer, et. al. 1949) is generally the most foolproof approach for combining probabilities (Whitlock 2005):
Estimated Z =(Sum of all n values of Z)/(square root of n)
The Z values should have opposite signs if they are different directions. And I might note that this is more often used for meta-analysis involving literature survey.
- Stouffer, S.A.; E.A. Suchman, L.C. DeVinney, S.A. Star, R.M. Jr. Williams 1949. TheAmerican Soldier, Vol.1: Adjustment during Army Life. Princeton University Press, Princeton.
- Whitlock, M.C. 2005. Combining probability from independent tests: the weighted Z- method is superior to Fisher’s approach. Journal of Evolutionary Biology, 18:1368–1373.
Consider running a factor analysis before compositing these constructs. If the items in these scales load onto independent factors with very little cross loading, then the composite may be inappropriate.
Matt, I definitely know the importance of factor analysis, but in my case, there's only two DVs, so any factor analysis would showcase them as loading on one dimension. The question is whether it is appropriate to create composite score on pure theoretical grounds for all correlations that are moderate and higher.
Lewis, I appreciate you for highlighting the Stouffer’s Z-score method. But do note that it is most robust for high sample conditions. It's definitely worthwhile but it does not address the reason for doing a z-score composite in the first place, which in my condition refers to pure theoretical starting position without resort to any additional and preliminary statistical analysis. I understand that if you take a hardcore empirical stance - that PCA is a necessity for all composite score constructions - establishing a composite by examining the zero-order correlation will not work. However, I read that there are some folks who favor the stance I am making, and henceforth my question.
my answer is yes.
a moderate correlation is reasonable, given the ubiquitous problem of test impurity for "domain-general" constructs such as working memory, attentional capacity, executive functions etc; in particular, in this case, given that one measure is in the verbal and the other in the spatial domain.
perhaps it would be desirable, again because of the impurity problem, to have more than two measures - but sometimes, for practical reasons, one must do with two.
see for instance
https://www.researchgate.net/publication/238043729_Issues_in_Working_Memory_Measurement_Testing_for_M_Capacity
where the median correlation was .25, and related follow-up papers.
Article Issues in Working Memory Measurement: Testing for M Capacity
As an alternative approach try using a Hamming map on the data, assuming both tests were on the same group. This is much more likely to indicate performance correlation.
You can then weight/score/rank each individual as needed.
I agree with Sergio. My answer would also be "yes".
This article may interest you:
http://journal.frontiersin.org/article/10.3389/fpsyg.2014.01475/full
The authors commented on a meta-analysis, which reported small observed correlations between different tests of working memory (r = .20), suggesting that the two kinds of tests measured different constructs. In contrast the authors used multiple tasks of each kind and estimated correlations between the latent constructs, which were quite high (about .70) and even argued that "more highly intercorrelated indicators may provide poorer coverage of the construct space".
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics