Are you sure you mean "item" as in a single question and not "latent factor/scale/variable"? You can test for convergent validity by correlating two (or more) latent variables within in the CFA framework. The resulting correlation coefficient will give you an idea as to whether your construct is reasonably similar, but no identical to others. Thus, correlation and CFA are not mutually exclusive.

Best

Marcel

Carlson, K. D., & Herdman, A. O. (2012). Understanding the impact of convergent validity on research results. Organizational Research Methods, 15(1), 17–32.

Confirmatory factor analysis allows you to examine measurement models and to estimate latent correlation coefficients that are corrected for measurement error. Measurement error can lead to an attenuation (underestimation) of correlation coefficients among observed variables (e.g., items), making the correlations look too small. CFA allows you to correct for that by specifying measurement models and estimating latent correlation coefficients (correlations between true score variables that are free of random error). So when everything goes well, CFA allows you to obtain more accurate estimates of convergent validity coefficients.

There are also specific CFA models that allow you to quantity convergent validity in terms of variance components based on a multitrait-multimethod measurement design.

Hi all,

I would rather think that mere correlations (as used in nomological networks) are a rather weak sign of validity as measures (especially in psychology) always correlate. In addition, there is always a lot of confounding going on that often blows up a correlation (sometimes it weakens it).

A more advanced way would be to adopt a functional perspective on validity and think "how does the construct work and, hence, how would my measure behave if it is a valid measure of the construct". That is, embed it in a full SEM with theoretical restrictions (=also strongly connected to validty!). Even better is to do some experimentation, of instance with a simple scenario experiment, as in

Antonakis, J., & House, R. J. (2014). Instrumental leadership: Measurement and extension of transformational–transactional leadership theory. The Leadership Quarterly, 25(4), 746-771. doi:https://doi.org/10.1016/j.leaqua.2014.04.005

The typical psychological validation study is

1) "construct the measure" (in the worst case create a pool and start with EFA and even worse PCA)

2) Measure some validiation scales

3) Create a correlation matrix

4) Conclude validity based on the strength (or worse significance)

I may be wrong but I very seldem see the list being enlarged by "include similar measures to test against lack of discriminant validity". Often, the overlap of measures (even with items beeing the same in several scales) is astonishing.

All the best,

Holger

Holger Steinmetz

pardon my ignorance, but how does the "functional" approach you suggested differ from correlating two either similar (convergent) or distinct (discriminate) measures? I believe I would have followed the same four steps you mentioned.

Thanks a lot!

Best

Marcel

Hi Marcel,

a correlation has nothing functional--is it just a correlation. As you of course know, two variables can strongly correlate and the correlation being completely due to confounding. A correlation is non-restrictive--a structural effect in contrast may imply a) adjustment for other variables and b) exclusion restrictions is much more restrictive and thus, informative--in case it works :)

The strongest evidence for functionality is when you manage to manipulate the whole thing and it behaves as expected from a theoretical perspective. Of course, this can still be wrong but the degree of "validity evidence" is stronger. Hence, you can locate these pieces of information (i.e., correlation, regression, SEM with restrictions (also IV estimation) and experiments) in a dimension of validity evidence.

All the best,

Holger