Prudence -

Some might suggest multivariate multiple regression, but I would think you might not have enough data. Of course variance matters a lot in that regard. But looking up "convergent validity," as that's a new one to me, it sounds like all you care about is the correlation between pairs, not prediction. That's interesting because if you needed to predict, collinearity between predictors can be problematic. So, it's good for you to do due diligence, but maybe, this time, 'everybody' is correct. :-) It depends on how you plan to use these variables.

- Cheers - Jim

Dear Prudence Butler

The correlation which you are discussing is basically "Discriminant validity" which is assessed either by Fornel Larcker Criteria, or by HTMT ratio, Cross loading.

However, the Convergent validity is assessed based on the followings:

1. Factor loadings and VIF for outer model

2. Cronbach Alpha, composite reliability, and average variance extracted.

Convergent validity is established when following conditions are met

Factor loading > 0.70, VIF < 5, Cronbach Alpha > 0.70, Composite reliability > 0.70, and average variance extracted > 0.50

@Butler convergent validity is established between observed variables and their respective latent variable. While running a CFA model, factor loadings from observed items to latent variable are averaged and it should be not less than .708 which accounts to Average Variance Extracted (AVE) above or not less than 50% (.50) thus establishing that observed items are able to reflect the latent variable sufficiently. As suggested by @Zahir correlations are considered for ensuring discrimination validity using Fornell and Larcker or HTMT criteria.

Thank you Zahid Bashir and Imran Anwar

I'm very inexperienced so need to clarify some things. Please correct me if I'm wrong but the methods you've described would show whether there is sufficiently high loadings b/w the observed variables and the latent variable WITHIN the measure but my understanding of convergent validity is that you're testing whether there is acceptably high/strong r/ships BETWEEN measures. So, if you were to build a SEM, you want to know the highlighted bit rather than the stars (see attached pic). I think I grasp concepts better with pictures. If you can point me towards any good videos that demonstrate these methods, that would be much appreciated.

Prudence Butler, addressing your question can be a bit tricky because different methodologists / statisticians mean different things by terms such as convergent and discriminant validity.

From what you have written in your question, I assume you're working from the perspective of a nomological network involving relationships between latent variables, or constructs (see Cronbach & Meehl, 1955; Strauss & Smith, 2009) rather than from a perspective that involves the relationships among items and factors within a single scale (see Hair, Black, Babin, & Anderson, 2014).

Here are the relevant references for what I've written above:

Cronbach, L., & Meehl, P. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 4, 281–302. https://doi.org/10.1037/h0040957

Hair, J. F., Jr., Black, W. C., Babin, B. J., & Anderson, R. E. (2014). Confirmatory factor analysis. In J. F. Hair, Jr., W. C. Black, B. J. Babin, & R. E. Anderson (Eds.), Multivariate data analysis (7th ed., pp. 599–638). Pearson.

Strauss, M. E., & Smith, G. T. (2009). Construct validity: Advances in theory and methodology. Annual Review of Clinical Psychology, 5, 1–25. Article Construct Validity: Advances in Theory and Methodology

Prudence Butler my respected colleagues Robert Trevethan , Imran Anwar and Zahid Bashir have already guided you on the (Reliability) Validity issues of a latent variable and have suggested you valuable sources to follow. However, would you like to share why you need to establish ONLY Convergent Validity of your "Measure of Interest" (8-items)?

Are you developing a new measure for a construct already having 3 existing measures in literature? Are you adopting/adapting these 8 items of your measure of interest from literature? Do you mean you have 3 other variables in your study when you mention "I have scores for three related measures"? I think these questions need to be addressed before discussing about sample size etc. Tq

Hi Muhammad Zia Aslam . Thanks for your message. I am using a slightly modified, pre-developed measure- it has been around for years but it hasn't had its psychometric properties tested in the specific patient population I'm interested in.

I am testing the reliability (internal consistency & test retest) and validity (face, construct & convergent). Because the measure relates to a unique psychological construct, there are no 'gold standard' measures to test criterion validity. So, we chose three measures that we hypothesized would relate to our measure in specific ways. All measures were taken at the same time, which precludes the ability to test predictive validity. If I had a larger sample size, and perhaps some non self report measures, my understanding is that I should be looking at using a multi-trait multi method approach but I don't believe I can do this with a small selection of similar measures and a small sample. So, I have to use the most suitable approaches with the data I have.

Robert Trevethan was correct in saying that we are testing convergent validity from a nomological framework point of view.

Prudence Butler, from what you have written in your most recent post, you are on the ball with your thinking. If I might offer one bit of advice, it would be not to go down the path of the multitrait-multimethod approach - or, at least, not to regret not being able to use that approach. I could be wrong, but that approach looks enticing but, in reality, is probably a bit of a nightmare. I tried to use it once, and simply gave up.

Even working out what criteria to use for convergent and discriminant validity is difficult: How high can a correlation be in convergent validity before you're essentially measuring the same construct but in a different way? How far above zero can a correlation be and still indicate discriminant validity?

I have seen research in which very similar, and moderate, correlations have been used to claim both convergent and discriminant validity - even within the same article. Evidently many reviewers are happy with mention of the words convergent and discriminant, and they don't much care whether those words are used appropriately.

I notice you've mentioned test-retest reliability. You're probably aware that it can be assessed in several ways - all good for particular purposes. Forgive me if I seem to be pushing my own barrow, but an article I wrote about 4 years ago might be helpful if you're considering intraclass correlation coefficients. Here's the reference:

Trevethan, R. (2017). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology, 17, 127–143. https://doi.org/10.1007/s10742-016-0156-6

In another article, just published, a colleague and I used four different ways to assess test-retest reliability. In case you're interested, here's the reference for that one:

Trevethan, R., & Ma, K. (2021). Influence of response-option combinations when measuring sense of efficacy for teaching: Trivial, or substantial and substantive? Frontiers in Education, 6, Article 723141. https://www.frontiersin.org/articles/10.3389/feduc.2021.723141/ful

All the best for your research.