For new scales, I think it's good to assess both convergent and discriminant validity if you want those scales to have substantial foundations.

For convergent validity, you need to identify a variable, or variables, with which your scale would be theoretically related, and hope to establish moderately high correlations. You wouldn't want to obtain really high correlations or your scale might be measuring something that already has something already measured adequately.

For discriminant validity, you need to identify a variable, or variables, that your scale would NOT be associated with - and therefore hope to find correlations that were close to zero.

You would certainly hope that the correlations you obtained for convergent validity did not overlap with the correlations you obtained for discriminant validity.

Some time ago I wrote an article about these kinds of things. It's available on RG, but I'm attaching a copy in case it helps. I wrote it because I was concerned about the measurement of people's conceptions about their foot health, but the principles generalise to other contexts.

All the best with what you're doing.

One basic approach would be to use an oblique rotation (correlated factors) and then examine the correlation. Below .6, you have obvious discriminant validity, and above .8, you have problems. For intermediate values, you should examine how the two factors correlate with other (third) variables.

In my opinion, these days whether old or a new scale, reviewers want to see construct reliability and validity statistics. If you are using structural equation modeling in SmartPLS, then:

1. Constructs' reliability is assessed with indicator reliability and consistency reliability

2. and, validity is assessed with convergent validity and discriminant validity.

Indicator reliability is assessed with item loading (you have called it factor loading) 0.6 or higher (Chin, 1998)

Consistency is checked with help Cronbach's alpha (>0.7), however, CR is more suitable (according to Henseler et al., 2009), with value >0.7 (Bagozzi et al., 1988)

Convergent validity is assessed with AVE (Fornell et al., 1981) (recommended value of 0.5 or higher (Hair et al., 2016)).

Discriminant validity can be assessed through 3 ways: 1) Fornell and Larcker Criterion (reference above), 2) cross-loading of items 3) HTMT Ratio (Hair et al., 2016).

Hope it helps,

Ali

References:

W. W. Chin, “Commentary: Issues and Opinion on Structural Equation Modeling,” MIS Q., vol. 22, pp. vii–vxi, 1998.

J. Henseler, C. Ringle, and R. R. Sinkovics, “The Use of Partial Least Squares Path Modeling in International Marketing,” in New Challenges in International Marketing, vol. 20, J. Henseler, C. M. Ringle, and R. R. Sinkovics, Eds. Bingley: Emerald Group Publishing, 2009, pp. 277–319.

R. P. Bagozzi and Y. Yi, “On the evaluation of structural equation models,” J. Acad. Mark. Sci., vol. 16, no. 1, pp. 74–94, Mar. 1988

C. Fornell and D. F. Larcker, “Evaluating Structural Equation Models with Unobservable Variables and Measurement Error,” J. Mark. Res., vol. 18, no. 1, p. 39, Feb. 1981

J. F. Hair Jr, G. T. Hult, C. Ringle, and M. Sarstedt, A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publishers, 2016

Ali Hameed Yousif. Thank you for posting your attachment. It contains some useful information. Are you able to provide a full citation for it, please?

Robt.