Validity is often assessed quantitatively in terms of correlation coefficients, and yes, it can involve inferential statistics to test whether the relevant correlations are "statistically significant." For example, in Campbell and Fiske's (1959) multitrait-multimethod (MTMM) approach to construct validation, correlations are used to examine convergent and discriminant validity. One of Campbell and Fiske's (1959) criteria for convergent validity is that the relevant monotrait-heteromethod correlations be statistically significant thus involving inferential statistics.

I would agree that having validated a measure in one population does not mean that the instrument is also valid in another population. Each distinct population may require a separate validation.

Reference

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016

@christian, does it mind if researchers simply apply convenience sampling technique Just for assessing the validity?

I hate to be a pain but I would like to remind you that experimental instruments need to have validity checks and that is a whole different thing

So when you use the term please remember that there are natural science researchers here. For specific examples see calibration methods in R For the ideas and methods. Best wishes David Booth

Dear Fajar Awradi, your question is quite useful for less exprerienced researchers.

However, it does appear to me that the way you worded that question indicates that you are only confirming what you know already.

Certainly , as previous contributors have indicated, it is obvious that any measurement scale is not less important that what is being measured with that scale.

Yes, it does apply