Hey Kara Ma

as to your first question: if you added another dimension/factor to the original 5, that would make it 6. Why would your reviewer ask you to demonstrate that your underlying construct still consists of only 5? Am I missing something?

as to your second question: CFA seems appropriate since you already have a clear structure in mind.

as to your third question: I am not really sure what nomological validity means. Presumably you are supposed to show that even after having excluded certain items your dimensions still correlate in a meaningful way. Do not quote me on this one!

Best of luck!

Marcel

@Kara Ma I agree with @Marcel on the point of number of factors after adding an additional dimension. As you mentioned in your query that there were already five dimensions (factors) that your borrowed from another research and then added a new dimension (factor) henceforth your observed items should load onto six different factors rather than five factors. It would have been better to understand what the referee exactly wanted if you had attached reviewer's comments with your query.

As far as nomological validity is concerned, it refers to "the degree that the summated scale makes accurate predictions of other concepts in a theoretically based model. The researcher must identify theoretically supported relationships from prior research or accepted principles and then assess whether the scale has corresponding relationships." And apprehensions of the referee are just as you have mentioned that you dropped eight observed variables due to low loadings. You need explain that despite deleting observed items with loadings, number of observed items still left in the model are able to measure the latent constructs and the model holds a good fit.

It sounds like you need to do CFA. First, this will allow you to determine whether the 6 factor model has a superior fit in comparison to the 5 factor model.

As for nomological validity, I prefer the definition "the degree to which a measure assesses the specific construct it is designed to assess, as formulated from the nomological network for the construct being measured." Again, you can test this via CFA, and possibly extend this into a Structural Equation Model to test whether the effects of your unmeasured variable are the same when you use the complete versus the shortened version.

Thank you Marcel Grieger , Imran Anwar , David L Morgan .

Sorry for the confusion, In terms of the factor analysis, I actually replaced 1 with a new dimension (I can justify why I do this with supported LR and logic). I used bootstrapping in smart PLS to evaluate my measurement model, and I think it is a confirmative factor analysis right? But it does not provide information on whether these five factors load on my underlying construct significantly. Should I use Amos or some other software to do CFA?

For the nomological validity, I have checked items that have been deleted, and most of them actually have overlap with those I have kept, does this support that my deletion does not harm the nomological validity?

I have found a way of doing it in Smart PLS! I have created a second-order formative factor (which is my underlying construct), with those five dimensions as first-order factors, then run factor analysis in PLS, the results show all my five dimensions load significantly on my underlying construct. Hope this addresses the reviewer's concern!

Exploratory factor analysis (EFA) is usually used to identify latent constructs or test a hypothesis about their possible underlying structures. Confirmatory factor analysis (CFA), on the other hand, is employed to analyze postulated latent construct structures or to gain a better knowledge of them. Since you had only added another dimension to your construct, provided that you have empirical evidence that the new items have been written to measure particular aspects of your construct, you could go for CFA. If not, you could better start with EFA. You might find Denis’s (2021) book helpful.

Denis, D. J. (2021). Applied univariate, bivariate, and multivariate statistics (2nd ed.). Wiley. https://www.wiley.com/en-us/Applied+Univariate%2C+Bivariate%2C+and+Multivariate+Statistics%3A+Understanding+Statistics+for+Social+and+Natural+Scientists%2C+With+Applications+in+SPSS+and+R%2C+2nd+Edition-p-9781119583042

Good luck,

Your second-order CFA results are very encouraging, and should allay the reviewer's concern on that point. Those results also help, somewhat, with nomological validity, by establishing that the dimensions relate to one another as theory would predict. But, ideally, a demonstration of nomological validity would go further and show that the second order construct, and/or its constituent dimensions, also relate as theoretically predicted with other, conceptually distinct constructs or variables. If your data set includes any such measures, the reviewer might like to know how they relate to your construct and its dimensions. (BTW, I spent an enjoyable and productive sabbatical semester in Dunedin, 20+ years ago, in the Maths/Stats department at Otago. A great place for work, and life.)

Thank you Burke D. Grandjean !

By saying " a demonstration of nomological validity would go further and show that the second-order constructs, and/or its constituent dimensions, also relate as theoretically predicted with other, conceptually distinct constructs or variables". do you mean I should show that the second-order construct is highly correctly to its' constituent dimensions, but all its 5 constituent dimensions should be distinct from each other? Could "discriminant validity" or R2 be used to show these relationships?

Sorry, the term "nomological validity" is so hard to understand, so I am still a bit confused. Here I have another idea. Because the measures in my model are all reflective measures rather than formative measures, and I have checked those deleted items, they are kind of redundant with those kept items, so I think the removal of those 8 items would not affect the essence of my construct. Is this convinceful?

Yes, Dunedin is a really peaceful place for life and study, I enjoy living here, too!

Thank you Mohialdeen Alotumi !

From my experiences, the first look at the reviewers' comments (for a major revision) is always frustrating. I usually read the comments very carefully, and read the paper again very carefully, then take a break. A few days later or so, read the comments again and start doing the revisions. Most of the comments I have received so far are constructive and useful. Best of luck!

It's always hard to know what a reviewer will find convincing.

In my view, validity is a property of a particular measure -- whether it measures what you claim it measures, or something else instead. "Nomological" validity is just one of several ways of showing that it does so -- namely, by showing that the measure is related as theory predicts with measures of its presumed causes and/or consequences. There are other methods for showing validity, called predictive validity, face validity, etc.

In your research, the reviewer seems to be worrying mainly about the validity of your 5 dimensions. Showing that they all relate as theory predicts to the second-order factor (and therefore, they all relates as theory predicts to one another) helps to show the (nomological) validity of each dimension. In other words, it suggests they are valid, using the nomological method for determining validity. Further demonstration of their (nomological) validity could involve showing that they (and/or the second order factor) each relate as theory predicts to measures of other concepts.

I don't think it's necessary, for that purpose, to also establish "discriminant" validity, unless theory predicts that some of the dimensions should relate differently to certain other variables from how the remaining dimensions do. Given that all 5 dimensions load on one second-order factor, there may well be no theoretical reason they should relate differently to other measurable variables. If there are such theoretical reasons, and if you can measure those other variables, then sure -- evidence evidence from the discriminant method of assessing validity would buttress your argument for nomological validity.

Your other argument, about the indicators you've eliminated seeming (to you) to be redundant, is worthwhile, and could be persuasive -- but that argument would not usually be regarded as a demonstration of "nomological" validity. It sounds more like "face" validity (an appeal to common sense), or maybe "content" validity (showing that the indicators you've kept still cover the full range of content covered by the original, larger set of indicators). It's important to keep in mind that these so-called different kinds of validity (nomological, discriminant, face, content, predictive, etc. etc.) are really just different ways of trying to demonstrate validity -- which only means, whether your measure is actually measuring what it want it to measure. Keeping that in mind might help reduce your confusion about "nomological" validity. At least, I hope this is helpful.

Thank you Burke D. Grandjean , this is so helpful!!!

Xuan Van Ha Thank you! Yes, I feel very frustrated in the beginning, but as I dig more into these comments, I have learnt a lot of new knowledge and I think my paper is improved. It is a good chance to have someone really carefully read your article and provide feedback.

Hi Kara,

From what i understand, you encounter two issues, one relating to factor analysis and another about the validity of the modified survey. First, we usually start with EFA (exploratory factor analysis) before CFA, which necessarily means EFA would likely be the intention of the reviewer. In your situation, if the construct is theoretically established as having 5 dimensions and your adding more items leads to it having 6 dimensions. Then, you have to justify the sixth dimension theoretically in your paper. Else, adding items is problematic.

As for validity, it cannot be solely based on statistical analyses. You need to draw on the theoretical framework/theory you're using. My suggestion is for you to check if the remaining items (after your deleting or adding) still measure/cover all five dimensions of the target construct. Also, turning to experts is another option to argue for the validity of your modified questionnaire.

Hope this helps you, Kara Ma

Lem