Hello Ting,

the high correlations among the variables let the reviewer question the discriminant validity of the measures. I guess you ran a exploratory factor analysis. Run instead a CFA which gives you more control.

In the course of the CFA (if it does not fit the data what is very likely), you will have to adapt the model. This may be slightly (eliminated a few times while the factor structure and your original factors still hold) or fundamental. The latter will surely be problematic but I guess the only chance to cope with that is to be transparent and revise the paper substantially. Beyond that, the limitations section has to include some discussion about potential problems of post-hoc adaptations / respecifications about which I have written something here, which might by helpful.

ttps://www.researchgate.net/post/How_many_modifications_can_we_apply_in_confirmatory_factor_analysis

I as a reviewer would, however, have more concerns about potential endogeneity and common method bias. Measuring IV, mediators, and DV with self-report and by the same person is the worst design we can apply (and yes I routinely do it.....)

Best,

--Holger

Holger Steinmetz I am not sure how one would run a CFA to resolve this case. Would you run a single-factor model and hope that the lack of fit would convince the reviewer?

In my opinion, the underlying problem is that reviewer is just plain wrong to criticize an SEM on this basis, but there is not much you can do about that. One possibility would be to switch to the equivalent set of regressions (i.e., a path model). The coefficients should be identical, but you will not get any fit indices for the overall model.

Hello David,

actually, a clean fitting multi-factor model is evidence that it is not all the same albeit the correlations among the latent variables will be even larger (due the correction of error). So this may be a boomerang...Yes perhaps doing a 1-factor model would be a practical means. That does not mean that I think that this is reasonable. Actually, I find the often-seen approach where authors compare their proposed model with k factors with a 1-factor model a nonsensical exercise as lower number than the theoretical model NEVER fit. But perhaps it helps with the reviewer.

As I said, rather then lack of discriminant validity, I would be concerned about confounder bias/common method bias which will be a problem.

I do not quite understand your last point. As I understood, she did not SEM. But even when doing a path model, she will get a model fit unless the model is saturated. His point were the high correlations.

Tings misinterpretation however (I forgot this in my post) is not that the reviewer thinks about multicolinearity (which is not the main issue) but the lack of conceptual differences or even discriminatory measurement. But that's my speculation.

All the best,

--Holger

But how can you run a multi-factor model with 2 IVs, 2 mediators, and 1 DV? All three of those factors would be under-identified.

And yes, she says she did SEM.

this looks like a regression analysis to me with unproblematic VIFs. If they were orthogonal, you would not need a regression analysis. I think you should write politely to the editor explaining why the reviewer is in error. If this does not work go to a better journal.