My first question would be: Why do you think you need a method factor? What kind of a method effect is this factor supposed to represent (e.g., the effect caused by using negatively vs. positively worded items?)?

Many loadings on your target (substantive) factor are not significant, indicating that the model may be misspecified and/or overparameterized and that the target factor is not easily interpretable. In Geiser, Eid, and Nussbeck (2008) we provided a critique of the random intercept factor model and presented an alternative approach for handling method effects that can be used to deal with item wording and other method effects in a more theory-driven way (e.g., by specifying a residual method factor only for the negatively worded items), in which both the target and the method factor have a clear a priori psychometric meaning and interpretation. See:

Article On the Meaning of the Latent Variables in the CT-C (M-1) Mod...

An additional issue in your application may be that you have a large number of items (20). The items may not be unidimensional even after accounting for wording or other method effects. There may be additional substantive factors (a single substantive factor may not be enough).

@Christian Geiser Thanks a lot for your thoughtful answer and sorry that I didn’t mention the reason for choosing the random intercept approach!

Reason 1:

I choose random intercept mode because in this case all the twenty items are in the same keying direction, where CT-C(M-1) approach is not applicable as far as I understand. But if it is incorrect, please point me out! And I'm not sure whether restricted CT-C (M-1) can be extended to multidimensional occasions?

Reason 2:

Based on my personal experience. I do agree with you that the underlying assumption of the random intercept approach is too strong: the existence of individual differences in scale usage (e.g., response style) but these differences only exist between respondents and keep constant across items within respondents (because all item loadings of the random intercept method factor need to be set up as 1 or constrained to be equal).

But I have privately explored different (E)SEM approaches (CT-C(M-1), random intercept, exploratory structural equation modeling) used to account for method effects when developing a new scale, I (maybe only intuitively) found that:

a) using any approach will significantly improve fit indexes and do clarify the factor structure compared with using conventional EFA;

b) the differences in the factor loadings and conclusions of using different approaches are negligible, maybe it is because my scale actually turned out unidimensional according to all approaches. But when someone is exploring a more complicated factor structure, using different approaches would lead to different results and conclusions);

c) most importantly, this is the second reason why I choose the random intercept approach (at least in this case because I choose ESEM when I explore the assumed complex structure of the new scale). I found that the validities of the method factors are more or less different. By mentioning validity, I personally mean the correlations of the factor scores of the method factor to other constructs. If the approach is valid to only account for method effects, then correlations of the factor score of the corresponding method factor to substantial constructs should be as small as possible. If the scores of substantial constructs also derive from the same approach, their correlations should be as close to zero as possible. I tried to figure out this with the data on my hand but the scores of substantial constructs are only sum scores for convenience… and I found that only the factor score of the method factor of the random intercept model has the smallest correlations to other constructs (below .10), factor scores of the method factors by using other approaches would significantly correlate with some constructs (and all the method factor scores almost perfectly correlated with each other, ranging from .66 to .99). Although the magnitudes of the correlations are quite small (usually below .20), they indicate that method factors of other approaches more or less unintentionally account for variance unrelated to method effects.

Again, I do agree with you that the assumption of the random intercept approach may be too strong and a bit unrealistic, but my data implied that the variance accounted for by the method factor of this approach is more ‘pure’, which is shown by the negligible correlations with other constructs.

Bo Wang : Thanks for your reply! This is an interesting topic.

My personal problem with the random intercept factor model is that in this model neither the target (substantive) factor nor the random intercept factor have a clear psychometric definition and meaning, especially when many items do not load significantly on the target factor. It is then very difficult to figure out what is actually represented by each factor. This seems to be the case in your application as well where the loading pattern is inconsistent and you have to rely on external correlates to figure out what each factor may represent (substantive variance? method variance?). I prefer measurement models in which all factors have a clear definition, meaning, and interpretation so these issues can avoided. Like you wrote, models like the random intercept model can always improve model fit, but that does not mean that they provide meaningful latent variables or that they help you better understand your measures and constructs.

In this case, I'd say I'd probably prefer a simple-structure factor model with multiple correlated factors (if possible) to reflect the multidimensionality of the items. Such simple-structure measurement models with correlated factors are often easier to interpret than models that contain what I would consider "obscure" method factors. Have you tried an exploratory factor analysis to see whether you can identify multiple substantive factors or subfacets?