Tabitha Osler Those are good and difficult questions.

1.) Regarding your "acceptability" question, a lot of people would argue that such a modification/additional parameter may be OK if it is not just data-driven (i.e., driven by the residuals or modification indices), but also theoretically justifiable. For example, when items are very similarly worded, when they tap into a similar facet or subdomain of a factor, or when they share common method or other specific variance, there may be good theoretical and/or substantive reason to include an error covariance or a residual method factor for those items. The decision should not be made purely based on the residuals or modification indices. There should also be clear theoretical-substantive reasons for including the additional parameter. Ideally, the modification should also be replicated/validated with new/independent data since it was derived in an exploratory fashion (i.e., based on peeking at the data "post hoc" after fitting the model).

2.) Your question regarding the scale score computation is more tricky since, as you correctly point out, the error covariance implies that these items are measuring an additional factor above and beyond the intended "common" factor. Maybe the safest thing then to do (especially if the error correlation estimate is substantial) would be to base the scale scores on estimated factor scores in the model version that includes the error covariance parameter so as to take this effect into account.

Thank you so much for the time you have taken to help me with such a useful answer!

I am wondering whether I should perhaps rather delete one of the items that is resulting in the high residual covariance score? It just seems unfortunate to loose this data that has high face validity and contributes to the high internal reliability score.

Hi all,

some time ago there was a discussion here on RG on post-hoc error covariance, that might be helpful:

https://www.researchgate.net/post/Theoretical-assumptions-for-correlating-errors-during-SEM#view=5f35166edc797978d53fef05

This should also help with your decision whether to exclude some items (which is ok for one reason that I discuss but not for the other :). If this stays unclear, we can discuss this further.

All the best,

Holger

Tabitha Osler It may not be necessary to delete any of the items. The questions that you should examine are whether the items

1) have something in common substantively above and beyond what they share with the other measures of the common factor (e.g., method variance, similar wording, specific content variance) and/or

2) whether they are better viewed as measuring their own dimension or construct (i.e., whether they are indicators of a more distinct/unique factor compared to the other items that load on the common factor.

These are primarily theoretical-substantive questions that you can often address by looking at the item wording/content and your theory about the factor. In addition, it is useful to examine the magnitude of the error correlation estimate and the magnitude of the standardized loadings on the common factor. If the error correlation is strong and the loadings are weak, this may be a sign that the items in question may measure "their own" more distinct factor. Perhaps this factor should then be modeled as a separate substantive factor if that also makes sense theoretically. In contrast, if the error correlation is modest and the loadings on the common factor remain strong after introducing the error correlation, this would indicate that the items are primarily indicators of the intended common factor.

Thank you both for your helpful responses. I am so grateful!

If I keep the items and use covariance to get better model fit - how do I then adjust the scoring for the scale?

Christian - you mentioned using the factor scores but I am unsure how I would do this? Also when you say factor scores, do you mean factor loadings? (Sorry for my ignorance - I am brand new to factor analysis!). Do you perhaps have a Youtube video on this?- I don't want to use up too much of your time.

Thanks again!

Tabitha Osler Many software programs for confirmatory factor analysis (CFA) will allow you to estimate and save individual factor scores as new variables to your data set. These scores can then be used in the same way as a conventional scale score (e.g., sum score of items). One advantage of factor scores is that they take the different item loadings into account and that they would allow you to also account for the error correlation parameter when estimating the factor scores.

I'm not sure which software program you're using. In any case, the program manual/user's guide should have information regarding options for estimating and saving factor scores.