Brown (2015) highligts that you need a good justification to add correlated errors between some indicators of your construct and these correlation, that you should not only model them to reach the common cut-offs for good model fit and to be consistent with the rule you apply.

He presents several potential reasons why such correlated errors can occur, like shared method variance due to different wording compared to other indicators, or specific item content(even you might not had this assumption a prior, i think you can defend them, if such reason is plausible). 

I can really recommend his chapter: 

Brown, T. A. (2015). Confirmatory factor analysis for applied research. Guilford Publications.

Hope that helps, BEST, Manuel

Manuel,

That is an excellent reference, thank you! Brown (2015) does give a full coverage of how to handle the correlated error. I am grateful for your kindness in responding. I also found an article I found that seems to give similar advice.

God bless you always,

Janet

There are two ways you can handle correlated errors during CFA. One is to remove the lower factor loading item between the two. And another is to constrain the effects of correlated errors through co-variation between the two measurement errors.

Jonathan,

I am grateful for this clear and specific response to my question. I will perform the correlations and determine the impacts as you described. Thank you for the article references also. I do have the Gerbing & Anderson article and reviewed the residuals from the data correlations and created a second order factor for unknown group specificity error that we are going to test. The information you provided gives me a good start for further analyses. I am grateful for your kindness in responding. PS My sample size is 351 with a 95% confidence level and 5.61 confidence interval. The survey has 20 items. I retained 19.

God bless you,

Janet

Zainudin,

I appreciate your response. I had thought to remove a factor however the possibility that their may be another model that is obscured by the unexplained error correlation above random made me hesitate. I did adjust the model for correlations between three pairs of  items though afterward  it was explained that I needed a strong theoretical rationale to do so and I can't really explain the non random error on a couple of pairs of items. I appreciate your kindness in replying. I will try Jonathan's suggestions and let you know what I discover. I am also going to run an E/CFA as described and recommended by Brown (2015). See previous reference. I will compare a variety of models with indicators as described in Brown as well. I am excited to discover more through these new to me methods.

Kind regards,

Janet

Well,pure variables have loadings of .30 or greater on only one factor. Complex variables may have high loadings on more than one factor which makes interpretation of output difficult.So rotation may be necessary to reduce the number of complex variables and enhances interpretation. If items have dual or triple loadings greater than .3 on more than one factor, such items must be interpreted with caution because simple structure is not apparent.However, if items were designed to measure a single construct,therefore it is expected that the factors extracted would be highly correlated.

Hi Janet,

Another possible approach is to develop a second-order factor analysis, if the correlated errors exist among sets of items and the sets could be considered as sub-scales by theory. Good luck!

Jing

Jing,

We have completed our analysis and used hypochoric correlations to get a unidimensional second-order factor with three subfactors.Thank you to everyone for your kind responses!!

Janet

Dear Janet,

Can you please help me how you got unidimensional second order factor@ janet?

Razia,

I attach the article we published addressing the issues in the correlated error between subfactors on the What's My School Mindst? scale. This should explain how we handled the issue. Let me know if you have any questions.

God bless you always,

Janet

Thanks alot dear.

On occasion, questions arise about correlated errors (or disturbances) in confirmatory factor analysis (CFA) and structural equation modeling (SEM). From my general knowledge of the topic and a brief literature search, I believe that correlated errors are acceptable and indeed necessary in some models. Correlated error terms, however, should be used conservatively and not just to improve model fit. SEM, after all, was conceived as a theory testing framework, so adjusting the model based on modification indices is in a sense antithetical to the whole endeavor. Ideally, then, an investigator would prespecify any correlations among error terms based on his or her complete theoretical model, which should include the core theory, auxiliary theories, and measurement theories (Duhem, 1914/1991; Worrall, 2003). That is, the model should fully address not only the theoretical relationships among (latent) constructs but also measurement error and, in particular, method variation.

        Some methodologists nonetheless suggest that item errors should never be correlated, but these concerns usually arise during an introduction to CFA and SEM. In general, and for those relatively unfamiliar with a topic, it is often better to start with general truths than all the exceptions. This is why some CFA and SEM instructors recommend avoiding correlated errors entirely. After all, haphazard correlations among errors can cause serious problems with interpretation of the model and vastly reduce the likelihood of replication due to overfitting. Bollen and Lennox (1991) cautioned that "although correlated errors are possible among items using similar wordings or appearing near to each other on the questionnaire, for purposes of discussion we assume that errors are generally independent of one another."

        There are a number of examples of correlated errors in CFA and SEM, such as multitrait-multimethod (MTMM) models. MTMM models involve two sets of factors. The substantive factors explain the variance of each observed indicator item that represents the "true" constuct it suposedly measures, such as in a standard CFA model or effects indicator model (Bollen & Lennox, 1991). The remaining variance is attributed to measurement or other error. When data are collected with multiple methods, however, a set of indicator variables collected with one method (e.g., self report) may correlate more highly among themselves than with indicators collected with a different method (e.g., observation). This variation associated with method, or perhaps reporter (e.g., child, parent), is assumed to be independent from the substantive construct.

        An MTMM model would include, in addition to latent factors for substantive constructs, one or more method factors (e.g., an observation factor) to explain the method variance among the indicators. Although the MTMM model assumes multiple methods, an equivalent model could be fit for any ancillary constructs that likely explain variance in responses but remain independent of the substantive constructs. These additional factors, whether they account for methods or some other putative cause of variability, allow for additional correlation among the errors in a theory-driven manner.

        MTMM models are relatively common, and Saris and Aalberts (2003) discuss different explanations of correlated errors in MTMM studies. They discuss seven alternative explanations for the correlated disturbance terms. The simplest, the correlated uniqueness (CU) model, allow correlations among all indicators from a single method. This model, proposed by Kenny (1976) and Marsh (1989), is very general and does not explain the correlations but simply allows them.

        Saris and Aalberts (2003) tested more restrictive models with seven different data sets to determine the most likely explanation for the correlations. They concluded that the best explanation of the CU model was the "methods effects" (ME) model. The ME model creates factors for each method, which is probably the most typical formulation of a MTMM model. Thurber, Shinn, and Smolkowski (2002) fit this type of model to a set of mathematics and reading tests. We created substantive constructs for reading, math computation, and math applications. We also fit a timed-test factor to account for correlated errors among tests administered with a time limit. The timed-test factor represents a testing method, which differed from other test methods, and the models fit well with this factor included.

        Saris and Aalberts (2003) did not rule out other explanations of the correlated disturbances in the CU model, although they note that the "methods effect" explanation appear to be the best. Other reasonable explanations of correlated errors may also exist. Many CFA models include data collected with only one method in the strict sense of the term (e.g., survey). Method effects can be expanded to include not just procedures (e.g., self report, interview, observation) but also reporter and setting. At its heart, MTMM just identifies additional variance from some other source, and there are many ways unrelated to the substantive constructs in which items can be dependent, such as items with similar response categories. For example, social behavior items on a Likert scale and those that request discrete counts may each covary within item type but independently from the social behavior construct.

        A repeated stem for a set of items may yield common variance across the items. Imagine four questions about a parent's response to child behavior that begin with the stem, "if your child refuses to set the table, do you. . . ?" Contrast those with four more items that begin with "when your child misbehaves, do you. . . ?" The first question stem concerns a specific scenario that may rarely or never occur for some children but happen frequently for others. This pattern can lead to covariance among items that are more associated with whether the parent's child ever declines to set the table. The second, more general stem likely occurs for nearly all children. The difference in the relative frequency of the child behavior—specifically refusing to set the table versus general misbehavior—might influence the frequency in which parents report their own response to those situations. That, in turn, can lead to correlations among error terms for subsets of items.

        It is important to recognize that CFA, as with all correlational techniques, simply capture associations within the sample represented by the data at hand. The sample can also affect the covariances. Measures collected from a highly restricted sample—only families of children with significant behavior challenges—will produce different relationships among variables, and therefore a different factors or numbers of factors, than a random sample of all families.

        There are, therefore, several potential ways to handle correlated errors. Ideally, the potential correlations among errors are chosen from theory, but as noted by this discussion, the central, substantive theory is insufficient. Measurement models must take a broader view. Unlike Popper’s (1963) falsification criterion—where “every genuine test of a theory is an attempt to falsify it, or to refute it” (p. 37) and any falsification (e.g., negative results) refutes the theory—Duhem (1914/1991) and Worrall (2003) cast tests of theories as a test of the central theory, instrumentation theories, and other auxiliary theories. Only with a broad view can scientists investigate the theory behind the measurement of key constructs without falling prey to the challenges of overfitting or other spurious, sample-specific associations in the data.

        It is therefore generally acceptable to allow error correlations. If the addition of correlated errors, either through specific, individual pathways or method factors, markedly change substantive parameters in the model, the correlated errors may represent missing piece of the theory. The addition of correlations among error terms, whether through correlations among specific errors or through methods factors, are most defensible when based on theory, a priori expectations, and previous research. The literature holds many articles about correlated errors, and they do not always agree. Hopefully, this brief overview lays some grownwork for interpeting the many views on these issues.

See Also

Correlated Errors & Model Modification in SEM (annotated bibliography)

Confirmatory Factor Analysis & Measurement Models (annotated bibliography)

Structural Equation Models (SEM) (annotated bibliography)

SEMNET archives

References

Bollen, K. A. (1989). Structural equations with latent variables. New York: John Wiley & Sons.

Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110(2), 305-314.

Corten, I. W., Saris, W. E., Coenders, G., van der Veld, W., Aalberts, C. E., & Kornelis, C. (2002) Fit of different models for multitrait-multimethod experiments. Structural Equation Modeling, 10(2), 193-213.

Duhem, P. (1991). The aim and structure of physical theory (P. P. Wiener, Trans.). Princeton, NJ: Princeton University Press. (Original work published 1914).

Gerbing, D. W., & Anderson, J. C. (1984). On the meaning of within-factor correlated measurement errors. Journal of Consumer Research, 11, 572-580.

Kenny, D. A. (1976). An empirical application of confirmatory factor analysis to multitrait-multimethod matrix. Journal of Experimental Social Psychology, 12, 247-252.

Marsh, H. W. (1989). Confirmatory factor analysis of multitrait-multimethod data: Many problems and few solutions. Applied Psychological Measurement, 13, 335-361.

Rubio, D. M. & Gillepsie, D. F. (1995). Problems with error in structural equation models. Structural Equation Modeling, 2(4), 367-378.

Saris, W. E., & Aalberts, C. (2003). Different explanations for correlated disturbance terms in MTMM studies. Structural Equation Modeling, 10(2), 193-213.

Thurber, R. S., Shinn, M. R., & Smolkowski, K. (2002). What is measured in mathematics tests? Construct validity of curriculum-based mathematics measures. School Psychology Review, 31(4). 498-513.

Worrall, J. (2003). Normal science and dogmatism, paradigms and progress: Kuhn 'versus' Popper and Lakatos. In T. Nickles (Ed.), Thomas Kuhn (pp. 65-100). New York, NY: Cambridge.