As my question may sound a bit bulky, please consider the following example:

You want to estimate the latent abilities of p participants who have answered i items in a questionnaire. The individual answers to the different items can be ascribed with the following model:

Answer_i,p = u_i + u_p;   u_i ~ Normal(0, sd) and u_p ~ Normal(0, sd)

where u_i and u_p are randomly varying parameter manifestations of item difficulty and participant ability, respectively.

Suppose you expect (from previous studies) that participant ability is driven by some covariate C that you can assess and/or experimentally manipulate. A regression of u_p on C does not suggest any significant association between participant ability and this covariate, although the effect (beta) points numerically to the expected direction.

Therefore you try to inform your psychometric model by including a (fully standardized) covariate C as a second level predictor of participant ability:

Answer_i,p = u_i + u_p;  u_i ~ Normal(0, sd), u_p ~ Normal(beta*C, sd)

Surprisingly, the beta of C now differs significantly from zero, as the estimates of participant ability have changed. Thus the estimates of participant ability become informed by a covariate, which did not have any predictive value before.

I repeatedly encountered this phenomenon. Nonetheless I must admit, that it remains quite elusive to me. Moreover this seems to introduce some kind of arbitrariness in the interpretation of predictors for latent abilities. Is the covariate indeed informative with regard to participant ability or is it not?

Does anyone have a solution to this issue? Proably such situations should be dealt with by appropriate model comparision strategies (i.e., one should refrain from interpreting estimates of such nested effects in isolation)?

Similar questions and discussions