Dear Jari,

no offense, but I have problems to understand your post. First, in the literature/community of factor analysis and structural equation modeling, it is level 1 knowledge that the error component of an indicator consists of both systematic and random error and that--when to measures share a systematic error, this will either lead to error covariances between these two items, or--when all measures share the error, will bias the variance estimation of the latent variable (i.e., the variance will be the sum of true variance and the systematic error. See as one example,

Scherpenzeel, A. C., & Saris, W. E. (1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25(3), 341-383.

Second, the reliability concept has absolutely no connection/implication regarding truth vs. bias. It means simply repeatability. Hence, the existence of systematic error is no contradiction to reliability--to the contrary: Systematic bias will (correctly) contribute to the bias. The ontological misconceptions of reliability as having something to do with "real" stems from the concept of the "true score" that classical test theory has made to prominent. In fact the "true score" has nothing with truth / unbiasedness in it--as it is simply defined as the expected value of a series of repeatitions. Hence, even if you have a totally nonsensical, invalid response, this response will have a true score. Again, systematic errors will (again) contribute to this true score as it moves the observed distribution by a constant (the error). This again does not change anything on the spread of the distribution (=random error).

Denny Borsboom has written a great deal about the problematic implications of the true score concept.

Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge: University Press.

Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425-440. doi:10.1007/s11336-006-1447-6

Borsboom, D., & Mellenbergh, G. J. (2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30, 505-514.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110(2), 203-219.

Best,

--Holger

Hi Holger,

Sorry for a longer note. There is only 5 points though...

Thank you for you points and references. The mass literature is SO huge and SO splintered. It is not possible to know everything alone. Hence, I asked for help. And that your for helping shaping my thoughts.

And no offence taken. However, I feel that we are talking of somewhat different matters. Some points come to my mind from your ideas.

1) Yes. It is well known that the measurement error includes random and systematic part. However, the traditional conceptualization by, e.g., McDonald (1985, 1999) condensed in, e.g. Cheng et al (2012) of the measurement model (xi = Li*Theta +ei and SUM(xi) = SUM(Li*Theta) + SUM(ei)) does NOT include any systematic part. As known, this conceptualization is used strictly in such estimators of reliability as McDonald's omega and maximal reliability, where the only element needed in the estimation is the factor loading Li (leading to error variance PSIi^2 = 1 - Li^2). The model and the practicalities assume that the element Li is errorless. However, factor loading, as being a correlation between an item and the factor score, includes systematic mechanical error that makes the value lower than the true association between the item and theta is. When Li is too low, the error variance PSI is too high. This kind of systematic error cannot be handled by using the traditional conceptualization. Or, did I miss something?

2) I’m pondering whether your statement “systematic errors … moves the observed distribution by a constant (the error)” is just partly correct. Obviously, we think different systematic error. The systematic measurement error I’m talking does not move the observed distribution at all because it is strictly linked to the capability of Li to reflect the association between theta and item. Each item carries its OWN systematic error that depends, among others, on item difficulty (or extreme distribution, if we will) and this error always LOWERS the magnitude of Li if interpreting Li to be a factor score, that is, correlation. Six sources of systematic error is named in the appended document and more can be found. The main reason of the underestimation of association in Li is the phenomenon called Restriction of Range (RR): correlation between an item and the score is always an underestimation because the scales differ from each other. If we would use “corrected Li”, that is, factor loadings (based on some other estimator of association than Pearson correlation) that are purified from this systematic mechanical error caused by RR, the magnitude of each Li would be higher and, consequently, the magnitude of each error component PSIi would be lower, and, consequently, the reliability of the score would higher. Or, did I miss something?

3) Yes, I agree, “the reliability concept has absolutely no connection/implication regarding truth vs. bias.” I guess, this holds when thinking the CONCEPT of reliability. I’m not talking of that but, instead, how to estimate the reliability and how to reach the “real” or “systematic-mechanical-error free” reliability by using the estimators that include elements of systematic underestimation leading to underestimation of reliability. From this point on view, obviously, the estimates of reliability are STRICTLY connected to bias in estimation. It’s difficult to name any estimator of reliability (except, maybe, estimators based on polychoric correlation) which would be free from this systematic bias. Or, did I miss something?

4) Yes, the true score is defined by repeated scores in infinite number of parallel tests. And, yes, stupidly behaving test-takers affect the estimate of reliability of the test score. If we have a test-taker in a mathematics test giving purposefully stupid answers in ten parallel mathematic tests, yes, his/her true score would be labelled 0 because of stupid behavior. And we do not know the “real” true score because this test-taker was not willing to show it. Then, yes, we never find the “real” or “true” reliability of the measurement instrument if we have these cases in the dataset. I guess, we need to live with that. It’s called measurement error. What I mean for the “real” reliability is something else. To me “real” reliability is to do with the “systematic-mechanical-error free” reliability of the instrument with the given dataset; if the estimate of reliability is mechanically underestimated, we did not reach the “real” reliability. Or did I miss something?

5) Finally, I’m pondering your categorical idea that reliability is solely “repeatability”. Yes, this was one of the earliest interpretations of reliability. However, the interpretation depends on the form of estimator, I guess. Reliability may be interpreted as internal consistency if the inter-item correlations are used as the basis of the estimation (in, e.g., alpha, maximal reliability) or discrimination of the test score if item-total correlations are used (also, in alpha, maximal reliability). Li can be interpreted as item-total correlation and this is strictly connected to item discrimination power. Hence, the interpretation of reliability as “accuracy of the score” make sense. This interpretation is obvious when thinking that the standard error of the measurement (S.E.m = S*sqrt(1 - REL): the discrimination power of the score depends strictly on the reliability. If reliability is estimated too low, it means that we do not get “systematic-mechanical-error free” estimate of the accuracy of the score. Or did I miss something?

Again, Holger, thank you for the comments. Hope I did not sound too critical. Our thinking gets clearer when challenged a bit.

Jari

If I have understood the point then I think that your post and article suffers from the very problem you describe.

Terminology is always tricky as we all have our own biases in terminology based on our background fields. Which scientific field you learn the basics in will always flavour your use of terminology throughout your career. This means that someone steeped in multivariate statistics will cause error and confusion when taking to an ecologist about 'factors', but the same will not occur when talking to someone in applied field that uses multivariate methods extensively. Such interactions will cause a consistent and reproducible misunderstanding unless terminology is defined explicitly. However some fields have a history of using factor both in terms of measured variables and in the latent variable sense which can lead to unpredictability in whether understanding or confusion will result.

In such as case we have a systematic source of misinformation (error or noise) that differs depending on the unappreciated (latent) differences in language generating processes between the two scientists. I.e. your systematic mechanical error is not uniform, but depending on what underlying latent variables are contributing to the outcome.

A specific example from the above discussion that appears to have arisen as the terms were not explicitly defined and justified. Both repeatability and reliability have a very specific definitions in analytical science and various bodies put forth formal definitions of those, so you should be more explicit in what you mean when you use those terms. I'm from an analytical background and so would see repeatability (sometimes called precision) as measuring the range of values you would expect the measurement to fall within if repeated immediately. Reliability (sometimes I've seen accuracy or agreement used in the same way), is the measure of how reliable the estimate is (how much it differs from the expected value). Basically they are as different as the mean and standard deviation, i.e. independent metrics. These definitions do not allow me to understand what Holger means by ' reliability ... means simply repeatability '. See attached an example

Article Reliability, repeatability and reproducibility: Analysis of ...

It seems you even have to define what you mean by 'real' and 'true' - an issue I am currently grappling with in a manuscript where I am examining how drift in the underlying data generating processes influence observations. It is a simulation so I have a reasonably high degree of control over the 'truth' in that scenario. In that context I am defining the data generating process as a totality of influences that includes 'signal' (desired information that is causal wrt the outcome of interest) and 'noise' (equally deterministic but unwanted and causally unrelated).

As a side note I think this article may be relevant to your discussion, it examines the impact of additive vs multiplicative effects on the suppression of the Pearson coefficient.

Preprint Corruption of the Pearson correlation coefficient by measure...