Obviously, IRT modelling has its challenges. However, I guess, the claim also means that a score or an individual item of a test of a mathematical test, as an example, would not measure mathematical ability - IRT just forms a weighted score of the test items. It is tgenerally known that the achievement score never reflects JUST ability. It is generally accepted and known that the score of a mathematics test includes also elements of reading ability, general intelligence, motivational aspects ETC. However, usually we think that the score reflects mathematical ability accurately enough in a sense that we may say that the main variability in the score comes from the mathematical ability if the test is proper for the level of the test taker (e.g., personally I would not score very high with verbal mathematical test items given in Nepali language, for example). Then, to me the claim feels not a new finding but the general knowledge within the testing business?

According to Measure Theory, a measure must be additive. For example, in case of two subscales, say M (math) and R (read), a measure (General Total Score) has the decomposition expression:

Overall Score = Sub-score(M) + Sub-score(R) - Score(M*R)

where (1) Overall Score is the score associated with all items in a test;

(2) Sub-score(M) is the score associated with M;

(3) Sub-score(R) is the score associated with R;

(4) Score(M*R) is the score associated with shared ability of M and R.

In case M and R are independent (in a given test), then Score(M*R)=0, then above equation becomes

Overall Score = Sub-score(M) + Sub-score(R)

In high-stake scoring, the above equation is heavily used. As we know, most likely, "M and R are independent" may not be true in practice, therefore, we need to calculate Score(M*R) to achieve the accurate Overall Score. For more detail on this topic, please see

Preprint Sub-scores and Overall Score