Try:

Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38-47.

I would stringky recommend Embretson's "New Rules of Measurement" which give a direct head-head comparison of CTT & IRT.

Hope this helps

Matt

Hi,

Here is two empirical example of comparison between these two methods: Article Using Classical Test Theory, Item Response Theory, and Rasch...

Article Comparison of Classical Test Theory and Item Response Theory...

You can directly download the articles at these addresses.

For recent theoretical presentation of both methods: Chapter Classical Test Theory and Item Response Theory

Best

CH

This reference looks comprehensive:

Article Using Classical Test Theory, Item Response Theory, and Rasch...

thanks CH

Dear all, thanks a lot for all your thoughtful suggestions. I checked the articles and book chapters all, but unfortunately, they are still based on the outdated CTT concept with axioms (that actually do not exist) and assumptions that do not follow from the basic equation of CTT. Nevertheless, all provide some interesting insights into their specific research questions. Embretson’s “New Rules of Measurement“, for example, is a nice book that I appreciate very much, but the CTT chapters are still based on the outdated CTT concept with axioms (that actually do not exist) and assumptions that do not follow from the basic equation of CTT. DeMars in her book chapter “Classical test theory and item response theory” still uses axioms based on the basic CTT equation to derive the most common formulas used in CTT.

In the article by Pretillo et al. (2015), again old CTT concepts are used. Instead of testing the dimensionality of the test or questionnaire first, item-total correlations as well as Cronbach’s alpha were calculated. Following Raykov (2011, p. 115 ff.), there are some misconceptions about CTT. First, there is no CTT model as is often claimed. A decomposition of the observed score as the sum of true score and error score can always be made as long as the true score is defined as an expected value, but this is not a model. Second, a model has to be defined as a set of assumptions that must be falsifiable. The basic equation of CTT is not a model because the basic equation cannot be falsified. Although CTT is not based on a model, it is possible to develop models based on CTT. Such a model would be, for example, the model of essentially tau-equivalent variables, which is a special case of a confirmatory factor model. This model is needed in order to estimate Cronbach’s alpha, it includes the assumptions of unidimensionality and equal item discriminations (factor loadings). Unfortunately, researchers still do not define models and they do not test the assumptions necessary for evaluating the structure of a test as well as the test’s reliability.

I just found an article in which CTT and IRT were compared:

Raykov, T. & Marcoulides, G.A. (2016). On the relationship between classical test theory and item response theory: From one to the other and back. Educational and Psychological Measurement, 76, 325-338.

And here is the reference of the book I found very helpful and informative:

Raykov, T. & Marcoulides, G.A. (2011). Introduction to Psychometric Theory. New York, NY: Routledge.

Thanks again for your help.

Kind regards, Karin

Karin,

Here are a number of reactions to your summary of comments:

First, there is not one CTT model. There are a number of such models with different assumptions about errors. X = T + E is the most basic model. Most often it is incorrect, but it’s a model nonetheless from which derivations can be made with added assumptions. I am baffled by your claim that definition of a true score as an expected value somehow disqualifies the equation from being a model. This condition would disqualify all linear forms E[Y] = a + bX = a + bE[X]. The vexing assumption of CTT is Mr. Brown taking an item over and over ad infinitum with an erased memory to arrive at T = E[X]. Denny Borsboom has some interesting things to say about the conceptual limits of either true score or latent variables relative to networks of measured variables. So perhaps modern measurement theory is also on its last leg.

Second, your claim that a model must be falsifiable appears to equate scientific or empirical propositions with “models,” yet models are used as tools or heuristics for examining propositions. There are good models, and bad models such as models with unidentified parameters or specification errors. It was Melvin Novick who quipped that all models are incorrect (at least in the social sciences), but some are more useful than others.

Third, models like tau-equivalent items are elaborations of the basic equation. These are commonly discussed under the rubric of CTT (as they are in the classic volume of Lord and Novick). Note that the “classical” component of CTT means older or original. By definition classical theories are outdated, but not in the sense those theories should be updated. Rather, they are replaced with newer theories that retain characteristics of the older theories as shown by Raykov and Marcoulides (2016).

Fourth, please note that the congeneric model is also considered a CTT model, unless you have a date in mind that divides different types of nonIRT models into categories along a timeline. BTW, the tau equivalent model shows that Cronbach’s alpha based on simple CTT assumptions most likely underestimates reliability. You conclude with the statement “Unfortunately, researchers still do not define models and they do not test the assumptions necessary for evaluating the structure of a test as well as the test’s reliability.” I think this is a falsifiable claim. The psychometric literature is replete with careful analyses of test structure within the framework of exploratory and confirmatory factor analysis as well as experimental studies.

Greg

Greg, thanks for your detailed reply. I know that it is always described in text books, but the equation X = T + E is not a model. Therefore, I do not agree with you concerning this point. Raykov (2011, p. 121) explains this quite clearly:

“In a considerable part of the literature dealing with test theory (in particular, on other approaches to test theory), the version of Equation (5.1) [X = T + E] for a given test is occasionally incorrectly referred to as the ‘CTT model’. There is, however, no CTT model (cf. Steyer & Eid, 2001; Zimmerman, 1975). In fact, for a given test, the CTT decomposition (5.1) of observed score as the sum of true score and error score can always be made (of course as long as the underlying mathematical expectation of observed score—to yield the true score—exists, as mentioned above). Hence, Equation (5.1) is always true. Logically and scientifically, any model is a set of assumptions that is made about certain objects (scores here). These assumptions must, however, be falsifiable in order to speak of a model. In circumstances where no falsifiable assumptions are made, there is also no model present. Therefore, one can speak of a model only when a set of assumptions is made that can in principle be wrong (but need not be so in an empirical setting). Because Equation (5.1) is always true, however, it cannot be disconfirmed or falsified. For this reason, Equation (5.1) is not an assumption but rather a tautology. Therefore, Equation (5.1)—which is frequently incorrectly referred to in the literature as ‘CTT model’—cannot in fact represent a model. Hence, contrary to statements made in many other sources, CTT is not based on a model, and in actual fact, as mentioned earlier, there is no CTT model.”

But there are models developed within the framework of CTT. If one posits assumptions about true scores and errors for a given set of observed measures (items), which assumptions can be falsified, then one obtains models. This is closely related to confirmatory factor analysis, because the CTT-based models can be tested using CFA. If one assumes unidimensionality and uncorrelated errors, this would be a model of tau-congeneric variables, because these assumptions can be tested.

Raykov, T. & Marcoulides, G.A. (2011). Introduction to Psychometric Theory. New York, NY: Routledge.

Steyer, R. & Eid, M. (2001). Messen und Testen (Measurement and Testing). Heidelberg: Springer.

Zimmerman, D. W. (1975). Probability spaces, Hilbert spaces, and the axioms of test theory. Psychometrika, 40, 395-412.

Kind regards,

Karin

I just realized that chapter 5 of Borsboom's book "Measuring the Mind" might be what you are looking for.

Thank you, Gregory, I will have a look at this book.

Kind regards,

Karin

I recomment to read an important book about IRT´s essentials (The basics of Item Response Theory). According to Frank B. Baker, Item Response Theory can evaluate a same scala the items and examenees and it allow that the measurement made with IRT are invariant respect to the item (on a test) used on the medition.

please check the following essay ( Numerical Comparisons across General Total Score, Total Score and Item Response Theory ):

Research Numerical Comparisons across General Total Score, Total Scor...

Please see this paper (IRT SCORING AND THE PRINCIPLE OF CONSISTENT ORDER):

https://arxiv.org/pdf/1805.00874.pdf

"When in 1940, a committee established by the British Association for the Advancement of Science to consider and report upon the possibility of quantitative estimates of sensory events published its final report (Ferguson eta/., 1940) in which its non-psychologist members agreed that psychophysical methods did not constitute scientific measurement, many quantitative psychologists realized that the problem could not be ignored any longer. Once again, the fundamental criticism was that the additivity of psychological attributes had not been displayed and, so, there was no evidence to support the hypothesis that psychophysical methods measured anything. While the argument sustaining this critique was largely framed within N. R. Campbell's (1920, 1928) theory of measurement, it stemmed from essentially the same source as the quantity objection." by Joel Michell

The above comment is also true for IRT because IRT has no its additive structure in its theoretical design (or more generally, IRT has no its decomposition theory). The theta parameter in IRT (also called IRT ability parameter) is nothing but a variable that is positively correlative to ability measure, but theta parameter in IRT is never the ability measure. Therefore, IRT causes concern about "... fairness and transparency may be compromised." discussed in https://arxiv.org/pdf/1805.00874.pdf

Everyone knows that the area of rectangle is measured by the product of its width and length, but we must not measure the area of rectangle by the summation of its width and length. This is because the product of the width and length has its additive structure in terms of the area while the summation of the width and length has not. Although the summation of the width and length is positively correlated to the area of the rectangle, the summation of the width and length is still an incorrect way to measure the area of the rectangle because it can not keep the fundamental structure of the area which is additivity. Mathematically, the IRT Theta parameter (IRT ability parameter) is equivalent to measuring the area of rectangle with summation of length and width.

Measure is an (isomorphic) mapping from the object which is measured to the real number domain (usually greater than or equal to zero). This object could be area, volume, angle, randomness, uncertainty, (linear) correlation, information or ability etc. Here, isomorphic mapping is the mapping which keeps the additive structure from the object which is measured to the real number domain.

Hi Karin Schermelleh-Engel

Many of the books about Rasch modeling include comparisons of CTT with IRT, e.g., Bond: Applying the Rasch model

Best

JK

Dear all, thanks a lot for your suggestions.

Kind regards, Karin

Test Theory: A Unified Treatment

by Roderick P. McDonald...

Halil, thanks, I know McDonald's book and like it very much.

Karin

try this:

Comparison of Classical Test Theory and Item Response Theory in Individual Change Assessment

Applied Psychological Measurement

2016, Vol. 40(8) 559–572

DOI: 10.1177/0146621616664046

Thanks, Halil and Victor, both references are very good and helpful!

Kind regards, Karin

Article Validation of a developed university placement test using cl...

Research Comparison of Classical Test Theory and Item Response Theory...

CHECK THESE TWO ARTICLE, YOU MAY GET SOME INSIGHT THERE. ALL THE BEST

Thanks Ado Abdu Bichi for sharing these articles, I will have a look.

Kind regards, Karin

Susan Embretson has a paper you might find useful. Her article in Psychological Assessment, 1996, Vol 8, No. 4, pages 341-349 is entitled: "The New Rules of Measurement" Best of luck, Linda

Thanks, Linda, this article is very interesting.

Kind regards, Karin

Hi Karin, you may want to have a look to this article we wrote some time ago discussing some claims of the classic books of IRT promotores...

Asún, R. A., Rdz-Navarro, K., & Alvarado, J. M. (2017). The sirens’ call in psychometrics: The invariance of IRT models. Theory & Psychology, 27(3), 389-406.

And maybe this one (although the ideas in this article may not be new for you):

Rdz-Navarro, K. (2019). Latent variables should remain as such: Evidence from a Monte Carlo study. The Journal of general psychology, 1-26.

The reference list on both articles may also help you find more comparisons between IRT and CTT ;)

I just want to say that classical theory as well as IRT theory concern only unidimentionnal tests of aptitudes and that they are often used for attitudes tests who are multidimentionnal and non metric. It is why i have developped a new theory that i call the functionnal method for use with attitude tests....You can find a description in my technical papers in frontiers in psychology....

Sincerely

Francis

Why Total Score in CTT has the relation between Overall Score and Sub-scores while IRT-Ability Score Has Not?

Please see " Item Response Theory and Its General Total Score ":

Preprint Item Response Theory and Its General Total Score