In practice, IRT (2PL/3PL) and Total Score may not be consistent. In case that, for two examinees, say S1 and S2, there are
IRT ability (S1) > IRT ability (S2)
Total Score (S1) < Total Score (S2)
Which one we should use for examinees' reported score?
Here, Total Score is the number of those correctly-responded items (testing questions).
Inconsistency between total scores and trait/ability parameter estimates is the simple result of letting the discrimination parameter vary among items. If you want to stick to the IRT framework, you should of course use the ability estimates instead of total scores.
The real question here is, shouldn't we bother with this feature of IRT? This inconsistency should be very disturbing because it would also mean that, the relative difficulty of two items might change depending on who are answering those two items. You would, of course, expect a participant with high ability level to answer an item correctly with a higher probability, compared to a low ability participant. But the relative difficulty level of two items should be independent of who is taking the test. 2PL and 3PL IRT models basically claim that "although they are measuring the 'same latent construct', item X can be easier than item Y for person A while item Y can be easier than item X for person B and there is nothing wrong with this varying relative item difficulties".
Those who see this possibility as a serious violation of measurement principles, would suggest you employ Rasch measurement models (a.k.a. 1PL IRT) instead.
Hi, Serkan,
Thank you for your comment.
(1) For a given examinees' population, the estimates of item's difficulty parameters are the same for any examinee from this given population. Therefore, it is impossible for "...item X can be easier than item Y for person A while item Y can be easier than item X for person B ..." to happen,
(2) Total Score is summation of dichotomous item scores (0 or 1). This summation is called additive structure and but, theta in IRT has no this additive structure. One direct consequence from this "IRT has no additive structure" is that, in MIRT, only those "ability" estimates associated with sub-scales are made while the "total ability" estimate associated with all items is never available.
(3) Measure Theory requires that all measure be additive. Therefore, theta in IRT is not a measure for ability measure. I believe the theta in IRT is nothing but a variable that is positively correlated with the measure of ability.
(4) Please see the project along with this question for more detail on this topic.
Let me clarify what I mean by "item X can be easier than item Y for person A while item Y can be easier than item X for person B".
When you invoke a 2PL model, you let your discrimination parameters freely vary among items. That would mean that your Item Characteristic Curves would not be parallel (i.e. cross one another). If your data fit your 2PL model, you would, of course end up with a single difficulty (location) parameter for each item. But this difficulty parameter only indicates the location of the item along the ability/difficulty continuum where the persons at that location have 50-50 chance to correctly respond to it.
By "Item X being more difficult than Item Y for a specific person", I mean that the probability of correctly answering Item X is lower than that of Item Y for this specific person.
But when ICCs cross one another, you would end up in situation like this:
i) the probability of Item X being correctly answered by a low ability person (Person A) is lower than the probability of Item Y being correctly answered by the same person
ii) at the same time, the probability of the same Item X being correctly answered by a high ability person (Person B) is higher than the probability of the Item Y being correctly answered by the same Person B.
That is, Item X, relative to Item Y, is more difficult for Person A but Item Y, relative to Item X, is more difficult for Person B. Therefore, even though one item might have a higher "difficulty parameter" than an other item, that does not mean that the probability of correctly answering the first item is "universally" lower than that of the second item.
Hi, Serkan,
Thank you for your clarification.
You are right on "in case that two ICCs have "cross point" and person A and person B are on the different sides of this "cross point", item X can be easier than item Y for person A while item Y can be easier than item X for person B".
Let's divide all persons into low-ability group and high-ability group according to "cross point" and then, calculate the correctly-responded percentages of item X and item Y for both low-ability group and high-ability group. Most likely, these correctly-responded percentages are consistent across low-ability group and high-ability group unlike the performance of those items with "cross point" in IRT.
What sort of quality control is there on total score? In Rasch measurement, N correct is the sufficient statistic for estimate the person ability parameter, but only to the extent that the data fit the model.
Total Score is only true under the following three assumptions:
(1) All items (testing questions) are dichotomous items which have two possible values "Right" or "Wrong" (in practice, "1" for "right"; "0" for "wrong"). That is to say that any item must not have partial credit(s), say, an item having possible values: 0,1,2,3,... is not allowed in Total Score.
(2) All items are of the same difficulty levels. Therefore, the examinee who correctly respond 10 difficulty items receives the same Total Score as the student who correctly respond 10 easy items.
(3) All items are jointly independent. Therefore, Total Score of two (correctly-responded) highly-duplicated items or even identical items is the same as Total Score of two (correctly-responded) less-duplicated or even independent items (A fair rule is that the Score for two correctly-responded identical items should be the same as the score for correctly-responding one of these two identical items). Also, "All items are jointly independent" implies that the scoring for those shared ability across different students is always 0, but, in real world as we know, this is not true.
Which those impossible to fulfill assumptions, the almost any IRT modeled estimate must be better.
(1) In practice, IRT is not allowed to include highly dependent or related items, say several identical items, which implies that IRT is not designed to fully handle all those interactions of all combinations of the items. Therefore, IRT has no its decomposition theory. Without decomposition theory, how IRT is able to correctly measure the concept of ability?
(2) Although IRT introduces the difficulty parameter associated with each item, these difficulty parameters are not directly related to the IRT ability estimate (theta estimate). Therefore, 1-parameter IRT (Rasch model) and those 2 or 3 parameters IRT is not consistent -- this IRT self-conflict implies that IRT is not a correct theory for high-stake scoring.
(3) In practice, Total Score is almost the only way for high-stake scoring -- people has already made their choices between IRT and Total Score by their actions.
But the measurement-prescriptive Rasch model is distinctly different from the (other) data-descriptive IRT models.Those who choose total score already ascribe to the Rasch requirements, it seems to me. Fan (1998) might conflate Rasch, IRT and TST by ignoring the issue of data fit to the model, but unidimensionality and interval measurement remain the striking Rasch advantages.
Trevor, Thank you for your comment.
In IRT, Rasch model is the closest to Total Score in terms that both Rasch model and Total Score produce the same ordinal scores although their scalings are not the same. In terms of information, the theta in Rasch model has nothing more than Total Score. Therefore, the "interval" information in Rasch, which is not available in Total Score, makes any sense?
Above comment is mainly based on "Total Score is the sufficient statistics of theta in Rasch model".
Seems to me that it is the logistical transformation of the data which develops the interval level measurement for Rasch. Invariant intervals are the sine qua non of measurement in the human sciences and those Invariant intervals amongst item measures and amongst person measures have been well demonstrated.
n.b., I have yet to read your original JAM paper.
Mr Gove’s major problem: Why Pisa ranks are wrong
This essay presents some comments and concerns by those psychologists who "... refutes the claim that item response theory is a settled, well-reasoned approach to educational measurement". Although, as this essay claimed, the statement "a century later, James’s less optimistic vision has materialised and that it may be time to bury scientific psychology, at least as a single coherent undertaking" can not be proved to be true, this essay strongly indicates that Item Response Theory is still far away from a well-done theory and thus, in practice, it is highly recommended to be alert in applying IRT to any high-stake scoring business.
https://paceni.wordpress.com/tag/holders-seven-axioms/
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics