Suppose that the phenomena we study comprise about 3 % of the population and we made a two item screening instrument (with dichotomous items). If we think in terms of Rasch /IRT, what is the optimal item difficulty for these two items? and if it's contingent of their covariance - how to test it ?
Dear Peter,
do the 3% represent the lower or upper end of a distribution or are they evenly distributed within the population? In other words: Are these extreme values? The optimal item difficulty would be at that point, where the item difficulty equals the persons ability. With IRT scaling, these are values around 0 usualy (depending on the scaling; the range is not fixed).
If the values are extreme values, you would either have the majority of items near the guessing probability, e. g. in case of a high aptitude test (in this case maybe probabilities between 50% and 60% according to classical test theory) and the other way round with low aptitude tests. It is however advisable to have a mixture of difficulties in order to better deal with heterogenous results. The test is more robust for varying person abilities then,
Best regards,
Wolfgang
To decide what is optimal it is worth knowing the costs and benefits of the different classification decisions. And it sounds like you have control over the item difficulty. Is this true or are you just picking among items? If picking among items and if you have a data set with all these items and the phenomenon of interest, could you just test the different pairs? I guess I am needing more context to understand why you are doing this.
Hi Peter,
Are the two mentioned items sufficient to measure the latent variable you are examining? Could you give more information on the topic and scope? Is the mentioned 3% group at the lower skill level or upper skill level?
The 3% represent the upper extremes yes. And the two items are fixed, and my test is just to see if they match optimal item difficulty. As it is now item1 get 2.8% positive answers and item2 get 0.7%. With a cut-off on 1 out of 2, the test actually screens out about 3% but it seems to me that the difference in item difficulty seem to large ?!
I don't know if this is feasible, but it is risky to have only two items. The reliability will be low. In general, it is a good thing to have items with varying difficulty. I do not think, the difference in the difficulty in your case is a problem. They are indeed very close to each other. It is rather, that you should include more items with a range of difficulties.
Certainly you are right about the problems with only two items. However, this is a somewhat established instrument and tested as such. (it's a test of problematic gambling). To me it seems inefficient to have an item with so low endorsement as item2 and more reasonable to have item1 slightly more difficult and item2 slightly less difficult but I can't justify this in any good way
I think you are right on that. A line of argumentation could be, that it is hardly possible to distinguisch the results of item 2 from zero. The 0.7% simply could be a random effect and you would need a large sample to prove, that it is different from 0 at all. Maybe it would help to do a power analysis with G*Power to show, that you would really need a lot of people to prove the validity of that item at all. And again: Even item 1 is really very close to 0 as well. And in any case: A pilotation of the scale could give you the best answer.
Thanks for your ideas on this. I really appreciate it! Since I work with a big sample on this close to 20,000 subjects, it differed significantly from zero. The approach to make a power estimation on how big a sample you need to differ it from zero was a good idea and I'll try that!
From an IRT perspective, the argument is to consider the purpose of the test and develop a test information function to match that purpose. For example, if the test is to classify examinees at a cutscore of -1.0, then put the items there. Admittedly, this approach fits better to longer tests and cognitive/knowledge situations.
Does that mean that you suggest the two items should be equally difficult or put differently is there a point in having different difficulty given that I'm only interested in a binary outcome of the test ?
From a pure IRT perspective, the test information function would be maximized if the two items had a Rasch difficulty at the value where you have your cutoff score for examinees.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics