Dear,
I'm interested in a methodology to estimate how similar or different test items are regarding their difficulty. I would like to design test items that despite covering each a different topic in a Biology course, are of equivalent (similar) difficulty. How can I measure that?
Many thanks in advance,
Aminael Sánchez
your question is (or seems to be) related to the "Item Response Theory" ans "Rasch model" (both terms efficient Google entries).
The problem was addressed in a way to produce a ranking of "students" independent (or hopefully so) of the difficulty of "items/questions", not to produce equivalent/similar difficulty items.
Dear Jean, thank you for you useful answer.
I have been checking "Item Response Theory" and "Rasch model" but in both cases item difficulty is assessed by considering students' performance (response). In that sense, I think that these models handle difficulty a posteriori. I'm interested in a way to set (modify) items difficulty a priori (during item design, writing). In that sense I have been following the methodology proposed by Cheng 2006 ("On varying the difficulty of test items"). Cheng discusses the existence of moderators of item difficulty, i.e. CONTENT, STIMULUS, TASK and give guidelines to increase or decrease items difficulty by manipulating theses moderators. Do you have experience in this sense? Or do you know of simular approaches.
I'm very grateful for your help,
Best regards,
Aminael
When constructing items state companies will often use subject matter experts to decide levels for different cut scores. In doing this they usually categorize the items on difficulty (along with saying whichever components of the construct they are measuring). However, as Jean says, IRT gives what you want, so they would also use the values from an IRT on pilot data prior to any test release.
Thanks a lot Daniel. It seems like a hybrid strategy between a priori and posteriori difficulty assessment.
PRE-TEST of a questionnaire is indispensable BEFORE the questionnaire is released. In a group of several dozens to hundreds respondents (please, care of sampling strategy and check the sample characteristics, how much they fit to your population), I check difficulty and usually exclude too difficult (less than 10% of correct answers) and too easy items (more than 90% - however, this borderline is arbitrary set), when I follow classical test theory. I do not have much experience with IRT. Of course, exploratory and/or confirmatory factor analysis and other psychometric tests are also important.
- Badges
- Science topic