Hi everyone. I have conducted analysis on items of a competency test and found that there are 2 items have negative biserial correlation (-0.01 and -0.03). I and several panel of experts did not find any problem with the keys and distractors. Because, both items are difficult items, so they were very good to discriminate against top performers. Do I have to remove the items for the final test composition? Thank You
if your answer key is correct then check if the multiple categories are mutually exclusive events and collectively exhaustive
How the experts think the items will perform matters less than how they actually did perform. I would recommend removing them.
How big is your sample and how many people got these two right? Sometimes difficult multiple choice questions have a wrong option that attracts many good students. Someone with low ability can have a higher probability of correctly answer the question (guessing) than someone with good ability (who gives the attractive error). This is why test makers avoid "trick questions" and it means that the assumptions of the IRT analysis (a smooth S shaped curve between ability and probability of answering each item correctly) is not meet. I'll attach a paper with this an example like this for an SPSS question.
In short, readers need more information in order to address your question.
Article A dialogue about MCQs, reliability, and item response modelling
- I think you must eliminate them, also you might take a look for the acceptable range of items difficulty, for example items which have a zero difficulty index according to CTT must be eliminated.
Thanks for the response. My data were item score from 280 thousand of participants who complete the 80 items of single test. The item parameters were calibrated using Minres factor analysis as conducted by TESTFACT. A result from different approach (i.e., Rasch Analysis) consistent with this finding.
In the one of the items that remain problem, participants were asked to reveal the meaning of certain concept of statehood. However, top performers tend to choose one of the distractors that are related to the definition rather than the meaning which is produced wrong answer.
Perhaps the relationship between the key and the distractor is not quite mutually exclusive as described by Dr. Abdulrazzak Charbaji. And also to Dr. Rana Momani, the difficulty index as reported by CTT for that items is 0:05. I also hope this explanation can answer the questions of Dr. David L Morgan and Dr. Daniel Wright.
With 280,000 test takers, and those low correlations, I would be confident that those two items are not measuring the same construct as the remaining items. Given the sample size, did you do some pre-testing with techniques like think-alouds?
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs