There are several methods to determine the difficulty index or easiness index of a set of test questions. Nevertheless, there are conflicting views as to when a test question should be thrown out. Moreover, throwing out too many test questions can be a moral dilemma for the course instructor because throwing out test questions can artificially inflate grades. Therefore, what percent of students incorrectly answering a test question justifies throwing it out?
Throwing out a TEST happens only when the taught content doesn't match with the Evaluation expectations...This normally do not happen as long as the teacher themselves set the evaluation.....However in case of Common tests administered, over a divers population this kind of a problem can occur.... the deviation can be when the content is perceived as a THEORY alone or on a PRACTICAL application... The T-P sync depends on the reasoning after evaluation... In my opinion however difficult the test may be, if more than 60% failure happens then that is not an evaluation....
During '80s, In Madras University for 4th semester Mechanical / electrical engineering has a course work called SYSTEMS ENGINEERING, only 10 percent of the class will clear.. there the evaluation will not be on text book or reference book orientation but on assimilation and extension of learning.
Only throw a question out if there has been something that was incorrectly taught and is now being examined. For example, today's
news overthrows what was in the textbook, or if a mistake crept into the
test, such as a question that was outside the syllabus or ambiguous.
If there is a common misunderstanding, e.g., because of poor teaching you should not throw the question out, otherwise you are hiding the inadequate teaching. Also, if a common mistake was pointed out to the students, and was never unlearned by the students, then they should be penalized.
Throwing out test questions will never inflate the grades, because you are marking out of less than 100%. It just reduces your confidence
in your mark.
Dear Reginald, researchers usually use Chronbach's alpha to analyse reliability of a test. If alpha ≤8 your test is reliable and you do not need to throw any question out.
I've run into this problem several times. One thing you can do is examine the test's reliability. If α ≥.80 then the test is reliable, or go as low as .70. You can also look at item-total correlations, or item-average correlations. Negative correlations indicate a definite problem.
However, when deciding to drop a question, I think it's more then just relying on statistics. I carefully examine the question and the answer options (if multiple choice). Questions that receive around 15% correct responses could indicate a problematic item. Sometimes it's an error on my part. Perhaps the question/answer may have been too hard or ambiguous, or worded in a way to suggest a different answer than the answer that I was expecting. In some cases the question/answer might not accurately reflect the lecture/textbook material, it helps to go back to see what I taught and to determine if the items correctly corresponds to the material (hence, importance of keeping lecture notes). Or it could be as simple as an error in the answer key.
If nothing wrong can be identified, it's a judgement call. I avoid dropping too many questions, to avoiding compromising the validity of the test. Maybe 1-2 item per test depending on length. For one exam I dropped the value of the question (e.g., made test out of 99 rather than 100) and kept the question as a bonus mark for those who answered correctly.
All the best,
Donaldo
I agree that percentages don't tell the whole story, and statistical analysis is helpful --one of the advantages of running objective tests on an LMS such as Blackboard is that item analysis is automated. However, there may be cultural or linguistic issues with the wording of the test item itself. In that case, review by an assessment specialist or a linguist might identify the issue. In one institution with a diverse faculty we routinely turned up issues in item review that were not detected by the item writer. When I was a student in the American South we encountered cultural issues with questions on 'standardised tests' written in Princeton, New Jersey. One that sticks in my mind was an analogy between peas and green--in the South, peas are not green.
I have had 2 related experiences. Once, I had a senior teacher agree with some of my students that my quiz was not fair, because he had not covered in his lecture the material on 3 of 10 of my questions. Discarding those questions made the test invalid to me. I had covered the material in question and the ones who got it right did not get credit.
In another case, a family member took a Contracts exam in law school, the only test in the course. It consisted of four, long essay questions,the third one very complex. According to the professor, only 2 of about 100 students got credit on that third question. One of those two students had spent half of his exam time on that discarded question, and thus scored his lowest score on the next (final) question. Without the inclusion of the third question's grade, that student got a B instead of an A in the year course, which was 6 credits. As a result, he was not first in his class on graduation, but second.
In such cases it would be fairer to always give credit for the right answer and to compute the grade based on the remainder of the exam only for those who do not get credit for a dubious question. So, 1 student might get 90 points out of 90, while another student gets 100 out of 100, if the first person got everything right except the discarded 10-point question, and the second person got everything right, including it.
- Badges
- Science topic
- Similar topics
- Philosophy
- Ethics