I'm working on move analysis and would appreciate if I can get your views on interrater process in analyzing text. The text analyzed is from engineering discipline so a 2nd rater with engineering expertise is needed to ensure the rating by the 1st rater who is a linguist interprets the text correctly.
Is it sufficient for the 2nd rater to interrate only half of the samples scripts instead of all the scripts. Reason given is to establish consistency in the 1st rater analysis to enable the 1st rater to conduct independent rating to the rest of the scripts.
It is understood that the interrater process is usually used to derive similarity index between two raters for reliability purpose and in medical area especially, interrater involves the whole sample. Would appreciate your views on using selective samples for interrater process in establishing consistency in 1st rater independent rating.
The idea to interrate only a small number instead of all the scripts is upon reflection on the practice in using 2nd rater for on exam script marking. The 2nd rating is done on a small number of the scripts rather than the whole exam paper as the purpose is to establish consistency in the 1st rater rating rather than checking on the similarity between the two markers.
I agree with Mohammad that you get the most accurate inter-rater reliability score if all judgments are included. That being said, I have seen studies in which randomly chosen subsets of ratings or codes were used to calculate reliability. It is crucial that if a sample is used it represents every possible type of decision, preferably more than once., so if the number of categories and items being categorized is small it will not work. It is also important that during training one rater not "indoctrinate" the other rater(s) too much and create a sort of group-think" whereby legitimate differences in how to code data are not allowed to become a possible way to refine the coding categories. I believe this comment has been made before though cannot say exactly by whom-- Lois Bloom maybe-- so I cannot take credit for it.
Dear Mohammad and Tina,
Thank for the insightful answers and references.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs