I am doing an inter-reliability calculation for an instrument measuring conceptual understanding using an open-ended question. The conceptual understanding instrument is composed of 6 related open-ended questions which will be rated using a rubric for conceptual understanding with 3 criteria (interpretation, application, explanation). Students will have scores for interpretation, application, and explanation using a 4-point scale. I have checked Cohen's, Kappa, and other inter-rater reliability measure but I can find suitable tool for this specific problem. Most stat tools only accommodate 1 question and one measure at a time. What could be the best strategy i can employ to objectively identify inter-rater reliability?
I assume that you intend to determine inter-rater reliability among raters of the instrument of 4-point rating scale. These raters are more likely assessors of students, correct me if I am wrong. If they are assessors, you can consider to use Cohen's kappa (two raters), Fleiss' kappa (more than two raters) and Gwet's AC1 (another statistics which correct prevalence bias of kappa statistics). Hope this helps.
Siong Ting Wong Thank you for your response. Yes, you are correct with the assumptions. My problem with the tools you mentioned is that it can only cater one question. My instrument contains 6 different questions. What I plan is do Pearson Correlation r and get the average pairwise run of Pearson R of the three raters.
Your instrument (which I assume is a questionnaire) contains six questions. Pearson's correlation will result in 3 pairs of results (3 correlation coefficients, r): Rater 1 vs Rater 2, Rater 2 vs Rater 3 and Rater 1 vs Rater 3). In addition to r, I strongly recommend you to run Fleiss' kappa and Gwet's AC1 using R software (Gwet, K. L. (2010). R functions for calculating agreement coefficients. [Online] Available from:http://agreestat.com/r_functions.html [Accessed 25th March 2018]) as Pearson's correlation, r is more "crude".
Siong Ting Wong thank you for sharing this information. I highly appreciate your reply.
- Badges
- Science topic