Two raters assessing 80 essays using a rubric (language use 6 points, content 6 points, organization 5 points, punctuation 3 points).
What if intraclass correlation coefficient is low?
After negotiation, are we, two raters, assess the essays again and again until the correlation is above 0.70?
Hello Hakan,
For essay scoring, you could:
1. Have both raters evaluate the full set of 80 essays independently, and average their scores if they agree to within some pre-set level of agreement that you would designate (for example, if scores on each area agreed to within 1 point), and
2. For essays not meeting that criterion, either: (a) submit the essay to a third rater, and average their scores with the previous ratings which they matched best (either rater A or rater B); or (b) have the two raters jointly discuss the essay until they reach consensus, at which point the consensus ratings are given.
If you're worried that the two raters are not following the scoring rubric similarly, you can use training essays until their degree of agreement (however quantified) reaches whatever threshold you might select. In moderate to high stakes test situations, this step would ordinarily come before turning raters loose on live papers to score.
Good luck with your work.
Sir, after averaging the two best matched scores, do I have to conduct a spearman or cohen's kappa test to see the interrater reliabilty? Thanks.
Spearman is probably safer. It is a Pearson correlation applied to ranked scores, so the two aren't all that different!
Good luck with your work.
- Badges
- Science topic
- Similar topics
- Phytochemistry
- Extracts