I'm preparing a final test for assessing students' speaking performances. To avoid subjectivity, I'm wondering about the important considerations in selecting the inter-raters. Also, I'm wondering about the rigor in selecting the inter-raters. Really appreciate your responses.
I have not seen formal criteria for selecting multiple raters, but your raters need to have sufficient ability to properly do a good job of rating the performance of the performances.
For evaluating inter-rater reliability, I like Krippendorff’s Alpha.
Hayes and Krippendorff (2007) said that a Kalpha of 1.000 represents perfect reliability whereas Kalpha = .80 is often thought of as the norm for good reliability.
A minimum of .67 or even .60 may still be satisfactory, if the coding is complex, or subjective (de Swert, 2012).
I think that rating English speaking performance IS subjective, so I think idea about Kalpha is good.
I also think that if the Kalpha is not high enough, it is legitimate to ask the raters to reconsider their subjective evaluations.
Here are the references.
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77-89.
de Swert, K. (2012). Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha. Retrieved from http://www.polcomm.org
/wp-content/uploads/ICR01022012.pdf
@Michael W. Marek, thank you so much for your valuable suggestion. Regarding the process of inter-rater reliability test, should I invite the raters to have a joint training session to reach an agreement in interpreting the assessment rubric or directly go to independent rating process without a joint training session?
May be this file will be sufficient.
http://webcache.googleusercontent.com/search?q=cache:v13w9aqrnOgJ:www.ccsenet.org/journal/index.php/ijel/article/download/67243/37740+&cd=19&hl=en&ct=clnk&gl=gr
The raters should be knowledgeable and proficient enough in the speaking skill. Also they need to know about the criteria/standards that you determined to assess students' performance. Usually, in reliable famous tests such as IELTS, speaking is assessed based on 4 criteria of Fluency and Cohesion, Lexical Resources, Grammatical Range and Accuracy, and Pronunciation.
- Badges
- Science topic