F

Good afternoon,

You asked, "

if you were a reviewer, what level of kappa would be your cut-off. I have some graduate students coding some data and the kappa statistics are ranging from .63 to .80. If you were reviewing, what would you think?

First, are you having pairs of students rate your product?   Cohen's was designed for pairs of raters but he does not not state what level of agreement is 'acceptable'. (sees Cohen's original article (http://journals.sagepub.com/doi/pdf/10.1177/001316446002000104). 

Hale and Fleiss looked at the distribution of the kappa itself (its variance).  It seems if there is a large variance with your calculated Kappa, the level of the Kappa might not be as meaningful. These authors describe the following, "moderate to high values of kappa, K = .50 and .75, under both sample sizes, 25 and 40"see http://www.jstor.org/stable/pdf/2532564.pdf) 

Here is a quote from Sim and Wright in Physical Therapy . Volume 85 . Number 3 . March 2005, page 264,  "The choice of such benchmarks,however, is inevitably arbitrary,' and the effects of prevalence and bias on kappa must be considered when judging its magnitude"

If you are using more than 2 rater's to evaluate a product, you might consider Fleiss' Kappa looks at multiple raters and he also provides a formula for the standard error of kappa.  This last point could help you determine the degree to which your ratings are good or poor.(see http://www.wpic.pitt.edu/research/biometrics/Publications/Biometrics%20Archives%20PDF/395-1971%20Fleiss0001.pdf) 

Have a great day!

Rich

PS...this is also helpful

Large sample variance of kappa in the case of different sets of raters.

Author: Fleiss, Joseph L.

Journal: Psychological bulletin

ISSN: 0033-2909

Date: 1979

Volume: 86 Issue: 5 Page: 974-977

DOI: 10.1037/0033-2909.86.5.974

R

Thanks for your input Richard. To add a little more info, we are using two independent coders. I have been aiming for .70 or above during our practice and training sessions before completing the actual coding.

Hi Gregory,

I think 0.7 is fine, as long as your variance is not too great...have you measured the variance among your responses?

Rich

I do not believe we looked at the variance yet,  but I will examine that next.

Good luck, sounds like an interesting study