I have 20 raters who rated 10 stories on two 7 point likert scale. What is the minimum acceptable score for ICC that can be reported in papers?
I have read following chart from Wikipedia. Is this applicable to my situation as well?
Cicchetti (1994)[13] gives the following often quoted guidelines for interpretation for kappa or ICC inter-rater agreement measures:
- Less than 0.40—poor.
- Between 0.40 and 0.59—Fair.
- Between 0.60 and 0.74—Good.
- Between 0.75 and 1.00—Excellent.
That's right. Any variable that correlates at less than .4 with all other variables may be deleted from the analysis (Kim & Mueller, 1978). A key assumption in the domain sampling model is that all items belonging to a common domain should have similar average inter-correlations. Low correlations indicate items that are not drawn from the appropriate domain and that are producing error and unreliability (Churchill, 1979).
