1- I read that Fleiss kappa can be used. yet, I dont know what rule should I apply for omitting the items?
2- I read another recent method where raters are asked to rate the extent to which statements relate to the several dimensions (for example, having 10 raters, 80 statements and 3 dimensions). I need to know in such a case should I enter the data on SPSS separately for each dimension and compare the means manually? Or is there a different statistical tool that I should be using?. The relevant article is attached.
Thanks.
Hi Hadeer,
I think if you have more than two raters the Intraclass Correlation Coefficient (ICC) might be more suitable than the kappa. I will include a link to an interesting paper that covers some of the methodological issues.
regards,
Christian
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/pdf/nihms372951.pdf
Are you trying to measure how well the raters agree with each other or are you using multiple persons to rate the content validity?
Krippendorff's alpha uses a very flexible approach that accommodates multiple raters.
ICC is a type of correlation, in the following link, a good demonstration using SPSS to run the test
https://www.youtube.com/watch?v=v2d6eyYoh4M
In the following link, simple demonstration of CVI, by using excel
https://www.youtube.com/watch?v=WpBJ2Lo-How
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics