I have been coding the likes and dislikes of playing videogames in different modes of play. After merging categories, ditching some that had too few cases, coding and recoding, and finally generating Kappas between two raters I have some codes that have have non-significant Kappas, some that are significant but low (under .4) and some that are significant and high (over .8).
Two questions:
1. I would assume that the non-significant Kappas indicate that those codes need to be looked at again and possibly eliminated or merged where possible, however would the same hold true for codes that generated low Kappas?
2. How would you treat the difference in agreement between these codes in terms of their interpretation and application?
Thanks!
In do not know the mathematical definition of a Kappa code. t sounds interesting except for the questions concerning the applications. In coding theory the only question
is how the distributions defining two random variables behave. Knowing one we need to know what is needed to describe the other. The details of the application are not needed.
Your questions are interesting. As I said I am not familiar with Kappa codes.
Sincerely, Jorma Rissanen
I am confused? But I will try to help.
Coding is typically used to QUALITATIVE Methodology. Your research on video games likes and dislikes will be best suited for Qualitative study. The application of Cohen's Kappa may not be suitable with the Qualitative Methodology.
For your 2nd question in coding. The research presents the data as is. If both data are in disagreement it is better to call back the informants to clarify their intentions. Again this answer assumes that you are following the Qualitative Methodology.
Dear Kellie Vella.
In my humble opinion, one of the usage of Cohen's Kappa is to check for interrater reliability between raters. You may review some papers regarding interrater reliability though. Basically, some researchers use those formula in their qualititative nature of data, e.g. having to code from documents.
Then, I agreed with what Mr. Aljoriz Dublin said above, if your data is some kind of dichotomous type (i.e. like & dislike), there might be other tests that suitable with it. However, if you insist to use it, modestly I would like you to refer to Mr. Dublin's last paragraph for both of your questions :-)
Cohen's Kappa is very much dependent on the raters and the processes beneath the coding exercise.
These are all my 2cents though, happy researching.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Coding