I'm attempting to code open-ended responses to a survey asking respondents to like their likes and dislikes of different modes of videogame play. I want to get a sense of the distribution of these likes and dislikes across the different modes of play. Each response is relatively short- from a few words to a few sentences. Each response can generate one or multiple codes. I'm attempting to gain some level of agreement on the codes by comparing the codes generated by 2 raters on a random 10% sample. I've then generated a kappa for each code. The problems I'm running into are these:
1. some codes are so poorly represented in the sample that they generate either perfect agreement (a Kappa of 1) or no agreement.
2. some codes aren't represented in the sample at all
3. some codes are so obvious that they generate a Kappa of 1
My questions are:
1. Should I be concerned about perfect agreement when the code is blindingly obvious? e.g. 'no dislike'.
2. Can I generate a new random 10% sample and only look for the codes that were not significant or generated a perfect level of agreement due to poor representation? Or do I have to look for all the codes, including the ones that previously generated acceptable and non-controversial statistics?
I'm finding it hard to locate information on this kind of analysis and any help would be greatly appreciated.
Thanks!
Check out the following article:
LeBreton & Senter (2008). Answers to 20 Questions About Interrater Reliability and Interrater Agreement Organizational Research Methods, vol. 11 no. 4 815-852
http://orm.sagepub.com/content/11/4/815.short?rss=1&ssource=mfc
Hope that helps!
Hi Sylvia,
I tried to wrap my head around that paper, but it doesn't seem to be dealing with the process that I'm needing information on: ascribing multiple binary ratings to each response (e.g. Response X - yes CODE 1 is present, no CODE 2, 3 & 4 are not present, yes CODE 5 is present) and then trying to ascertain the level of agreement across two raters for each of the codes, across a percentage of the total number of responses. Please correct me if you think I'm wrong! I did struggle a bit with the content.
On the up side it did have some useful suggestions regarding what to do about too high/too low agreement.
Thanks for the suggestion.
Kellie, you are right that the article on multi-level modeling is more complex than what you need for evaluating 2x2 tables.
Your problem seems to be due to using too many low frequency codes, so that some of the tables you are analyzing are nearly empty, and hence too "sparse" to give meaningful results. You thus need to do two things.
First, set a minimum cut-off so that you will not evaluate a code unless it was used at least X times by one or the other coder. What you choose should probably be a function of sample size, so that you might use a cut-off of 5% with a sample of 200, where a code would have to be used at least 10 times before you considered worth considering in your analysis.
Second, you should "collapse" as many of the low occurrence codes as you can into more general categories that thus have higher frequencies. This amounts to organizing your codes into a "hierarchical" format, with more specific codes listed under more general or high-order categories.
Finally, with regard to perfect agreement on obvious codes, that is just fine -- exactly what you would expect.
Hi David, thank you for those suggestions. I was wondering if you would make use of correlation (pearson's r) to determine if certain codes should be collapsed into others?
I don't think Pearson's r will help you because you will still be in a 2x2 framework with very sparse cells, so you almost certainly won't meet the assumptions of either parametric or non-parametric statistical tests. (For a 2x2 Table, r is equivalent to Phi and also related to Chi-square.)
Instead, collapsing is almost unavoidably a subjective task, and just like some things are more obvious in coding, the same is truly in setting up hierarchical relationships.
You should also think about creating an "Other" category to hold things that don't fit into clear substantive categories. In particular, the size of the Other category is important as a piece of data, because it says something about how variable the responses were.
I agree with David 100%, his suggestions comprise the best way to deal with such situations. Recode into fewer categories.
Thanks - and yes, that's exactly what I'm now doing. Much appreciate the help!
Check out Aldwin & Yancura (2006) for the ways we dealt we similar problems. .
I agree with Jagdish and David, Providing more options will not help your cause.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Coding