To answer these types of questions, I prefer to have the data in front of me - just to make sure that I am working on the right problem. Otherwise, I might end up answering the wrong question.

How often are two codes used, and does the order matter (so is saying AB the same as BA)? If it is only a couple of ones using two codes then you could either ignore the second choice or treat them as different categories (so there are categories A B C D E F AB CD), perhaps collapsing some if they have similar meanings,  and then using Cohen's kappa or some other measure depending on what you are after.

If there are lots of items with two codes then it would get trickier. It might work having the marginal probabilities sum to a value greater than 1, and just calculate kappa as normal. Hopefully somebody reading this knows and responds. 

Interesting problem, thanks for sending this.

Daniel, thanks for thinking about this.

Order doesn't matter and about 20% have two codes - and it isn't consistently the same two codes. Thanks for

Here is a subset of the data, deidentified. Part of the project is to define and apply the codes, so I am not ready to provide definitions for each, yet.

That would make a lot of response patterns so the first approach would not work with n=197.

I guess a simple answer is that you can give a low estimate by anytime either (or both) give two codes, to code the responses so they don't match (if both give two codes you have to choose which codes to use) and a high estimate by coding as a match if either of the codes are the same (and if both are the same, again, you'd have to choose which to use). If all you want to do is show that kappa is above some lower bound using the low estimate might be enough.

An alternative I might try is to calculate kappa with a confusion matrix that double counts those with two codes. Construct the confusion matrix such that if item 1 has rater 1 saying A and rater 2 saying A and B then there are two entries for item 1 (one in the AA cell and one in the AB cell). Then just input the confusion matrix (which now sums to greater than 197) into your kappa function. In R for example

kappa

You can use the Generalizability Theory. It calculates agreement and consistency. I have used it in a similar work: http://www.edesdeproject.eu/images/documents/F%20-%20DESDE-LTC_Usability%20Report.pdf

Every code must be a variable for each rater with two values:0 no present / 1 present

You can use the "EDU-G" software (http://www.irdp.ch/edumetrie/englishprogram.htm)

and also the Genova programs to compute (http://www.uiowa.edu/~casma/computer_programs.htm).

If you need more information, plased contact me.

Carolina, can you give a little more information on how the approach works with having a subset with multiple codes on only a subset. I don't use G theory so have only seen a couple of textbook examples without the special situation of Amy's data.

Let me see if I understand. Have the data as a long rectangular set, with 197*6*2 rows and four columns (one for the response, one for item 1-197, one for characteristic 1-6, and one for rater 1-2). Then use something like

logit(pr(response=1)) ~ b0 + b1-b6 with 5 dummies for characteristics + sigma1 for items + sigma2 for raters

(so in lme4, glmer(resp~as.factor(char) + (1|item) + (1|rater),family=binomial))

and then the sigma2 variance term is a measure of non-consistency? Is this close?

Sorry for any confusion, this is an interesting question.

Here is what I would do:

I assume that when a rater assigns two codes, it is because he/she is (equally?)undecided between them. so my first step would be to "decouple" the data. For example, for transcript 8 you have:

  8 ---- EHS/ ISG ----EHS/TA 

which I would "decouple" as

8 ---- EHS ---- EHS

8 ---- ISG ---- EHS

8 ---- EHS ---- TA

8 ---- ISG ---- TA

Obviously, one has to make adjustments in the calculations to make sure that Transcript 8 is not interpreted as 4 different transcripts or subjects (instead of one in which the raters were not sure how to classify). That can be taken care by using frequency or importance weights (most statistical packages would allow you to do this).

Now, I would simply proceed to perform a Kappa analysis on you data to measure the two inter-rater agreement when there are two or more ratings. I did that with the decoupled data and obtained an observed agreement of about 45% (compared with an expected agreement of just 20%), a Kappa value of about .33 and a corresponding p-value < 0.0001. I am including a spreadsheet with the decoupled data.

Hope this helps. Good luck with your analysis!

Hi Daniel, to use GT you can organise the data like in the example file. The desing of GT model and the software may modify the data structure. If you need more information contact me, GT is a very wide theory and can be used in many different situations.

Carolina.

Carolina, I am going to work on this a bit and will likely get back to you with questions. Thank you!!

Ok, perfect Amy. When you want!