Hello.

I have a question about which test is appropriate in my study (Cohen kappa or Fleiss' kappa)?

My study includes a response variable (categorical scale) with three values: yes, maybe, no. Also, I have three raters. Same raters are used to judge all observations.

One requirement when uses Cohen's kappa is: there are 2 raters. The same 2 raters judge all observations.

In Fleiss' kappa, there are 3 raters or more (which is my case), but one requirement of Fleiss' kappa is the raters should be non-unique. This means that for every observation, 3 different randomly raters are selected. For example, observation number1: 3 different raters are selected to judge it. For observation number 2: other 3 different raters randomly selected to judge it, and so on.

But in my study, I have 3 raters (in this case Fleiss' is suitable), but the same three raters judge all observations (in this case, Cohens' kappa is suitable).

So, which test is appropriate in my study to test the reliability?

Thank you