Hello Leonie,

Thanks for an interesting question.  Here is a link to an article entitled "On The Krippendorff’s Alpha Coefficient" by Gwet. http://www.agreestat.com/research_papers/onkrippendorffalpha.pdf

He runs some faux data, and has results in which Fleiss differs from Krippendorf by less than 0.02 for the coefficient value.  Gwet uses this formula to calculate the result:

a = pa-pe / 1-pe,

"where pa is the weighted percent agreement and pe the weighted percent chance agreement"

I worked with a colleague on agreement among multiple raters, which was Fleiss' original purpose.  His original article can be found here http://www.wpic.pitt.edu/research/biometrics/Publications/Biometrics%20Archives%20PDF/395-1971%20Fleiss0001.pdf

and his formula is  

k = p(bar) - p(bar)e / 1-p(bar)e

where p(bar) is the mean of all the proportions of the raters who agree, and p(bar)e is the mean of the proportions of raters agreeing solely due to chance.  You can see the similarity between the two formulas, so I would not be surprised at their nearly identical values. Indeed, I think this offers a good measure of validity for your study.

Cheers!

Rich

Leonie,

Check out this resource. I hope this helps.

http://digital-activism.org/2013/05/picking-the-best-intercoder-reliability-statistic-for-your-digital-activism-content-analysis/

Dear Richard, dear David!!!

Thank you so much for your fantastic help...now my aim is within reach.

Greetings from Germany!

Good news...look forward to hearing of your results

Cheers!

Rich

May I please request you for a help in a confusion I am facing in analysis of inter rater agreement.

There is one Kendall coefficient of concordance represented by W. This is a a sound measure of the ranking type of statements where different raters rank different variables, however, it is only applicable to Ranking.

I came to know about the krippendorff alpha as a superior measure. Does this apply to both rating and ranking kind of data?