I have a dataset comprised of risk scores from four different healthcare providers. The risk scores are indicative of a risk category of low, medium, high or extreme. I've been able to calculate an agreement between the four risk scorers (in the category assigned) based around Fleiss' kappa but unsurprisingly it's come out very low - actually I managed to achieve negative kappa value. I've looked back at the data and there are many cases where, for example, three of the scorers have said 'extreme' and one has said 'high'. Based on normal kappa, this comes out as disagreement, but of course the cases are adjacent so whilst its not agreement is an awful lot better than, say two scorers saying 'extreme' and two scorers saying 'low' as agreement does not fall into adjacent cases.
I understand the basic principles of weighted kappa and I think this is the approach I need to take but I'm struggling a little with weighted kappa given its multiple raters. Does anyone have any experience on this and advice how is it best to tackle this?
Krippendorff's alpha can deal with ordinal data, have a look at
http://en.wikipedia.org/wiki/Krippendorff%27s_alpha
Hello, Claire.
In your study of reliability there are more than 3 observers and more than 3 categories (low, medium, high and extreme risk). I have worked with weighted kappa with solely two observers and several categories.
I use to work with EPIDAT 3.1 to assess the agreement. It is a free software available in several languages, from Galicia Health Service:
http://www.sergas.es/MostrarContidos_N3_T01.aspx?IdPaxina=62715
I attach an example with EPIDAT: 5 patients, 4 categories of risk, 10 observers. The program shows the global kappa and each kappa per category. Confidence intervals are calculated with a jackknife tecnique.
Hope it helps.
Your interesting question led me to read in deep my notes. I came across these papers, with regard to weighted kappa generalizations. There are several approaches of weighted kappas for multiple raters and it seems there is no clear consensus.
Berry KJ, Johnston JE, Mielke PW. Weighted kappa for multiple raters.
https://www.researchgate.net/publication/24033178_Weighted_kappa_for_multiple_raters.
Warrens J. Equivalences of weighted kappas for multiple raters. Statistical Methodology. 2012;9(3)407-22.
http://www.sciencedirect.com/science/article/pii/S1572312711001171
Regards.
Article Weighted Kappa for Multiple Raters
Thank you Sergio, it's always good to have an extra resource to support your understanding, very much appreciated
High Claire:
I suggest you to see the following web site:
"Cohen’s kappa is a measure of the agreement between two raters, where agreement due to chance is factored out. We now extend Cohen’s kappa to the case where the number of raters can be more than two. This extension is called Fleiss’ kappa. As for Cohen’s kappa no weighting is used and the categories are considered to be unordered" (taken from http://www.real-statistics.com/reliability/fleiss-kappa/)
I can recommend this paper
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/
which recommends that you calculate intra-class correlation for ordinal data to determine the interrater reliability:
"The intra-class correlation (ICC) is one of the most commonly-used statistics for assessing IRR for ordinal, interval, and ratio variables. ICCs are suitable for studies with two or more coders, and may be used when all subjects in a study are rated by multiple coders, or when only a subset of subjects is rated by multiple coders and the rest are rated by one coder."
I just happened to reach here through google searching for answer to a similar question. What about using Kendall's W?
is there any thing which we can do for continuous data like this. apart from bland altman plot, can we do weighted kappa for similar situation? if so, which software we can do it
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics