I have a dataset that has both nominal and ordinal data in it and would like to comput the krippendorff’s alpha on said dataset. Is this possible? If not, is there a way I can compute the krippendorff’s alpha on a single case i.e; a single question that raters have answered?
If nominal= dichotomous and ordinal= for instance Likert 5 scale you may estimate alpha using method= ordinal or nominal.
If nominal= truely nominal (i.e. mom, dad, son, daughter) you should estimate alpha separately.
R script: comments in "Netherlands"
#Title: Bootstrapping Krippendorff's Alpha: the best Inter Rater Reliability (IRR) Statistic (with chance-adjustment)
#Chance-adjusted index: Krippendorff's alpha
#Package: kripp.boot Version: 1.0.0
#Authors@R:Polina Proutskova, email = [email protected] & Mike Gruszczynski, email = [email protected]
#References:
#Proutskova, P. (2014). kripp.boot - an R function for performing bootstrap replicates of Krippendorff’s alpha on intercoder reliability data.https://github.com/MikeGruz/kripp.boot
#Editted by: Dr. PMM van Haard 2020
#Description:
#This function implements Prof. Klaus Krippendorff's algorithm for bootstrapping Krippendorff's alpha coefficient.
#It may compute confidence values (Inter Rater Reliability estimates) for a chosen probability (most often .95).
#Interpretation output:
#alpha > .80 (better > 0.86); biomedical > .95 , and always Lower Limit of 95%CI > .667
#The minimum acceptable alpha coefficient should be chosen according to the importance of the conclusions to be drawn from imperfect data.
#When the costs of mistaken conclusions are high, the minimum alpha needs to be set high as well.
#In the absence of knowledge of the risks of drawing false conclusions from unreliable data, social scientists commonly rely on data with reliabilities
# a = .800, consider data with 0.800 > a = 0.667 only to draw tentative conclusions, and discard data whose agreement measures a < 0.667.
#The minimum acceptable alpha coefficient should be chosen according to the importance of the conclusions to be drawn from imperfect data.
#When the costs of mistaken conclusions are high, the minimum alpha needs to be set high as well.
#In the absence of knowledge of the risks of drawing false conclusions from unreliable data, social scientists commonly rely on data with reliabilities
# a = .800, consider data with 0.800 > a = 0.667 only to draw tentative conclusions, and discard data whose agreement measures a < 0.667.
#NOTA BENE
#If interval_alpha> nominal_alpha may occur because disagreements happen to occur largely among neighboring values,
#visualized by occurring closer to the diagonal of the coincidence matrix, a condition that alpha interval takes into account
#but alpha nominal does not.
#If you have 98% of all responses fall into a single nominal category
#(in this case 'yes') then basically any kind of reliability measurement is pretty meaningless.
#So I suppose that the LOW reliability statistic is technically correct.
#Krippendorff's alpha compares 'observed' disagreement with the 'expected' disagreement.
#The 'observed' disagreement is low in your data, so intuitively we can expect a high reliability.
#But, since the expected disagreement is strongly influenced by the ratio of values (2-to-98 in your case),
#the expected disagreement is very, very low in your case, about 4-times lower than the observed one.
#Combining the two, you will have a low reliability statistic, but as I mentioned, this isn't very indicative in your case.
#KOLOM 1: header=rij 1: leeg; daarna: obj_1..obj_n (objects=variables=(Script Subjects));
#RIJ 1: header kolom 1: leeg; daarna: judge_1...judge_n (judges=raters)
#objects mogen ook gewoon de VARnamen dragen
#judges mogen ook gewoon namen of Ids=nummers zijn (ws. geen PINs!!)
#Clearing the workspace
rm(list=ls());
search()
ls()
#Start script
library(kripp.boot)
library(irr)
library(irrCAC)
#library(agreement) #hier is iets mis mee
library(obs.agree)
library(DescTools) #voor: agree
library(lpSolve)
library(tidycomm)
library(psych)
library(Rfast)
#TESTFASE 1
#the "C" TEST data from Krippendorff 4 raters en 12 objects
#standard format:
#raters=coders=judges (rows) vs objects=subjects=ratings (columns)
nmm
You can also rely on the Alpha Cronbach method, as well as your method.
Hi,
This article talk about Measuring inter-rater reliability for nominal data:
Article Measuring inter-rater reliability for nominal data - Which c...
- Badges
- Science topic
- Similar topics
- Engineering
- Computer Engineering
- Computer