Hi there,
I am trying to estimate the agreement of judges on the ranking of certain individuals - in particular, the agreement of all other judges with the first judge.
I was thinking to use the Kendall’s W between the first judge and - one at the time - all other judges, skipping the part where I run the Friedman test and the post-hoc test. In the end if I use the Kendall’s test between the first judge and all other judges this measure will include whether they agree or not and which ones agree with it.
When I calculate the Kendall’s W between the first judge and another judge, however, its value is always mysteriously high. Even if the second judge gives the same value to all individuals (e.g., all zeros).
E.g.
0.26 0
0.16 0
0.36 0
0.38 0
0.34 0
... ....
(see attachment)
Kendall's W=0.5
Does anyone have any idea of what the problem might be?
Cheers!
The case of one judge having a range of values and the other judge having all the same value is an unusual one. You shouldn't get any kind of meaningful correlation, but there could be some kind of effect from the software that reports a small correlation.
I wonder if there is some confusion about the different statistics in the output ? Or if there is some error in how the data is being inputted or passed to the software ?
Let's try an example, and see what you get.
If the values for A are 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
And the values for B are 45, 65, 25, 55, 85, 15, 35, 95, 115, 105
You should get a tau of 0.42, a T value of 32, and a p-value of 0.101
If you want to run this in R, you can at: https://rdrr.io/snippets/ , with the code:
A = c(10, 20, 30, 40, 50, 60, 70, 80, 90, 100)
B = c(45, 65, 25, 55, 85, 15, 35, 95, 115, 105)
cor.test(A, B, method="kendall")
plot(A, B)
I think I actually figured out what the problem is! I think that Kendall's tau has a problem with ties. Now I'm looking into inter-rater reliability measures as a more robust solution. Thanks!
I think the text file Emanuele Giacomuzzo attached was meant to have a second column of scores, all equal to 0, like shown below. Using those data, I get the following output using Stata's -ktau- command:
. ktau a b
Number of obs = 62
Kendall's tau-a = 0.0000
Kendall's tau-b = .
Kendall's score = 0
SE of score = 0.000 (corrected for ties)
Test of Ho: a and b are independent
Prob > |z| = 1.0000 (continuity corrected)
0 0.262295081967213
0 0.163934426229508
0 0.360655737704918
0 0.377049180327869
0 0.344262295081967
0 0.377049180327869
0 0.344262295081967
0 0.245901639344262
0 0.262295081967213
0 0.344262295081967
0 0.311475409836066
0 0.344262295081967
0 0.295081967213115
0 0.524590163934426
0 0.491803278688525
0 0.19672131147541
0 0.475409836065574
0 0.278688524590164
0 0.459016393442623
0 0.131147540983607
0 0.475409836065574
0 0.442622950819672
0 0.491803278688525
0 0.327868852459016
0 0.327868852459016
0 0.163934426229508
0 0.229508196721311
0 0.327868852459016
0 0.327868852459016
0 0.262295081967213
0 0.180327868852459
0 0.278688524590164
0 0.377049180327869
0 0.245901639344262
0 0.426229508196721
0 0.377049180327869
0 0.426229508196721
0 0.311475409836066
0 0.377049180327869
0 0.262295081967213
0 0.278688524590164
0 0.60655737704918
0 0.540983606557377
0 0.524590163934426
0 0.540983606557377
0 0.557377049180328
0 0.524590163934426
0 0.475409836065574
0 0.508196721311475
0 0.540983606557377
0 0.639344262295082
0 0.426229508196721
0 0.360655737704918
0 0.508196721311475
0 0.0327868852459016
0 0.229508196721311
0 0.0491803278688525
0 0.0491803278688525
0 0.0327868852459016
0 0.0163934426229508
0 0.0163934426229508
0 0.0327868852459016
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics