Hi Guys,
I have adapted and normed an assessment that includes 25 rating sections (rating scale from 0 to 8 [9 points]) - so the data is ordinal. For each of the 25 sections I have completed a test-retest on 9 individuals. The data for each individual cannot be collapsed together as each section provides a different measure. What statistical test should I use to evaluate the concordance between test-retest? I was looking at Kendall W or Tau.
Can you please help? Thank you.
With multiple raters and ordinal data, I think the Kendall W would be most appropriate. For a lot of detailed information, references (including websites, articles, and textbooks), and a nice explanation I found the linked site really useful.
https://stats.stackexchange.com/questions/3539/inter-rater-reliability-for-ordinal-or-interval-data
Hi Kenneth,
Thanks for your reply. I am actually looking at test-retest here. Don't know if this is applicable to what I need.
If you will be comparing each question, you will want to use a method that can handle ordinal data. Ordinal regression is a good option, especially if the software you use allows for mixed effects so that the repeated measures nature of the data can be taken into account. A slightly less complicated option is to use permutation tests that are appropriate for ordinal data. The simplest approach would be to use the two-sample paired rank-sum test. This is a traditional nonparametric test, and it is somewhat controversial to use this kind of test with ordinal data.
Examples in R of each of these are at the links below.
On the other hand, if you total responses for each section, you might be able to treat the data as continuous. You might then look at paired t-test, but be sure you meet the assumptions of this test. You could use the nonparametric two-sample paired rank-sum test on the summed scores as well.
http://rcompanion.org/handbook/G_04.html
http://rcompanion.org/handbook/K_03.html
http://rcompanion.org/handbook/F_06.html
- Badges
- Science topic
- Similar topics
- Instrumentation Engineering
- Instruments