Hello, I have 91 items ranked by groups of participants. The rank lists for each group are therefore all the same length and contain identical items.
I know I can perform correlation measures (e.g. Kendall's Tau) or intersection measures (e.g. Rank Biased Overlap) to determine how similar the rankings from different groups are, in general. But are there any statistical tests available to tell me which items in the lists are ranked significantly differently?
Thank you!
This is an interesting question (to which Anand gave a wrong answer).
My first idea is to create a common list of ranks from the two lists of ranks (taking the original ranks as measured values, and get their ranks in the union list). In this list, the difference between the k-th and j-th rank should have a Beta(k-j, n-(k-j+1)-distribution, which can be used to get the significance of the obseverd difference.
See: https://en.wikipedia.org/wiki/Order_statistic#The_joint_distribution_of_the_order_statistics_of_the_uniform_distribution
But I am not sure if this works well in this case, because the union list will have all ties (each original rank value will occur twice in the union list).
I agree with Jochen Wilhelm that it is an interesting question. But I'm not sure I understand the nature of the data. Can you provide a sample dataset (e.g., in CSV format) with 5 items rather than 91 items and 2 or 3 subjects per group? That would help me (at least) understand the nature of the data much more clearly. Thanks for considering.
Jochen Wilhelm thank you so much for taking the time to reply - I will have a look at this!
Bruce Weaver I don't have complete data sets for subjects in groups. Subjects performed 50 pairwise comparisons of a random subset of the 91 items. I used the Elo algorithm to produce a rating and ranking of the 91 items per group, using the data from all subjects in that group. I hope that makes sense! I can still send a sample dataset if that would be useful.
Thanks!
So you can look at the pairwise comparison data rather than the rankings.
Not sure I understand what your research question is. Is it to identify items that are very different from each other (and choose(91,2) is about 4k, so difficult to know what is meant by significantly there). Can you say what your research question(s) is(are)?
Hi Daniel Wright thanks so much for your response. I have 91 items of interest, which can be paired up for comparison with each other (4095 independent pairs). My questionnaire participants are randomly allocated 50 pairs and they have to choose which in the pair is "more important". The outcomes from the pairwise comparisons by all participants, or a subgroup of participants, are fed into an Elo algorithm to produce a rating score and ranking of the 91 items. My research questions are about the relative importance of items as a whole, and where groups of participants disagree - i.e., which items are ranked (or rated) significantly different from each other. I hope that helps!
- Badges
- Science method
- Similar topics
- Biological Science
- Ecology