You need to give number for each rank, then analyze. Regards

The question is a little unclear. Do you have, e.g.

"Q1A was too hard." - SA, A, N, D, SD

"Q1B was too hard." - SA, A, N, D, SD

And then you want to see if Q1A was rated more hard than Q1B ?

In that case, you probably want to treat the responses to each item as ordinal categories.

Important question: Are you recording data so that you know the response from Student A on Q1A and from Student A on Q1B ?

If yes, you might use a two-sample sign test for each comparison.

If no, you might use Wilcoxon-Mann-Whitney for each question.

In either case, a good implementation of ordinal regression would be an ideal, though potentially a little more involved.

Hello Katie,

If your intention is to compare individual question pairs, and participants have responded to both of the pair members, then the answer depends on what your specific research question is:

1. What is the degree of consistency in ratings of the alternate worded pairs?

Cohen's kappa would work for this.

2. Is there a relationship between ratings of the alternate worded pairs?

Chi-square test of independence (for significance); Cramer's V for a correlation estimate.Alternatively, the Spearman rho correlation coefficient could be used as a measure of strength of relationship.

3. Is one item consistently rated higher than another (vs. the null hypothesis of no difference)?

Wilcoxon signed-ranks sum test; McNemar's test (for 5 categories---sometimes called Bowker's extension); Norman Cliff's proportion dominance statistic; Binomial test. All are non-parametric methods.

Should you be running many such comparisons (for example, more than 3 pairs), you should give thought to adjusting the significance level of individual tests so that the aggregate risk of Type I error for the set of tests is not intolerably high.

Good luck with your work.

Here's another question for you, Katie Reid: Is there some systematic difference between the A and B versions of each question? Or is the assignment of A and B labels arbitrary? I assume (and hope) it is the former situation. If so, what is the difference? Thanks for clarifying.

Sal Mangiafico - thank you for your excellent questions - yes I want to see if Q1A was rated more hard than Q1B ? I'm not recording the results from each student. If I was to use the Wilcoxon-Mann-Whitney for each question - would I divide the data into 5 groups, representing the levels of agreement or 2 groups representing Q1A and Q1B, I feel like it the the latter.

David Morse thank you for your brilliant question, I am trying to establish your point 3, Is one item consistently rated higher than another (vs. the null hypothesis of no difference)? in which case presumably I analyse by way of 2 groups, i.e. Q1A v. Q1B?

Bruce Weaver thank you for your excellent observation, there is a systemic difference between Q1A and Q1B, all the A questions contain an elongated question stem and all the B questions and abbreviated question stem.

Thanks for clarifying, Katie Reid. It sounds as if the data file could be structured as follows:

ID quest stem toohard

1 1 1 rating

1 1 2 rating

1 2 1 rating

1 2 2 rating

1 3 1 rating

1 3 2 rating

etc.

2 1 1 rating

2 1 2 rating

2 2 1 rating

2 2 2 rating

2 3 1 rating

2 3 2 rating

And it sounds as if your main interest is in comparing ratings for elongated vs abbreviated stems. If so, it looks to me as if a multilevel ordinal logistic regression model would make sense. I don't think you've said what software you use, but if you do a search on with the name of your stats package tacked on the end, you should get some good hits. HTH.

Katie Reid ,

If you are going to use the WMW test to compare the two questions, you use two groups, Q1A and Q1B, as you suggest. You just translate a response of SA to a 5, A to a 4, N to 3, and so on. *

So if you had a six responses for each, Q1A was (SD, D, N, SD, D, N) that would translate to (1, 2, 3, 1, 2, 3). And let's say Q1B was (N, A, SA, N, A, SA), that would be (3, 4, 5, 3, 4, 5)

The WMW test would be, in R:

Q1A = c(1, 2, 3, 1, 2, 3)

Q1B = c(3, 4, 5, 3, 4, 5)

wilcox.test(Q1A, Q1B)

### Wilcoxon rank sum test with continuity correction

### W = 2, p-value = 0.01093

You will probably want to look at the effect size in some way, and maybe express the results in bar plots of counts of each category for each question.

One method in R:

library(rcompanion)

cliffDelta(x=Q1B, y=Q1A, verbose=TRUE)

### Statistic Value

### Proportion Ya > Yb 0.889

### Proportion Ya < Yb 0.000

### Proportion ties 0.111

###

### Cliff.delta

### 0.889

So here, if you were to compare all possible pairwise values for the two groups, Q1B > Q1A 88.9% of the time, and Q1A > Q1B 0% of the time. This is summarized in the Cliff's delta statistic.

______________

* The numbers don't matter as long as you are consistent and they are in the correct order. For example, you might translate (SD, D, N, A, SA) as either (1, 2, 3, 4, 5) or (-2, -1, 0, 1, 2). It won't matter.

After reading Sal's most recent post, I have another couple of questions for you, Kate.

1. How many questions are there?

2. Are you interested in those questions specifically? Or are you thinking of them as a random sample of some larger population of questions, with the real interest being in the (fixed) effect of elongated vs abbreviated stems?

I would guess you are thinking of the questions you have as a sample from some larger population of questions. But I may be wrong.