In my study data, I have matched pairs (same persons with two different observation points (pre/post test)), measured on a 5 point Likert scale which I consider ordinal. I intend to use the Wilcoxon signed rank test to analyse differences, yet there are two questions I found no clear answer yet in the literature that I consulted so far:
1) Effect size r
I’m confused about the effect size calculation with r = Z/squareroot N. In the literature, I find varying definitions of N (assuming I have 10 pairs, 2 have a difference of 0 between pre and post test, thus are eliminated):
- Amount of pairs (in the example = 10)
- Amount of pairs minus those pairs that have a difference of 0 (in the example = 8)
- Amount of observations ((in the example = 20)
What is your view?
2) Constructs/Items
In the case of Likert scales, can I sum the results for the items (observed variables) to compare at the level of constructs (latent variable)? Or how should I perform the test on the level of the constructs?
Thank you so much for your advice!
Susanne
I'll attempt to answer question 1.
a) I recommend using the number of pairs as N. Not everyone agrees, though. One simple argument I make is that if you use the number of pairs for N, the r statistic in theory can range from -1 to 1. If you use twice the number of pairs, the r statistic can only range from -0.707 to 0.707. Unfortunately, the r statistic doesn't always reach -1 or 1, even when the differences in pairs all have the same sign. This is easy to check with a simple data set.
A = (2,4,6, 8,10,12,14,16,18,20)
B = (3,6,9,12,15,18,21,24,27,30)
I get z = -2.8, N =10, r = -0.885
Another argument for using the number of pairs for N is that the statistic would be the same if you used a one-sample statistic on the difference in the pairs, or a two-sample statistic on the paired data.
b) How ties (zero differences) are handled is up to you. [ That is, I don't know of any consensus advice or references. In any case, explain in your methods what you did.]
Imagine you have data like the following, with half of the pairs having zero difference:
C = (2,4,6,8,10,12,14,16,18,20)
D = (2,4,6,8,10,13,15,17,19,21)
Would you like to report the statistic as c. -0.708 or as -1 ?
Here I get a z of -2.24. If I use N = 10, I get r = -0.708. If I use N = 5, I get r = c. -1.
c) I would recommend using the "matched-pairs rank biserial correlation coefficient" (rc) instead of the r statistic we've been discussing. It's simply better behaved. Assuming no zero differences, it will reach -1 or 1 when the differences are all of the same sign. It will tend to tend to higher than r (or greater negative).
A reference is King, B.M., P.J. Rosopa, and E.W. Minium. 2011. Statistical Reasoning in the Behavioral Sciences, 6th ed.
I also think it doesn't make a lot of sense to calculate a z-value on the way to an effect size, when we can calculate the effect size directly from the ranks of the differences, which is what we want in the end.
There are different ways to handle zero differences. Here, I describe three ways: Wilcoxon, Pratt, and none: https://rdrr.io/cran/rcompanion/man/wilcoxonPairedRC.html
For our data set:
C = (2,4,6,8,10,12,14,16,18,20)
D = (2,4,6,8,10,13,15,17,19,21)
For none, I get rc = -0.727.
For Wilcoxon, I get rc = -1
For Pratt, I get rc = -2.67, suggesting that this method isn't great for this kind of toy, unrealistic data.
Sal Mangiafico Thank you so much for your immediate response. It is of great help to me! And a special thanks for the refences and examples. I'll dive into it in the coming days.
as Sal Mangiafico noted, N is the number of pairs. About the second question, I think you mixed up some specific words that cause to confuse me. What is your aim? Please explain more.
Babak Jamshidi Thank you! For my question 2: for example I want to measure the construct of Psychological Ownership which is established to be measured by a multi-item set of 6 items on a 5-point-Likert Scale. I now wonder if I can only conduct the Wilcoxon signed rank test on item level, or also on level of the construct. thanks a lot for sharing your thoughts!
Susanne Marx I don't really work with Likert scales as constructs of summed Likert items. But if I understand the gist of the question....
I would say that the whole point of using this kind of tool is so that you can sum up the results of the Likert items into a scale value for the construct. And it's the construct value that you are really interested in.
Usually people treat the summed values as continuous in nature.
One thing to note here is that if you are summing up the results of Likert items, you are implicitly treating them as interval in nature, not as ordinal. That is, you have to assume that one person reporting a 2 and a 4 sum to be an equal result as for someone else reporting a 3 and a 3. That means that the Likert item response options have to be equally spaced. This usually doesn't bother people.
Also, the first step in the Wilcoxon signed rank test is to subtract the one value in the pair from the other value in the pair. If you are using this test on Likert item results, I think you are assuming that the values are interval in nature. That is, the difference between a response of a 2 and a 1 is equal to the difference between a 3 and 2, and so on.
Thank you, Sal Mangiafico Agree. Thanks for your explanations, makes it all clearer to me. King et al was also a great source for me, thanks for recommending!
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS