Hello ResearchGate community,
I'm currently analyzing a dataset derived from a survey of ~200 paired responses across two time points. The survey is of teachers and students, using 60 Likert-like items to assess beliefs about education. After factor analysis, I derived three core factors.
I'm now trying to assess the relative magnitude of change in factor scores over two time points. I say relative magnitude because, for all factors, the scores decreased. So I need to see 1) whether changes were significant and 2) the size of those changes.
Preliminary tests, including Shapiro-Wilk, Q-Q plots, and outlier detection, indicated non-normality, guiding me to utilize a Wilcoxon signed-rank test.
However, I'm at a crossroads regarding the appropriate effect size measure. Traditional non-parametric effect size measures, like rank biserial correlation, seem to fall short for my purpose, as they primarily address the probability of difference -- rather than the magnitude of change I'm interested in capturing. I've established that two factors saw a statistically significant change using Wilcoxon signed rank. But I need to understand how big these deceases were and hopefully compare the two.
I'm contemplating justifying the use of Cohen's d or exploring median-based measures for a more accurate reflection of the change magnitude. But I'm struggling to find relevant info online. I've seen references to things like Hodges Lehmann, using simple median change, etc. But nothing solid.
Does anyone have insights or references on how to effectively apply a median measure in this context or justify using Cohen's d with the Wilcoxon signed-rank test for ordinal Likert data?
I appreciate any guidance or shared experiences on this matter.
You can calculate a statistic like Cohen's d. It's just math. The issue is that, if your data are quite non-normal, is dividing the difference in means by the standard deviation a useful metric ?
There is a nonparametric analogue using the difference in medians and the median absolute deviation. It's a simple idea, but I couldn't find any reference to it, so I named it after me †. If you use the median absolute deviation that has a constant of about 1.48 ‡, it will be close to Cohen's d for normal samples.
Or you could just use a un-standardized effect size statistic, like the difference in medians.
It may be helpful to also report the matched-pairs rank biserial correlation coefficient, because it's an r-like statistic, that may be familiar to your audience. You could also use Pearson's r, where one variable is the dichotomous variable, (or Spearman or Kendall) if that fits your purpose better.
† https://search.r-project.org/CRAN/refmans/rcompanion/html/mangiaficoD.html
https://rcompanion.org/handbook/F_05.html
‡ https://stat.ethz.ch/R-manual/R-devel/library/stats/html/mad.html
I think it's also important to consider the test you're using, what it does, and what effect size statistic makes sense. You're using the signed rank test, correct ?
If so, this test starts by taking the difference in pairs and compares these differences to zero (after first rank transforming). So really, the effect size that makes sense assesses how far these differences are from zero. A lot of the effect sizes we're discussing are appropriate for comparing two independent groups, as in the rank sum test.
That's why the matched-pairs rank biserial correlation coefficient makes sense.
If you are interested in the mean, if might make sense to use a one-sample permutation t-test.
But probably you just want to use a different model. Have you tried a simple transformation on the measured variable ?
Also, have you examined the distribution of the differences in the response variable ? That's all that matters for the t-test.
If you want Cohen's d, it might make sense to use the permutation t-test, and then maybe bootstrap the mean of differences and the sd of differences and use that for Cohen's d.
Adam R.S. Barton
In addition to the answer Sal Mangiafico
https://easystats.github.io/effectsize/articles/standardized_differences.html - compressive review.
https://www.taylorfrancis.com/books/mono/10.4324/9780203803233/effect-sizes-research-robert-grissom-john-kim - book.
- Badges
- Science topic
- Similar topics
- Biological Science
- Physiology
- Physiological Processes
- Stress