I'm conducting nonparametric analyses using Conover post-hoc tests - specifically following a Friedman test (for repeated measures) and following a Kruskal-Wallis test (for independent samples). While p-values indicate significance, I need to report effect sizes as well.
Should I use r = t/√(t²+df), where t is the t-statistic from the Conover test? For clarity, the t-statistics are defined as:
For Conover following Kruskal-Wallis:
t = |R̄ᵢ - R̄ⱼ| / √[S² × (1/nᵢ + 1/nⱼ) × D]
Where R̄ᵢ and R̄ⱼ are mean ranks, S² is rank variance, and D is a correction factor
For Conover following Friedman:
t = |Rᵢ - Rⱼ| / √(A × B)
Where Rᵢ and Rⱼ are rank sums, and A and B contain adjustments for the block design
Is there a more appropriate effect size metric specific to these rank-based post-hoc tests?
No, I don't think that would make sense.
(And it usually doesn't make any sense converting a t-test to r; only in some cases. But that's another story.)
You want an effect size statistic for each comparison, right ?
There are effect size statistics for the overall KW or Friedman test.
For the individual comparisons, for KW, I would use the Glass rank biserial coefficient. (Equivalent to Cliff's delta.) And there are a bunch of others.
For individual comparisons, for Friedman,
if you want to treat the comparisons as if they're like a Wilcoxon signed-rank test, you would use matched-pairs rank biserial correlation coefficient.
if you want to treat the comparisons like a paired-samples sign test --- which Friedman test is more like --- You can use a dominance statistic, which is just the proportion of paired differences that are greater than zero.
What are you trying to do? There is no single effect size metric usable for all purposes. Rank based and related tests like the sign test are particularly difficult to get useful effect sizes from.
For instance if you want to compare effects with the same data and different analyses then tests compared from the ranks and so forth might be reasonable. I imagine these can also be useful for power analysis. (Though I'd rather simulate from raw data I think if I had to do this).
For comparing between studies or for most practical applied purposes they will be pretty useless.* Usually you want a metric in the original units or some easy to interpret transformation of them. For rank based tests that probably involves some kind interpolation. For this reason if the size of effect needs to be interpretable and useful I'd avoid these kinds of statistical approaches and use something like a bootstrap or a Bayesian model that makes it easy to handle weird distributions in the original metric and then get a useful estimate of a difference on a meaningful scale.
* This applies also to standardized effect sizes like d or r or R^2 which are likewise not great when comparing between studies or for practical interpretation.
- Badges
- Science method
- Similar topics
- Teaching