Hello Jari,

I think it safe to say that Jacob Cohen identified multiple effect size metrics because there are multiple kinds of statistical comparisons that can be made.

The fact that under certain conditions (e.g., 1 df hypotheses), Cohen's d and Cohen's f (and, for that matter, Cohen's f-squared and Wilks' lambda) can be shown to be derivable from one another doesn't mean that Cohen's d is applicable to other conditions (e.g., multi-df hypotheses).

The general basis of of d (a measure of a single DV score difference in the metric of SD units) is different from that of f or f^2 (or lambda) which are derived as functions of variance accounted for indicators of ES, and are so described in his text (e.g., f^2 as analog of signal to noise ratios).

Does that help in any way?

Good luck with your work.

Thank you David! I was counting on you... These simplified formulae disturb me. I'm preparing revisions in an interesting ms. dealing with combining estimators of reliability with effect sizes. This challenge of comparing f and d came evident because I need to give the quick-and-dirty threshold for small, medium, high (and so on) ESs. In the empirical dataset, I noticed that there is a systematic error in f in relation with d when using the traditional formula of f (based on eta squared) which appeared to correspond with d only when p = .50. I needed to find relevant "real" or general formula for f in relation with d. I just noticed that the important correction factor (SQRT(pq) seen in the formula using Rpbis and t-test) was missing in the formula of f. Even Cohen seems not mention that. Just pondering would it be possible that we are missing something in the basic form of f (using eta2 or F-test or partial variances) that would react to the unequal sample sizes in polytomous settings the same manner as we need in the binary/dichotomous settings? If so, what could that corrction be? Why we need that SQRT(pq) in the binary case? Where it comes? What it indicates? And, more important, what would be its parallel in the polytomous, multinomial case...?

Despite the fact that Cohen devotes a few pages to comparing f to d, for me, the point is that f is a more related to r-squared than to d.

eta-squared = f-squared / (1 + f-squared).

So, f is just sqrt ( eta-squared / ( 1- eta-squared ) ).

I think this avoids the problem with the quick-and-dirty thresholds, because d and f are different kinds of effect size statistics. Cohen gives thresholds for each, but they won't line up in all cases (like different sample sizes in two groups). Just like d and an effect size statistic for non-parametric test won't line up, unless additional assumptions are made.

Sal Mangiafico, Thanks! I buy the idea of eta squared and r squared being more comparable - after all they ARE identical with binary/dichotomous cases and close each other in the ordinal categorical case. Anyhow, there IS a strict technical/algebraic relation between d and f. My challenge with f vs. d is that the thresholds for low, medium and high ES are traditionally given/suggested so that the thresholds for f are 1/2 of those given for d as if f = 1/2d (i.e., 0,2 - 0,5 - 0,8 for d vs. 0,1 - 0,25 - 0,4 for f). However, this 1/2 appears to be SQRT(p1p2) which changes the thresholds dramatically for f when p1 ≠ p2. If, for example, p1 = .90 and p2 = .10, the corresponding thresholds would be 0,2 - 0,5 - 0,8 for d vs. 0,06 - 0,15 - 0,24 for f (assuming that those for d are "correct" or "justified"). On the other hand, If we think that those for f are "correct" or "justified", then the thresholds for d are incorrect except the very case that p1 = p2; instead of 2f, they should be fx(SQRT(p1p2)^-1. Then, for p1 = .90 and p2 = .10, the thresholds would be 0,1 - 0,25 - 0,4 for f vs. 0,46 - 1,15 - 1,84 for d). Just trying to solve this puzzle: which would be the better way think the possible correction...? Anyhow, the strict technical/algebraic relation between d and f hints that there is something to alter in our practices, I think...

I think the underlying issue is that they're just different effect size statistics, giving different information.

We can't expect the threshold values to correspond across different types of effect size statistics. At least without additional assumptions about the data (or sampled populations).

Take a simple example. Let's look at non-standardized effect size statistics. We could indicate the effect size with the difference in means or the difference in medians. If we take the role of Cohen, let's say we decide that a medium effect size for income is a difference of $10,000 per year. It's obviously possible to have a medium effect size considering the difference in means, with a small effect size considering the difference in medians. Unless we make additional assumptions about the distribution of the data (or populations).

I think I got the point. I buy the idea that f and d may tell about different kind of information. I'm just thinking the matter from the practical users viewpoint - and from the formulae viewpoint. From the practical users viewpoint, the traditional cut-offs seem to be based on wrong (= simplified) formula leading (in most cases) to a wrong interpretation of the effect size. From this viewpoint, I like more the "common language" estimators: the result means something practical. From the formula viewpoint, both f and d measure the (summed) mean difference in relation to the weighted group variances or standard deviation (see the familiar forms in the appended fime). From this viewpoint, I do not see much difference in interpretation. Or did I miss something? While formulae are almost identical, could it be just a coincident that they measure same thing only in the specific dichotomous case? Maybe?

I think you're right about the common language interpretation. But as you've noted, the devil is in the details. (Sample sizes. Maybe how variances are pooled ? Or if variances are constant ? )

For me, I think it's more important that f be consistent with r-squared. I've never checked with Cohen's calculation of f to see if it's always consistent with r-squared.

I think it's also valuable to note that f isn't used much as an effect size. Except when it's used in power or sample size calculations.

Usually, people will either use Cohen's d for pairs of treatments † or r-squared for anova-type models (including eta-squared, partial eta-squared).

†Which can also be adapted to more complex models, though I don't entirely understand it. See: https://cran.r-project.org/web/packages/emmeans/vignettes/comparisons.html , and search for Cohen.

Need to agree... Personally, I'm using quite often f because of working mainly with simple anova/GLM which gives eta square strictly. The formula of f is simple to use even with two groups. Hence, it was a dissapointment that the thresholds are not fixed, seems so. It is a nuisance that, by parroting, I have given faulty thresholds in my textbooks...

This option of using common language ESs seems temptating in a sense that those could be used as a new kind of benchmark for the thresholds of f (remembering that Cohen used population overlap as the criterion). The thresholds for CLESs could be used as a new type of criterion (a kind of "80/100 refers to high effect size, 90/100 refers to very high, and 95 refers to huge effect size"). Namely, I have shown that a simple transformation of Somers' D and Goodman-Kruskal G can be used as CLESs not only in the binary/dichotomous settings but also polytomous settings (see Preprint Somers' delta as a basis for nonparametric effect sizes: Gri...

Jari Metsämuuronen , that pre-print looks very interesting. I'm eager to dig into it, but I won't have time now. I appreciate that you included a worked example.

I'm curious to see how what you're working on compares to Freeman's theta and the epsilon-squared used for Kruskal-Wallis.

P.S.

In my rcompanion R package, I've tried to include many effect size statistics for nonparametric tests and ordinal variables.

Missing from the list in the documentation is Grissom and Kim’s Probability of Superiority, Glass rank biserial correlation coefficient, Agresti’s Generalized Odds Ratio for Stochastic Dominance, Dominance statistic for one-sample data, Dominance statistic for two-sample paired data, Difference in medians divided by the pooled median absolute deviation

Cramer’s V, Cohen’s g and odds ratio for paired tables, Cohen’s h, Cohen’s w, Vargha and Delaney’s A, Cliff’s delta, r for one-sample, two-sample, and paired Wilcoxon and Mann-Whitney tests, epsilon-squared, and Freeman’s theta.

Like Sal Mangiafico, I don't have a lot of time right now to study this thread carefully. But a quick glance at some of the posts made me wonder if this 2020 article by Correll et al. might be of interest.

https://www.sciencedirect.com/science/article/abs/pii/S1364661319302979

Article Avoid Cohen’s ‘Small’, ‘Medium’, and ‘Large’ for Power Analysis

Cheers,

Bruce

I added comments specific for "Somers' delta as a basis for nonparametric effect sizes..." at the publication link: Preprint Somers' delta as a basis for nonparametric effect sizes: Gri...