Effect size for dependent samples t-test can be estimated using Cohen d (divide the mean of the differences by the SD of the differences) or r squared (paired t squared/ (paired t squared + df)). Which one is more recommended to be reported?
Another complex question, is there a way to appropriately compare the paired version of Cohen's d with the Cohen's d for independent data (from another study)?
Alexander - I disagree. Standardizing does not solve the problem of arbitrary scales it merely hides the problem. For the standardized metric to solve the problem you have to assume that the standardiser (SD or variance) is constant across all comparisons. This is a extremely strong assumption that will almost never be true (though it may be approximately true in some cases).
Cohen developed his approach for power/sample size estimation. For this application one generally wants a signal to noise measure (which d is) rather than a measure of effect size (in the sense of an estimate of the practical or theoretical magnitude of an effect).
See
https://www.researchgate.net/publication/23481657_Standardized_or_simple_effect_size_what_should_be_reported
To be controversial: if a measure is uninterpretable unless standardized then it is essentially a bad measure and the standardisation process won't rescue that.
Article Standardized or simple effect size: What should be reported?
hi my friend ... both are fine... but the next one the preferred (r squared (paired t squared/ (paired t squared + df))...best luck
Thank you so much my friend, happy to see your answer... I found articles uses both of them. But I think I have to go with the formula mostly used in articles with the same topic that I am studying (for the purpose of comparison).
Warmest regards...
I personally would prefer d as effect size measure because I find it easier to interpret.
Regarding your second question: It is not possible to compare relative effect sizes like d between studies made under different conditions. But you can compare the absolute effect sizes (= the mean differences) of the two studies. This would make sense to me.
Aside estimating effect size through Cohen's measurements, one can also do that through correlation analysis. This correlation method can conveniently be done using statistical software like SPSS. It is also a wise decision to authenticate Cohen's effect size with correlation analysis.
Thank you.
Dear Thomas, you mean the paired correlation in dependent samples t-test output, is not it?
No please. I mean normal bivariate correlation of a dependent sample with an independent sample to determine effect size.
In medicine we are not just interested in whether a value is statistically significant but equally interested in its clinical significance. A p value < 0.05 is often used as cut off for determining statistical significance but, is it clinically significant? The answer to that can be determined by Cohen's effect size. I am saying in addition to Cohen's effect size, a normal correlation analysis (not in t-test) can also give an idea of effect size.
Thank you.
Cohen's effect size is a relative measure and may therefore not be suited to judge a clinical relevance. It really depends on the context which effect size measure (absolute or relative) might be useful and how the size is to be interpreted. This is pretty much an art, depending on a lot of expertise, a good understanding of the whole clinical setting, and on the context in which "relevance" is to be valued. This is very difficult and there is clearly no simple all-for-one answer that would generally make sense. Using "standards" here may seem kind of objective but it usually is very devastating for the patients, the hospitals, and the society.
So it really is a step in the right direction to abandon p-values for interpretation, but it should not be done by substituting this by other (non-sensical) "standards".
They are both useful, but Cohen's d is easier to interpret (and cite). For that reason, I typically find Cohen's d to be most useful. However, it can sometimes vary based on my audience, domain, and purpose.
The two quantities have different meanings and purposes (and there are other alternatives for specific designs such as pre-post). The former scales d as one would in a non-paired (independent design) and is more useful for comparing effects between studies. The latter excludes error due to individual differences and is more useful for estimating sample size for a new study with a paired design).
Arguably both are inferior to the simple (unstandardised difference in means). Scaling in terms of SD turns the measure into a signal to noise ratio rather than a pure measure of how big the effect is. That is to say, anything that influences the SD but not how big the effect is (sample variability, measurement error, range restriction etc.) will influence your effect size estimate even if you don't want it to. On the other hand, a signal to noise ratio is useful for sample size estimation.
EDIT: As Jochen noted, I misread the Q. My first point was contrasting the comparison of calculating D from the raw (unpaired data) with using the t statistic.
Thom, I very much agree with your 2nd paragraph, but not with the 1st. For a paired design the mean difference would surely be calculated as the mean of the individual differences (between the paired values), so that this measure wil no contain any inter-individual variability. This means that the standard deviation is calculated from the pairwise differences.
Example: data given as pairs of pre/post measures in 5 different individuals:
2,3; 8,7; 34,36; 15;12; 4,3
You can see that the individual values vary greatly, compared to the differences from pre to post measurements. The 5 differences (post-pre) are:
1; -1; 2; -3; -1
The mean difference is -0.4, the standard deviation is 1.94 (also calculated from the 5 differences!), so Cohen's d is -0.21.
Ignoring the inter-individual variance one would look at the pre- and post values separately: (2;8;34;15;4) and (3;7;36;12;3). The pooled standard deviation would be 12.9, the mean difference still is -0.4. So here Cohen's d would be -0.4/12.9 = -0.03. The relative effect measure is much smaller because the variance is larger, and this is larger because it includes the inter-individual variance (which could simply be eliminated by considering the 5 pairwise differences).
Jochen - you are quite right. I skimmed the original message and assumed they were contrasting the:
(m1 - m2)/ SDpooled or (m1-m2)/SDbaseline
formula (that ignores the correlation) with the formula that works directly on t. Serves me right for rushing ...
Alexander - I disagree. Standardizing does not solve the problem of arbitrary scales it merely hides the problem. For the standardized metric to solve the problem you have to assume that the standardiser (SD or variance) is constant across all comparisons. This is a extremely strong assumption that will almost never be true (though it may be approximately true in some cases).
Cohen developed his approach for power/sample size estimation. For this application one generally wants a signal to noise measure (which d is) rather than a measure of effect size (in the sense of an estimate of the practical or theoretical magnitude of an effect).
See
https://www.researchgate.net/publication/23481657_Standardized_or_simple_effect_size_what_should_be_reported
To be controversial: if a measure is uninterpretable unless standardized then it is essentially a bad measure and the standardisation process won't rescue that.
Article Standardized or simple effect size: What should be reported?
Alexander, you raise some good points.
1) Correcting for measurement error is non-trivial (e.g., if you have no estimates of measurement error or errors are correlated). You can do this in latent variable models, but people do this surprisingly rarely. Crucially, however, standardising coefficients in a complex model doesn't avoid problems with interpreting effects scaled in terms of SD or variance. It doesn't address differences in variability between samples, range restriction and the scaling of effects when the variance is a function of the mean or one or more predictors is problematic and so on.
2) The robustness of standardised effect size will vary from data set to data set and there are examples where it isn't robust. Crucially you won't know unless you look also at the unstandardized effect. Examples appear in published work quite a lot. For example in subgroup analysis.
3) This is true, but in my field floor and ceiling effects are detected by looking at the unstandardised measures. These are not visible in d or r.
4) Standardisation can help interpretation but you can also miss things. So I'd rather know that the two groups differ by 0.1 or 0.8 points on a 5 point scale rather than the standardised difference. Relating back to floor or ceiling effects for instance you can get large standardized differences from small absolute when there are floor or ceiling effects. Ideally I'd also prefer to use ordinal regression in these cases (though that can get tricky). Comparing different measures of the same thing is a tricky issue, but I'd rather rescale using POMP scores (or similar). Essentially this identical to standardisation without the rescaling using the sample mean as both assume a 1:1 linear mapping between the two scales. Thus standardisation makes additional assumptions of equality of population and sample variances. You can also part-standardise by just transforming X (if you have a meaningful outcome) or Y (if you have a meaningful predictor).
5) If you can't interpret the raw score you can't interpret the standardised score. Now if you are saying that you don't like the scaling then that's fine - you can convert to a more interpretable scale - but why standardise using the _sample_ SD? If you are going to standardise use the population SD (e.g., from appropriate norms).
I should note that my paper was deliberately provocative because standardized effect size is so widely misunderstood and the paper ends with guidelines rather than strict rules. One compromise is to use both metrics - but if the two metrics lead to different conclusions one would generally trust the unstandardised effect size. Furthermore these are easier to understand for meaningful outcomes.
You make a general point that you see this most in weak research (weak measures but also small sample sizes and questionable practices such as subgroup analysis). This is true - but this is true for most problems. The fact that you can use standardized effect size carefully and cautiously doesn't mean that there an't problems with it. You can do subgroup analyses carefully and cautiously, but generally they are done badly.
Conference Paper The perils of subgroup analysis: the case of standardised effect size
You can calculate the effect size by using regression analysis so you can use Amos program.
You can calculate the effect size by using regression analysis so you can use Amos program.
- Badges
- Science topic
- Similar topics
- Psychology