Thanks everyone for your helpful responses !

I thought I would send an update for anyone following this question.

Geoff Cumming provides information for calculating confidence intervals for Cohens dav in chapter 8 of his book "introduction to the new statistics" . There is also a handy calculator within the ESCI software that accompanies this book ( from the following link https://thenewstatistics.com/itns/esci/ select "ESCI for chapter 3-8" and the calculator is within the "Data paired" tab). EDIT: this sheet uses the method outlined in equation 4 of this paper to calculate confidence intervals http://journals.sagepub.com/doi/pdf/10.1177/0013164403256358

Notably calculating confidence intervals for dav does still require Sdiff (which requires knowledge of r). r can in some cases be estimated from exact t or f values as outlined in this article. http://www.tandfonline.com/doi/pdf/10.1080/715114664

If you are using metafor in the R package to conduct your meta analysis then you can use the equation below to then calculate the variance of cohens dav based on the confidence intervals (courtesy of Michael Brannick - who has a bunch of really useful youtube videos I would highly recommend for anyone using metafor) :

Vdav=((CIupper-CIlower)/2*1.96)2

If you do not have r available to calculate Sdiff, using the method to calculate variance for Cohens d will yield a wider confidence interval. Therefore although not optimal this is a conservative alternative with limited information. I calculated variance using both methods to compare the differences. 

Wouldn't it make much more sense to explicitly give both, the (absolute) effect size and the relevant variance(s) of the data instead of intermingling these two in a relative effect size measure (like Cohen's d) and thereby loosing a clear real-world interpretability of the figures?

Hi Jochen, 

Thanks so much for your quick response !

Sorry perhaps I should have been a little clearer in that I am trying to obtain a measure of effect size in order to conduct a meta-analysis of a phenomenon typically measured in a within subjects design. From what I have read it seems pretty standard to use Cohen's d as a measure of effect size for meta-analyses. However I am new to this! Please could you clarify what you mean by absolute effect size and relevant variance(s) so that I am sure I understand ?

Many Thanks, 

Becca

absolute effect size = the actual difference/change/ratio of the response variable. In your example this would be Mdiff. Note that this is the only measure allowing to asses any practical relevance of the impact of the predictor (diseas or treatment etc). 

relative effect size: absolute effect size divided by some dispersion measure. In your example you were asking about what dispersion measure you should take. The answer depends on the context and the purpose for which the relative effect should be reported. I have difficulties understanding the practical value of such relative measures. This is similar for measures like t, F, R² or the famous p-value, which are also relative measures with unclear practical meanings.

The key problem is that you don't know if the reported effect size is large because Mdiff is large or because the dispersion is small. In a meta-analysis, one should (as I do think) have a look at both: effects and dispersions obtained in the studies. If there is one study reporting a vary large d, it might be that d is large only because they had an extraordinary small dispersion. Then the question is: why? Or different groups used different assays/instruments/protocols with differing dispersion: if all measure the same (absolute) effect, they would report different values for d; or if some protocols measure a larger effect but with larger dispersion, the d values would all look similar, although there would actually be evidence that the studies might not all measure the "same thing".

Thank-you for your clear explanations of absolute and relative effect sizes. Please correct me if I am wrong but isn't it the norm to use an effect size estimate like cohen's d for meta-analyses? I am not sure how the absolute effect size would be implemented in my situation.

Thanks, 

Becca

In fact to amend my last answer I think this chapter explains the reasons for choosing raw or standardised effect size measures and helped me understand the problem!

https://www.meta-analysis.com/downloads/Meta-analysis%20Effect%20sizes%20based%20on%20means.pdf

However even if I were to use raw (unstandardised) mean difference I think I would still face the problem as to how to calculate the variance for studies using a within subjects design if a measure of correlation between scores is not known?

i.e. 

even if the difference is given by:

D=    x1-x2

and the variance is 

VD= Sdiff2/n

where n is the number of pairs. Sdiff still requires the estimated correlation between measures, r,  to be known:

Sdiff=sqrt(S12+S22-2*r*S1*S2)

So I am still not sure how to calculate variance in effect sizes for within subjects designs if r is not known? and this problem seems to exist regardless of whether or not the standardized or unstandardized d is used?

Without knowing r there is no chance to do this.

I haven't read the Lakens paper you mention, but this Cohen's dav measure cannot possibly be an accurate reflection of the effect size for a repeated-measures difference. Ignoring the correlation between the repeated measures drops important information. Two datasets with the exact same Mdiff, the exact same SD1, and the exact same SD2 can have very different relative effect sizes if the pairing between measures is different. Here's an example:

Hi Stephen, 

I agree that ignoring the correlation is not optimal. However when conducting a meta-analyses the correlation or covariance between variables is not always available unless you have access to each studies raw data. From what I understand Cohen's dav is for use when the meta analyst faces this situation. 

There is more about how this is calculated in equation 4 and 5 of this paper http://journals.sagepub.com/doi/pdf/10.1177/0013164403256358 and this is also discussed in Cumming's 2012 discussion of the new statistics.

From researching around after this I understand that the correlation could perhaps be estimated from other information in the articles (such as the exact t statistic, f statistic or p value). So perhaps this is another route to take. 

Another conservative approach could be to still utilise the method used for independent groups (final equation in the question). This would not be 100% accurate but the variance calculated should be, if anything, more conservative? 

For a straightforward design with two paired conditions, if you know t and N you can calculate d with basic algebra.

t = diffmean / (diffSD / sqrt(N) )

and

d = diffmean / diffSD

therefore

d = t / sqrt(N)

Hi Stephen, 

Thanks for your response.

Apologies if I have my abbreviations confused. I think you might be referring to calculating basic Cohens d from t and N? 

I was meaning to calculate r from the t value or f statistic.

r = SQRT(t2/(t2+df))

this would allow you to utilise other more accurate estimations of Cohens d for a within subjects design Such as drm :

Cohen's drm=( Mdiff/sqrt(SD12+SD22-2*r*SD1*SD2))*sqrt(2(1-r))

or dav calculate the variance of dav using Algina and Keselman's  method in equation 4 and 5 (http://journals.sagepub.com/doi/pdf/10.1177/0013164403256358).

I see; yes, I was only referring to so-called dz, so if you definitely need the r to calculate other measures then I guess this might not apply.

The topic is a complex one. In general one wants to do the meta-analysis that removes the correlation from the estimate of measurement error error but not from the estimate of standardized effect size (particularly if combing d values from independent and repeated measures studies). This is hardly ever done in my experience. Alternatively do the meta-analysis for unstandardised effect sizes.

The standard approach to missing correlations that I have seen is to impute the missing r value from the non-missing correlations. This isn't a great approach but defensible if most effect sizes have known r. A better alternative is probably a Bayesian approach as this would allow you to mode r as part of the overall analysis and correctly incorporate uncertainty over its true value.

See the attached reference for some alternatives.

Chapter Meta-analysis. Online Supplement 1 to Serious stats: A guide...

Thanks everyone for your helpful responses !

I thought I would send an update for anyone following this question.

Geoff Cumming provides information for calculating confidence intervals for Cohens dav in chapter 8 of his book "introduction to the new statistics" . There is also a handy calculator within the ESCI software that accompanies this book ( from the following link https://thenewstatistics.com/itns/esci/ select "ESCI for chapter 3-8" and the calculator is within the "Data paired" tab). EDIT: this sheet uses the method outlined in equation 4 of this paper to calculate confidence intervals http://journals.sagepub.com/doi/pdf/10.1177/0013164403256358

Notably calculating confidence intervals for dav does still require Sdiff (which requires knowledge of r). r can in some cases be estimated from exact t or f values as outlined in this article. http://www.tandfonline.com/doi/pdf/10.1080/715114664

If you are using metafor in the R package to conduct your meta analysis then you can use the equation below to then calculate the variance of cohens dav based on the confidence intervals (courtesy of Michael Brannick - who has a bunch of really useful youtube videos I would highly recommend for anyone using metafor) :

Vdav=((CIupper-CIlower)/2*1.96)2

If you do not have r available to calculate Sdiff, using the method to calculate variance for Cohens d will yield a wider confidence interval. Therefore although not optimal this is a conservative alternative with limited information. I calculated variance using both methods to compare the differences. 

Thanks for sharing!

Hi Rebecca,  

i'm a little reluctant to impute a pretest-posttest correlation. i wonder if you know any studies that has used the above Vdav formula (without correlation), or is there any paper that has actually recommended doing so?

Hi Everyone,

I hope that this might still be helpful to some people!

I just thought I would let you know that the meta-analysis resulting from this question in which we discuss this problem a bit more is now available online and is in press:

https://www.sciencedirect.com/science/article/pii/S0149763417307674?via%3Dihub

This is a problem I still find interesting and am hoping to focus on a bit more post-thesis!

All the best and happy meta-analysing!

Becca

EDIT: data and scripts used for analysis can be found via OSF https://osf.io/hnszv/

However I owe massive credit to the awesome spreadsheets available on Daniel Lakens OSF:

https://osf.io/vbdah/

and the great youtube tutorials from Michael Brannick:

https://www.youtube.com/channel/UCeUhxAtX2eK0N14KdEVnSzg

Hi everyone,

Keeping this discussion up-to-date given the relatively high and maintained interest in the topic!

Guido Hesselmann @Guido_Hesselmann2 kindly contacted me to advise that R users may encounter an error when copying the formula in the comment above directly into R, and might need to add a second set of brackets:

" Vdav=((CIupper-CIlower)/(2*1.96))2"

Thanks Guido!

Becca

PS. The equation in our manuscript, mentioned above, is still accurate and in line with other tutorials (e.g. Michael Brannick's videos), but creating this note for anyone who may try and copy and paste straight from researchgate to R. As always I would also recommend seeing the supplementary sheet from Daniel Lakens (https://osf.io/vbdah/) for various equations in excel and to help with implementation/understanding.

good luck for you