I've generated pairwise comparisons after running change over time analyses on my data (I have 3 time points so have used multilevel modeling). I now know the mean difference from each time point to the other, and whether this is significant. However, I'm unsure how to calculate effect sizes for these differences. SPSS has automatically performed a Sidak adjustment for multiple comparisons. If anybody knows the answer to this I'd be extremely grateful, thanks!
An effect size, in your case, is the difference between the means. To get a relative effect size you have to divide it by the standard deviation of the residuals.
Is it sensible to compare the differences between time-points? Is it possible to fit a model of the change over time? It would better use the available data and possibly provide results that are better interpretable.
I have found changes over time using MLM but ran this analysis to find out if there had been more change between baseline and time 2, or baseline and time 3. I wasn't sure if it's a Cohen's d, Hedges' g, or something else I need to calculate!
The adjusted Hedge's g (g*) should be the best. Cohen's d is slightly more biased. However, the difference is negigible for moderate to large samples. If you have clearly unequal variances in the time points you might consider using Glass' d.
I'm not sure if this information helps, but here goes:
Hedges g statistic can be used to measure effect size for a pairwise contrast in a multiple comparison procedure following an ANOVA test (eg. Tukey, Fisher-Hayter) – a measure of effect size related to Cohen’s d. This measure indicates the proportion of DV variance accounted for by the two selected conditions of the IV. This measure is based on dividing the difference between the two condition means in the comparison by pooled variance (the square root of MS_ERROR).
As with Cohen’s d, a g value of 0.2 or lower is regarded as a small effect, a g value of around 0.5 (plus or minus .29) is a medium effect, and a g value of 0.8 or higher is a large effect.
If this is what you are looking for, I have a worked-through example I could email to you, if you want (if I try to copy it in here, it gets 'mashed up'!).
- Badges
- Science topic