I am attempting a meta-analysis of the effectiveness of interventions using mean gain scores (no other useful data is reported in my pre-post studies which are not all randomised) and am aware that I need the pre-post means, SDs, sample sizes and correlations in order to do this. RevMan appears only to allow the input of 'final score' means, SDs and Ns which is inappropriate for this type of analysis. RevMan also seems unable to compute study effect sizes from mean gain scores and does not appear to allow the overall study effect size (computed using an online effect size calculator) to be entered directly and used to produce plots (eg funnel plots). Is there any (preferably free) software which will be able to calculate Hedges g from the data I have, and produce plots whilst also being usable for someone who is a novice to meta-analysis?
In addition, in the situation described above, Rosenthal (1993) suggests using a 'conservative' estimate of the pre-post r, 0.7, in the calculation of ES from mean gain scores, and this is the approach adopted in recently published meta-analyses (eg Khoury, 2013), however, more recently published literature on statistical methods, and answers to recent posts on this site, suggest a correlation of 0.5 should be used in conjunction with a sensitivity analysis. Is this the new 'standard'? Is there a standard approach?
Hi Caroline,
First, on free meta-analysis software, here
http://cran.r-project.org/web/views/MetaAnalysis.html
is the R task view for the couple of dozen meta-analysis packages written in R. meta, rmeta, and metafor seem pretty popular in what I have read.
Second, I assume you are trying to estimate the population effect size for the difference in post-pre. Do the papers you cite have this mean difference but not the SD of this difference? You might need to estimate the SD from the other statistics, but this would seem more straight-forward.
Third, are gain scores all you are interested in? Often it is preferable to predict post from pre (see references on Lord's paradox, particular Wainer (1991, Psych Bull). For other papers see references in the linked paper). Might be worth comparing the results from these in your meta-analysis.
Dan
Article Comparing groups in a before-after design: When t test and A...
Hi Dan,
Thank you for your response, and the article you attached is very interesting.
R looks like it would be perfect, however, I'm unfamiliar with the programme and believe learning the computer 'language' for it could be problematic, given I have no experience with different computer languages. I'm a researcher in occupational psychology, so use SPSS most of the time.
My papers don't state the pre-post mean diffs or SDs, though the former can obviously be easily computed. The latter, I thought, were reliant on knowing the pre-post correlations also, but these are also not reported. None of the studies report t-tests based on the difference in gain scores (any t-test results reported are based on differences between groups' final results) and results from repeated measures ANOVAs / MANOVAs vary based on the number of time points which have been taken into account and so I'm not sure the results of these are comparable. I'm not sure how I would estimate the mean gain score SDs from these statistics?
Thanks again for your help,
Caroline
R is really useful, but it has a different type of interface from SPSS and most people say is more difficult to learn. I'll put a brief intro on it below (I used to use it and SPSS when I taught psych students in the UK) and lots of info on http://www.r-project.org/other-docs.html. (and there will be questions on ResearchGate asking for info on R).
As far as the measures, what are the stats that are in most of the papers?
Chapter A very brief introduction to R (ch. 1 of Wright & London, 2009)
Hi Dan,
Thanks for your quick response. I will look into the links, thank you.
As for the measures, the only ones which are consistent across the studies are the pre-post means, pre-post SDs and pre-post sample sizes. The others are a range of ANOVA and MANOVA results indicating F and p values, some of which include df, some of which don't, and some of which include eta sq / partial eta sq and some of which don't. Not all the p values are exact. Hence, and given the non-randomised design of several of the studies, I decided my best bet was to use the pre-post means and SDs to guage the effectiveness of the interventions but with no pre-post correlations reported I ran into the issue of computing mean gain score SDs, which RevMan (which had been suggested to me) doesn't appear able to deal with. Hence my foray into looking at estimations of r based on Rosenthal's (1993) work...
Any thoughts / suggestions would be much appreciated, thanks.
Caroline
It is a pain some authors don't report the sds of the gain scores. And if the studies are old they may not have those values handy.
Thanks Diana, I will take a look. I've also been experimenting with a free trial of Comprehensive Meta-Analysis which is very easy to use. It will be interesting to compare it!
And Dan, I agree, it is frustrating that the stats for meta-analysis are not all reported by authors. My studies are very recent, actually, but it doesn't seem to be the norm, at least in my field, to report gain scores and their SDs. Still, thanks for your help, I'm much clearer now about a way forward anyway.
There is additional software for R, which makes it easier to use if you're just getting started: Deducer.
It makes the interface more user-friendly. One could compare it to SPSS. You can click on buttons etc.
http://www.deducer.org/pmwiki/pmwiki.php?n=Main.DeducerManual
You may have a look at:
http://davidakenny.net/meta.htm
http://www.danielsoper.com
best regards,
JP
Thanks Donna, that resource has indeed been very helpful.
All the best,
Caroline
To Cohens' d you can use the excel provided by Cummings (http://www.latrobe.edu.au/psy/research/cognitive-and-developmental-psychology/esci)
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics