Don't rush it. I would start by reading some good text books on meta-analysis such as

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, USA: Sage Publications.

and

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Susse, UK: Wiley.

Then go from there.

Dear Ted,

To clear it out the outcomes are usuallly presentes as (1) baseline value, (2) end of study value or/and (3) mean change. Each of these values has its own meassure of variablility and you shouldn't use mean change together with SD corresponding to end of study.

The best option and is to meta-analyze mean changes as these values are independent from baseline variability. sometimes it is however impossible to extract mean change and than end of study values may be used with caution in the same meta-analysis.

What you need is a valid estimate of the treatment effect (Experimental - Control) from the given study and a valid estimate of the standard error. These are the inputs to a meta-analysis. The treatment estimate does not have to be based on change scores (outcome -baseline). Note that change scores do not deal with baseline imbalance and so checking for baseline imbalance does not form part of any rational strategy for choosing between change scores and raw means. In fact neither is logical and analysis of covariance is better than either. However, if you are doing a meta-analysis you are stuck with what authors have used, so you should use whatever measure they have taken.

Pace, Stefano, the question of fixed or random effects is a separate issue. The former approach is useful for establishing if there is an effect. the latter my be superior in estimating what it is. See Senn, S. J. (2000). "The many modes of meta." Drug Information Journal 34: 535-549.

Stephen

Ted,

Einar cited very good positions but I understand you may have not enough time to read them carefully:) However please read these extracts:

http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter_9/9_4_5_meta_analysis_of_continuous_outcomes.htm

http://www.mrc-bsu.cam.ac.uk/cochrane/handbook500/chapter_16/16_special_topics_in_statistics.htm

Comparability is the key issue. Pre-post effect sizes and post-test only effect sizes are different. One is the effect size of the difference in gain scores between the two groups (the ratio of the gain relative to the spread), the other is the difference between two groups (the ratio of the score relative to the spread). The standard deviations of the gains (pooled between both groups) and the standard deviations of the scores (pooled between the two groups) will probably look quite different. I disagree with Stefano, I think most meta-analyses in education use the difference between the the groups at post-test (ie the difference between the groups rather than the difference in the gain).

Lee Becker's pages are good: http://www.uccs.edu/lbecker/index.html

and Rob Coe's http://www.cemcentre.org/attachments/ebe/ESguide.pdf

I disagree with Steven that these two approaches measure different things.

1. In the ideal clinical trial the baselines would be perfectly balanced. For such an ideal trial both approaches would deliver the same measure of effect.

2. For any given trial the two approaches will not be the same but if you looked at a very large meta-analysis of trials you would find that the overall estimate was pretty much the same and that from trial to trial, and apparently randomly, sometimes one was larger and the other smaller and vice versa.

3. That being so the difference between them is only a matter of precision. If the variances at baseline and outcome are the same, the change scores are more precise if the correlation coefficient is greater than 0.5 but less efficient if it is less than 0.5.

In practice, analysis of covariance beats them both but as I said before you are tsuck with what the researchers who carried out the study used unless you have original data.

I am most grateful for all the above comments/suggestions/references.

To clarify my question: my issue relates to identifying an indicator of variability for the respective groups that allows for valid calculation of the MD (or SMD); we have a STRONG suspicion that our included reports/publications will NOT provide an indicator of variability for the mean change scores. I also will admit to being skeptical that imputing the SDs using the correlation coefficient approach from the Cochrane handbook (thank you Piotr) will be valid as 1) there is unlikely to be a representative study from which to calcuate the coefficient coefficient; and, 2) the imputation process would have to be used for the vast majority of studies.

Previously , we have used the 2nd option I outlined in my original question (where there is baseline equivalence, use the post means and SDs) despite an intuitive discomfort with this approach. I was hoping to get a sense if the 2nd approach (post-pre change with post-SD) is predictably biased in either direction or simply nonsensical.

We use RevMan software and are satisfied that the lit search has produced studies in sufficient number and quality to legitimize a meta-analysis provided we can extract valid indiators of effect.

Don't use a hybrid. Don't follow Cochrane advice regarding imputation. In my opinion a lot of their advice regarding this is unsound. If you only have post SD then use post means. These give valid (if inefficient) treatment estimates. If the studies are randomised, then you should forget about baseline equivalence.

Stephen

The other issues between groups you can look at dichotamous outcomes, such as number who dropped out (used, with some controversy, in the CATIE studies of schizophrenia), number recovered, or number died

Our scales are often a proxy measure for what we are really interested in. A patient is more interested in being well than having a HAM-A (anxiety) score of 7. In many conditions -- hypertension, diabetes &c -- the real output we are interested in is disease (IHD, renal failure) or death.

And sometimes the proxy measures give the wrong answer. Consider the work on lignocaine preventing Q or R in acute cardiac syndrome -- it tiook a reanalysis using death as the outcome to demonstreate that people given lignocaine died more.

So I would advise you to look at ALL the outcomes, while reading the Cochrane handbook http://www.cochrane.org/training/cochrane-handbook

I would review and use the combination of Cochrane's RevMan (Review Manager) and GRADE shareware to at least do the preliminary work. The former will also let you get a sense of publication bias since you can switch to a funnel plot view. It is not impossible for the math related to the diff of diffs behind the system to be questioned, the organisation who commissioned our study did so and required a more conservative method, but this did not change the big picture results. This procedure will make sure you are on the right track and let you check heterogeneity etc

If the main measure of interest is the change in the response variable, my opinion is that we should try to meta-analyze this. In one project we recently completed, some papers only reported the pre and post mean values, along with associated SDs. While not perfect, we thought it was acceptable to use the weighted, pooled SD (from the pre-post values) as the SD of the change. Reviewers did not have any problem with this approach.

To repeat what I said before, since on average over all randomisations the mean difference between baselines is zero, you are measuring the same treatment effect whether you use raw outcomes or change-scores (difference outcome minus) baseline. The only issue is which is more precise. Analysis of covariance is more precise than either but in any case in practice you have to make do with what is reported. I would strongly advice against imputation of any form. Avoid it if you can.

In REVMAN you can do a sensitivity analysis by response measurement. I'm conservative in this and still prefer to see a weighted mean difference by the different scales rather than an effect size. I'm familiar with the scales in my feild and no roughly what five points looks like in a patient. An effect size is a little too abstract.

If you have continuous outcomes and standard errors (or standard deviations and sample numbers) on the same original scale for all studies I would advise using these measures rather that a 'sigma-divided measure' such as Glass's effect size. This is an approach of last resort when combining different scales. It has the unfortunate property that the same treatment effect can appear different simply because the inclusion criteria for a trial were stricter (say) leading to a smaller standard deviation.

See Senn, S. J. (2011). "U is for Unease: Reasons to mistrust overlap measures in clinical trials." Statistics in Biopharmaceutical Research 3(2): 302-309.

Abstract: In recent years a number of authors have promoted approaches to measuring the results of clinical trials that depend on the degree of overlap between the distribution of results from the treatment and the control group. In their enthusiasm for such measures, however, many have overlooked a particular problem, namely that while they seem to overcome a certain arbitrary element in measuring the shift between groups in a given trial, they still depend very strongly on the distribution of the results from trial to trial.

Interesting set of exchanges. In point of fact, theoretically, the effect sizes should be identical whether you use post only comparing control group to treatment or using treatment pre and post. Consider the basic estimation of d. Mean One substracting Mean Two divided by average standard error. In post only, comparing control to treatment you have, in theory, with random assignment, two groups that are equivalent at time 2, only difference is one received treatment the other did not. With pre versus post, same things except same person.

The t statistic is more powerful when using within person design than comparing between person design. In the 50 or so meta-analysies I have done and in most comparing within and between group designs, almost never found a difference in magnitude of effect.

Two further comments. First the issue in my experience of educational meta-analyses is that the distributions are different, so the (pooled) standard deviation of the gain scores is different from the (pooled) standard deviation of post-test. I take your point that they both represent the change between the experimental and control groups relative to the distribution, but when I've compared them they are not always the same, so I usually recommend going for consistency in the method used (or at least checking for these issues in a moderator analysis - though probably underpowered). Second point is about measurement error - I recommend reading Stephen Gorard's" The propagation of errors in experimental data analysis: a comparison of pre- and post-test designs": http://www.tandfonline.com/doi/abs/10.1080/1743727X.2012.741117 as food for thought.

Useful discussion though!

In my opinion, cross-sectional comparisons of longitudinal data are misleading and do not take into account any differences at baseline which might not have been significant at time zero but attract Type I error over time due to failure to account for mean and SD differences. This is particularly important in clinical trials where some may progress and some may regress. Without repeated measures, non-linear covariate analyses or other procedures clinical significance of results will be lost. While multiple imputation may be off-putting, it depends on your sample size as to what procedures for handling missing data are needed. Missing data should not be ignored.

Dido

thanks to everyone for their contribution to this dialogue; I suggest we consider this thread closed. Best wishes.

In my opinion Effect Size, SD & Power of test along with level of significance could be the best option

I agree with several comment mainly those from Stephen and Mike. One last suggestion is to look at the software you are going to use to run your meta-analysis documentation what arguments are necessary to reach the desired results. Im sure that metan in Stata and package metafor in R-project are able to handle meta-analysis of continuous outcomes.