I guess you are referring to decision making in terms of choice of model when undertaking a meta-analysis.. My question is why is there a choice in the first place when, statistically, random effects estimators are inferior to fixed effect estimators regardless of heterogeneity?

Well you've pointed out an interesting question. There is a tendency to conduct random effect meta-analysis when important statistical heterogeneity is suspected but i believe both models answer different review questions with regard to treatment effect assuming one would like to conclude about treatment effect. And i agree with Elmer that the choice of random effect model or a fixed effect model should not based solely on statistical heterogeneity but rather on the question being investigated first. I also believe choosing one or the other has implications for small studies effects on the pooled estimate. But one could also implement both models , discuss their results and make appropriate conclusion rather than choosing one.

My point is that maybe they do not address different review questions at all. I agree with you and Elmer that this is the current view, in part confirmed by the Cochrane, but what is the evidence for this view? Both fixed effect and random effects methods use exactly the same inputs (study effect and SE) so if their outputs differ it only should be in terms of variance gains and nothing more. Given that the random effects estimator actually ignores variance of the studies as heterogeneity increases, what is the basis for its use in the first place? For example, Stephen Senn has stated that "if it is decided that a wider purpose should be met, that of saying something about trials in general, it may become sensible to perform a random effects meta-analysis but not until something is understood about the differences between the various trials in the meta-analysis in terms of their purpose. There seems to be generally a great deal of confusion about what meta-analysis can deliver here" - Stat Med .26:1417-1430, 2007. I am not exactly sure what this means but my reasoning is that if both have the same inputs, can we really expect statistical manipulation to enable generation of pooled estimates with different interpretations - I would not think so unless someone can suggest how this could be achieved. Until then, I think the only purpose of different methods of weighting is variance reduction and in this respect the random effects estimator seems to fail when compared to the fixed effect estimator.

That's the point of my post - these are just hypothetical constructs and theories. They are probably not true in practice. If there is no underlying true effect that we are attempting to estimate through meta-analysis, we should not do a meta-analysis. if there is some variability in true effects but they are not "dissimilar", then we are still trying to estimate a single underlying true effect that serves to represent these "similar" but varying effects. The other point you have raised is quite true, the random effects method penalizes large studies if there is heterogeneity and the weights shift from large to small in an attempt to equalize the weights as heterogeneity increases - this makes no sense unless we believe that big studies are always faulty. That brings us back to my question - why should we do a random effects meta-analysis when (in almost all cases) the fixed effect estimator has a lower variance and MSE - will we then not be better off doing a fixed effect meta-analysis in almost all cases?

It's inevitably that there is variability in trials (because who would want to replicate the exactly same clinical trial if there is already one). The fact that most meta-analyses are done using random-effects is simply because the random effect model is an easy way-out to pool data together for most reviewers. I agree that we should always do both fixed effect and random effect meta-analysis. More importantly, we should put more focus on exploring the sources of heterogeneity than on 'choosing' which model to use. It's meaningless to present a bunch of numbers without interpret them in the clinical context.

Agreed, there is variability in trials due to random or systematic errors or even heterogeneity of populations and protocols but which remain "similar" enough to meta-analyze. But my argument is that the random effects estimator gives no additional information beyond that achieved with the fixed effect estimator and may even mislead, so why should we use both? In my opinion, if the results from both disagree we simply have to discard the random effects result, so what is the gain here?. Again it comes back to the same question I asked  - is it because of the perceived difficulty in computing the appropriate model based variance for the fixed effect estimator (ie if we use it for heterogeneous studies we get a spuriously narrow confidence interval) that the random effects estimator dominates the literature? Indeed, that is most likely the reason why fixed effects are shunned for heterogeneous studies - the computed variance underestimates the statistical error and is overconfident in its conclusions. So why are statisticians not addressing this directly - why instead suggest another estimator of greater variance that also continues to underestimate the statistical error?

Yes, you have understood me quite well and yes, that is exactly what I was implying. Yes, I can provide the evidence as well. Regarding the random effects estimator failing to correct the model based variance underestimation of fixed effect estimators, here is the evidence:

Noma H. Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections. Stat Med. 2011 Dec 10;30(28):3304-12

Brockwell SE, Gordon IR. A simple method for inference on an overall effect in meta-analysis. Statistics in Medicine 2007; 26:4531–4543

Regarding the evidence that the random effects estimator has a larger variance and MSE than fixed effect estimators, here is the evidence:

The random effects estimator weights simply equalize (migrates towards the arithmetic mean) as heterogeneity increases. A basic rule of statistics is that  inverse variance weights minimize variance and therefore it follows that with increasing heterogeneity, fixed effect estimators will have a lower variance and possibly a lower MSE if bias is relatively small in relation to the variance reduction. Thus, it clearly means that the random effects estimator weights have been messed with in a way that certainly makes estimator performance worse.

Hi Jelle, thanks for this detailed response. I am afraid you are not right on several fronts and I will attempt to describe these.

First of all, inference in terms of “the set of studies” or “future studies of the same type” or “a random effects assumes….” are theoretical concepts and really not important until we understand the statistics. Also, the fact that we are trying to estimate the mean of a varying effect versus a common underlying effect are again theoretical constructs and whether we think that we are trying to estimate a mean effect or a common effect is not as important to this discussion until you are clear about the statistics so for now I will only respond to the the statistics only. I may come back to these issues once we are on the same page regarding statistics. or in my future posts. 

The fact that we often get a lower p-value when doing a fixed- compared to random effect meta-analysis has nothing to do with representativeness – it is a feature of overdispersion. In other words, the fixed effect model based variance underestimates the true variance much more than the random effects model variance does - in other words it has to do with the gap between true and model derived variances in each case. .

Regarding true effects, keep in mind that the study has an expected true effect before random error and I am alluding to this NOT the underlying (or grand) effect – this distinction is very important so when you talk about "no variability in true effect sizes" you are not referring to what I am referring to.

I agree that weighing is done in two steps: Your step (1) that 1/se^2 is applied is correct. Your step (2) is incorrect. The weight factor is NOT scaled to the variance in effect sizes between studies. The common between studies variance is added to the se^2 of each study. Thus as this added value gets bigger, the weights become more and more equal (approaching the arithmetic mean). Thus, a large study with small s.e. does NOT retain a high impact in the random effects model at all, it is penalized because it is big. If you seriously believe that bigger studies less accurately estimate the true effect size, then only can this make sense.

"That brings us back to my question - why should we do a random effects meta-analysis when (in almost all cases) the fixed effect estimator has a lower variance and MSE - will we then not be better off doing a fixed effect meta-analysis in almost all cases?"

Here you missed my point completely. This statement above has nothing to do with a P value. There are two variances that we talk about in meta-analysis a) the true (simulation observed) variance of the estimator b) the model derived variance. When I talk about lower variance and MSE I am referring to a) (true variance) NOT to b) (model derived) and that is where you have gone off the track.

-------------------------------------

next section

-------------------------------------

"Yes, I can provide the evidence as well. Regarding the random effects estimator failing to correct the model based variance underestimation of fixed effect estimators, here is the evidence:

Noma H. Confidence intervals for a random-effects meta-analysis based on Bartlett-type corrections. Stat Med. 2011 Dec 10;30(28):3304-12

Brockwell SE, Gordon IR. A simple method for inference on an overall effect in meta-analysis. Statistics in Medicine 2007; 26:4531–4543"

Again you have missed the point completely. These studies demonstrate that the true estimator variance under random effects is underestimated by the computed model based variance. These studies are certainly not about disproving random effects meta-analysis. They are trying to correct the faulty model based variance that underestimates the true variance, but they have not succeeded. They do not build on any principal (as there is none to build on) – they simply suggest ways to make the faulty model based variance work better.

"Regarding the evidence that the random effects estimator has a larger variance and MSE than fixed effect estimators, here is the evidence:"

Again, you have completely missed the point. I am not referring to model variance here but true variance. Of course the model based variance under random effects is also larger than under fixed effects but that was why the model was created in the first place. The problem is that this larger model variance does not serve its purpose, because the true variance is also larger and yet the model variance is unable to keep pace with the true variance. Of course, I agree that the overall estimate becomes less accurate when there is more heterogeneity between studies, but my comparison is between random and fixed effect estimators and the degree to which the random effects estimator deteriorates is greater than that with fixed effect estimators at the same level of heterogeneity. Thus, all the theory about inferences with random effects estimators is nothing more than a statistical swindle.

I also meant that this discussion be constructive and I do hope it has clarified some of my points that were misunderstood in your response.

Hi Jelle, again a few points you have not quite understood.

First, the true variance of an estimator IS its observed variance under simulation. The model derived variance is the variance we compute through the model and use to create its confidence interval. If the model derived variance is too small (both fixed and random effects estimators but more so with fixed) then the confidence interval will be too small and leads to spurious significance. This is what is meant by “underestimates the true variance due to overdispersed data”. The assumption is always that the model is correct but data are behaving abnormally, but the reality is that the model is in fact deficient. This has nothing to do with ignoring or not ignoring the population data under study since the inputs (ES and SE) are exactly the same to both models.

There is also no post hoc decision making in this regard. The best estimator is one with the smallest MSE (MSE can only be computed based on true/observed variance) and this is always better (less) for fixed effect estimators. The fact that model derived variance is larger for random versus fixed does not have anything to do with estimator performance. It does however impact on the extent to which a particular model result will under-estimate the statistical error.

We do not judge weighted estimators by theoretical arguments, we judge them by performance first and foremost (eg MSE). Clearly, the random effects estimator has a much poorer performance and no interpretation can fix that. I agree, we usually specify models based on several assumptions (you note that I have removed the word “correct”) but if they perform poorly, those assumptions are wrong – not the other way around.

Performance measures (MSE, true variance, bias) have nothing to do with P values or with overdispersion. The P value however is indeed impacted directly by overdispersion (by which we mean that the model computed (or theoretical) variance underestimates the true or observed variance). This will lead to poor coverage of the confidence interval (well below the nominal level for random effects estimators too).

I am quite happy to discuss assumptions and inferences, but not until we are on the same page regarding these statistical concepts.

Hi Jelle, lets try to sort this out by first understanding what these concepts mean in real terms. If we first make the assumption that MSE of the estimators are equal, let us see what only underestimation of model variance does in meta-analysis of heterogeneous studies. Look at the attached figure (example1) and let us assume the true effect is the "null" effect. Let us assume that all estimators produce the same point estimate by chance because they all have the same MSE  (and this point estimate is not null). Because the model variance is too small with random and fixed effect estimators, there is spurious significance with both. Had we known the correct way to compute the variance for the fixed effect model (third diamond) this would have been what would have been the result (no significance).

Now lets take example2. Here we assume that MSE is bigger for the random effects estimator (which it is). What this means is that it has a higher probability of a more extreme effect compared to fixed effect results and we get the results shown in example2.

That now brings us back to my original question: Why are we not using the estimator depicted by my third diamond?  The answer seems to be that statisticians have no idea how to compute the correct estimator variance.

Now you tell me, how a "better fit" is forced by model specification? There are no assumptions in simulation - these assumptions are only used to derive a larger model variance for random effects (example1). In so doing they destroy the weight structure and create an estimator with larger MSE (example2). The better fit (MSE) for fixed effect is certainly due to better model specification but does that not imply that random effects has faulty specification and therefore is based on wrong assumptions? You will ask at this point what could the wrong assumption be? Well, in meta-analysis, we do not randomly sample studies from a super-population of studies and therefore the random effects estimators violates the first principle of randomization in statistical inference.

To fix this situation and avoid the large numbers of spuriously significant meta-analyses, we need to return to fixed effect estimators and figure out how to rectify the model variance. Cochrane refuses to address this possibly because they will need to retract thousands of meta-analyses which may cost them their reputation. They thus prefer to use their clout (and the ignorance of the public) to continue to perpetrate a statistical swindle. Of course, I may be wrong, this is not a swindle - maybe they truely do not know and believe all the mumbo jumbo about assumptions in the model and the completely false assumptions regarding exchangeability on which ridiculous concepts such as "prediction intervals" are based.

In summary, the solution to this problem lies in returning to fixed effect estimators with correct model variance and the random effects estimator is like the "Kings new clothes"

Hi Jelle, wow, I am impressed, you actually did the simulations! Yes, you are right, the fixed effect is better because heterogeneity is never normally distributed in most cases and meta-analysis (by definition) means a biased selection of studies and there is no such thing as "normally distributed random effects". Even if that happens by chance the fixed effect estimator will still outperform the RE estimator when there is heterogeneity except as you have mentioned - the overdispersion. I also did extensive simulations and I found that the fixed effect estimator, as you did, was indeed better. Yes, there is the overdispersion issue, but I have a fix for that using the quasi-likelihood approach - attached (which Cochrane never bothered to do).  So please re-run your simulation and use the attached variance and not 1/sum(w) for the fixed effect estimator - now you have no overdispersion at all with the fixed effects estimator and indeed the coverage probability keeps exactly to 95%. Now you will be left with overdispersion only with the random effects estimator (coverage can drop well below the nominal level). Question: Is it now time that we dropped this concept of random effect estimators in meta-analysis since FE is better and there is no overdispersion through my fix?

Hi Jelle, that was not a weighting factor that I attached, that was the expression to compute the variance of the fixed effect estimator. The weights are the same - inverse variance.

The overdispersion occurs because the original variance expression in the fixed effect formulation cannot keep pace with heterogeneity and this fixes it completely and therefore the only rationale of RE models (some incomplete degree of correction for overdispersion) is gone since there is completely no overdispersion for fixed effect estimators with my fix.

Regarding outliers, true, a large study will move the fixed effect estimator in that direction and if the large study is faulty we get a point estimate that is deviated.The opposite applies to RE estimators - if the large study is really good, the RE estimator will penalize it and move the point estimate away from it and will be equally deviated. Both scenarios are equally bad and therefore there is no difference in this respect between estimators. However, the MSE of fixed effect estimators is smaller and this will make them less deviated under the same circumstances of outliers than RE estimators under usual conditions. Even if there is an outlier, with the correct variance expression (above) you will get the correct degree of uncertainty around the point estimate and thus it would not matter but it would matter with the RE estimator because it underestimates the true variance. By the way, one outlier can NOT cause the effect to become significant (if other effects are zero) if you get rid of overdispersion (with my fix) so it can only cause the point estimate to deviate. It can however cause significance with the RE model because the overdispersion remains.

The final point you raised is, can the error distribution ever be normal? well think about it this way - to have normality, we need to randomly sample our studies from a superpopulation of studies. If our studies (in a meta-analysis) are the ONLY studies that exist ( as we must find all) then there is absence of the randomization required for any statistical inference is there not?

.I would be happy to hear what you find when you simulate the fixed effect estimator replacing its variance with that I have suggested above - and thank you for the opportunity to have an open and very interesting discussion!

Let's answer the question and consider some facts.

"why do statisticians (and Cochrane) continue to promote and use the random effects estimator when there are heterogenous studies?"

The simple answer is: because they believe they get a better estimate of what they really want that way.  Do they?  Maybe, maybe not.  (They may also believe they get more appropriate confidence intervals that way, but I shall not comment on that).

Consider the following facts (if any of my facts are wrong, I hope someone will correct me).

1) In each case, we obtain a weighted average of the sample estimates.  The weights are estimated from the samples.  None of the estimators are unbiased (though they may be asymptotically unbiased) and, if proper model assumptions are made, all the estimators are consistent

2).  What makes an estimate better?  Accuracy (small bias) and precision (small variance).

3) You talk about the "variance" of the estimator, as though you know it.  You do not know it, all you know is the estimated variance.  So, much depends on how the variances are estimated.  How best to estimate the variances depends on the underlying distribution (for example, the best estimate of the variance of a Poisson distribution is given by the sample mean) which, unless you have a census, is again unknown.  Also, as is well known, how good a variance estimate is depends enormously on sample size.  Typically, it is assumed that the variance among the samples cannot be negative – should that estimate be zero (or forced to be zero when the data suggest otherwise), the two estimators will be identical.

 I hope the above gives you some idea of all the factors involved when choosing weights in a meta-analysis,

Hi Robert, true, they believe they get a “better” estimate of what they really want (a point estimate closest to the target – whatever that may be, underlying true or mean effect). My argument is that they do not.

1) I agree, in each case, we obtain a weighted average of the sample estimates. The weights are estimated from the samples. All weighted estimators are indeed biased except the “naturally” weighted estimator (arithmetic mean). This bias is because the effects and weights are correlated. Why do we accept biased estimators in lieu of the unbiased arithmetic mean? Because the bias comes with a variance reduction that is greater than bias squared and thus still results in a smaller MSE than the arithmetic mean (unless number of studies is huge which will reduce variance of all estimators without the need for weighting).

2). Indeed, what makes an estimator better is its MSE which is the sum of bias squared and variance, thus small values of both are desired. Because weighting actually increases bias, only small values of variance is the goal of the weighted estimator.

3) We know the variance of the estimator through simulation, and if this is well done it is accurate. We must distinguish theoretical variance (that calculated through the model for the confidence interval calculations) and the true variance through simulation. The latter is what I am talking about. It is only the theoretical variance that depends on the underlying distribution (for example, the best estimate of the variance of a Poisson distribution is given by the sample mean). Also, sample size does not matter if estimators are compared on the same data-set generated through simulation.

Thanks for your input and hope this helps

Hi Francsois, with random effects we do assume a distribution of study level effect sizes that vary randomly around a true effect size of the population from which these effect sizes are supposedly sampled. The assumption is wrong because these effects are the only effects that exist and thus it is unreasonable to assume that the effect sizes have some sort of distribution, or even worse to say that this distribution might be normal.

Using the fixed effects model when there is large heterogeneity is more sensible than using random effects models although there may be other options as you are aware. All meta-analyses are based on the concept of one underlying “true” effect, whether that is a fixed true effect or a mean true effect and the distinction is not important because if the variations across “true effects” (in each study not due to random or systematic errors) is huge we should move from meta-analysis to some form of standardization.

Since, unfortunately, it is true that there are serious problems with the random effects model for meta-analysis (and it does not even address overdispersion properly), my suggestion is that don’t ever use it (or don’t use it in isolation as you may not get published if you don’t provide these faulty results in your manuscript. So do two analyses – a correct one for your conscience and the faulty random effects one to get published because the majority of reviewers out there have no idea about these concepts and will simply judge you based on “recommended guidelines”. You will ask what do we use for our conscience – use the fixed effect estimator with a correction for overdispersion. That correction is given in the posts above.

Sudhail, regarding 3), if by simulation you mean using a large number of bootstrap samples, then I agree.  Otherwise, you have to make assumptions that might not be true when you do your simulations.  

Robert, yes, I used a very large number of bootstrap samples and have attached the simulation protocol I used below. Although the effect size I used is binary (odds ratio) this can be modified for continuous outcomes as well.

Very good question - indeed they are both fixed effect estimators, and as you have correctly inferred from the discussion there can be no "random effects" when we talk about meta-analysis since the effects seen are never random samples from a population. The QE estimator adds bias variance to the study variance thus effectively reducing the inverse variance weight when bias is present. The IVhet estimator uses the same QE estimator but sets bias variance to zero (thus effectively not needing quality input and using only inverse of study variance weights). Thus the IVhet model derived pooled effect is exactly the same as the conventional fixed effect estimator but uses a different theoretical (model derived) variance that overcomes overdispersion. The QE estimator, because it uses bias variance as an additional input, manages to decrease estimator variance and MSE even further than the standard fixed effect estimator when quality assessment is honestly done and regardless of subjectivity. There has been a lot of argument about subjectivity of quality assessment but if we translate that to bias variance the way we do, then the model will only become similar to the "random effects" model when quality input is random (eg fraudulently computer generated from a random numbers table). If there is even a 10% value to the quality assessment, this model will have better performance than the random effects model. Indeed, we can have up to 50% subjectivity in quality assessment and the model outperforms even the fixed effect estimator.

The question then arises (which I cannot answer) - why on earth are we using random effect estimators in meta-analysis today? It seems to me akin to some sort of organized scam being perpetrated against the research community!

Some robust responses, I thank you all. however, I have one more issue I need clarification on in a meta-analysis. when you are unable to explore heterogeneity further using multivariate meta-regression due to the small number of studies, what are the options? It is often said that a rule of thumb is 10 studies per explanatory variable. so say you have 20 studies to model up to ten variables, is a meta-regression acceptable?

I would say that in this situation meta-regression is not feasible. In fact I dont believe in random effects meta-regression - but that is a separate issue.

The way out would be to create homogenous subgroups for meta-analysis and explore study characteristics that differ across these subgroups. The creation of such subgroups would be done through reduction of both visual as well as statistical heterogeneity

We have presented the alternative to the random effects model discussed above finally and this is available at:

http://www.ncbi.nlm.nih.gov/pubmed/26003435

Hi Suhail, Excellent. Well done!

When you say that "If our studies (in a meta-analysis) are the ONLY studies that exist ( as we must find all) then there is absence of the randomization required for any statistical inference is there not?". Given the dynamic nature of things in general (populations etc) I tend to think the opposite. For example a population of people giving their opinion about something is swayed in the day to day dynmics of information, life in general.

Hi Ana, Thanks. The randomization applies to the random effects common variance - i.e we assume "true" effects vary randomly and therefore there is a variance. That's where the whole thing fails. Individual studies have a robust variance that can be used for inference.