As many (most?) of us will not have access to SAS - can you post a summary of what is specified and what results are obtained\?

Thanks.

From a non-meta-analysis perspective it is common to use the random effects approach to explicitly model the heterogeneity and not assume homogeneity. And here are the results of some simulations

https://www.researchgate.net/publication/299604336_Fixed_and_Random_effects_making_an_informed_choice

and here are some arguments pro random effects 

https://www.researchgate.net/publication/233756428_Explaining_Fixed_Effects_Random_Effects_Modeling_of_Time-Series_Cross-Sectional_and_Panel_Data

both are not applications in meta - analysis

Article Explaining Fixed Effects: Random Effects Modeling of Time-Se...

Article Fixed and Random effects models: making an informed choice

Thanks Kelvyn. As James Hodges says, it depends on if the random effects are "new style" or "old syle" in terms of interpretation (see attached). The problem is with new style random effects and the result is shocking - the random effects actually worsen the estimation and do not improve it. I think that robust error variances are much better than adding random effects and in the case of meta-analysis this is indeed true. I have added the simulation logic as a pdf as you requested but running the code above in SAS is more powerful as a proof.

Random effects models do perform better than fixed-effect alternatives in the presence of ANY heterogeneity in simulation settings. See: http://www.ncbi.nlm.nih.gov/pubmed/11252006 and http://www.ncbi.nlm.nih.gov/pubmed/21148194. However, if you have an extremely unusual set-up with a trial with thousands of patients being meta-analysed with one with 50 patients, the weighting might mess up things. I remember reading about a couple of hybrid approaches that try to use the FE weighting with a random-effects model. they seemed promising for such settings.

I don't use SAS either and I can't comment on your code however this is not a new area of research and the consensus is that it is better to use a RE model: it is far from perfect but it is less problematic than a FE approach.

Assuming all your code is correct you might have come across a particular specification in which a FE approach performs better than a RE approach. however, it would need to be well thought and complete to be publishable since this is a rather congested research space and I don't think you have come across a "solution" or "something that indicates that biostatisticians have failed the research community".

The biggest problem is the inaccuracy of the heterogeneity estimators which is well known, e.g. see http://www.ncbi.nlm.nih.gov/pubmed/23922860. However, RE models are STILL better than FE approaches despite these limitations.

Thanks Evangelos, I agree that is the position of mainstream biostatistics which you also concur with. In evidence of this position you have cited Brockwell & Gordon (Professor Ian Gordon is from a nearby university to mine and I have indeed called him up to speak with him about his paper) and of course your paper in SMMR.

There is a problem however with both these papers. The simulation protocol assumes a random effect and therefore the data generation and the method of analysis are exactly the same leading to a self-fulfilling prophecy (a prediction that directly causes itself to become true).  As Hodges (attached paper) has stated, "However, based on journal articles and communication with colleagues, many statisticians appear to believe that for simulation experiments evaluating a method involving new-style random effects, data should be simulated in the same manner, with repeated draws from the random effects. Views on this matter do not seem to correspond to any Bayes-frequentist divide; it seems to be the default position of most statisticians of both persuasions. We will argue, to the contrary, that in simulating data for experiments intended to evaluate methods involving new-style random effects, repeated draws should not be made from those new-style random effects; it may be inappropriate to use even one such draw."

I therefore believe that papers that use this approach cannot provide the evidence that can allow researchers to make any considered decision regarding the superiority of one model over another.

My SAS code avoids this error in simulation and provides the shocking result that indeed it is the FE estimator that is more robust (less error) under heterogeneity. The only issue with FE estimators has been poor error estimation and that can easily be fixed using a quasi-likelihood approach. Keep in mind that RE estimators also have sub-optimal error estimation as also reported by Brockwell and Gordon.

Finally, I agree, if there are hundreds of studies (not really just large study sizes) then variance will drop and the bias - variance trade-off can become unfavorable. Barring this, the only conclusion I can reach is that biostatistics has failed the research community in advocating for the RE estimator over the FE estimator.. 

 Borenstein and colleagues at Albert Einstein College of Med have laid out their view in an online Mongraph  (Metanalysis: fixed effects and random effects).    Does it help?

https://www.meta-analysis.com/downloads/Meta-analysis%20fixed%20effect%20vs%20random%20effects.pdf

Thanks Suhail

Not sure what you mean with the draws exactly. Speaking for our simulation, it was assuming heterogeneity and it was simulating it appropriately for the data. In that sense it is of course a self-fulfilling prophecy: RE models account for heterogeneity and hence they tend to perform better on average (unless there is a weighting issue as I explained).

Anyway, apologies if I have misunderstood. The critical issue is getting your statement through peer review, which I imagine will bring to light any issues (at least in major methodological journals it will).

Hi Adrian, nice to see you on the blog. Not really, because Dr Borenstein also assumes the mainstream biostatistics position. I have commented on Dr Borenstein's position in a paper in PPS (attached)

Hi Evangelos, what I mean is that you generate random effects in the simulated data from N[0,tau_sq] and then analyze data (RE) assuming N[0,tau_sq] in the model and therefore obviously you will find a self-fulfilling prophecy. The problem is that this is never the mechanism through which meta-analytic data comes into being - we should view random effects simply as an analytical tool.

Regarding peer review, we will try, but we also know that it will be difficult because of confirmatory bias which is defined as the tendency to emphasize and believe experiences which support one's views and to ignore or discredit those which do not. Indeed, this is exactly in keeping with Mahoney's argument that even though it is clear that these obtained results should be of interest to the scientific community, they will be viewed prejudicially because they do not conform to current theoretical know-how. According to Mahoney, this is because there is the tendency for ..[peer reviewers].. to seek out, attend to, and sometimes embellish experiences which support or "confirm" their beliefs. As Mahoney puts it, "It is only contrary-to-prediction experiments which carry logical implications" and the mandate for us all must exclude this "dogmatically confirmatory tradition".

Actually we tried various distributions for heterogeneity and results agreed. Also in the plos one paper I cited above we found that the distributions we generated for heterogeneity closely matched what we observed in the Cochrane database with thousands of "true" meta-analyses (http://www.ncbi.nlm.nih.gov/pubmed/23922860).

Agreed about the short-comings of peer review which are well known. Until a better alternative emerges (and I can't think of one) I can't see how you can validate your findings and for the research community to accept them. Until this point for example, I don't see what you think is wrong with these simulations.

Getting back to your original question and reply

Gary King has argued that using robust SE's is like taking a canary down the mine; if the robust SE's give different results , there is something wrong with the model, it is not OK to interpret it. For what it is worth I agree with him and thus I think it is better to have an explicit model that does take the heterogeneity into account and not just treat as a nuisance to be corrected - random effects modelling can do this.

http://gking.harvard.edu/publications/how-robust-standard-errors-expose-methodological-problems-they-do-not-fix

I am of course again writing about random versus fixed in general and not specifically about meta analysis. 

Evangelos, our findings are validated by peer reviewers already (attached) but that does not mean that biostatistics will agree. It will be quite embarrassing for Cochrane and the SMG (and quite expensive too if they have to rerun all their meta-analyses).

To move the agenda forward will require each researcher to decide for themselves if they are being taken for a ride by biostatistics. That is where the SAS code comes in - but only for SAS users. I will add R code once that is ready.

We have such examples previously of well accepted methods being questioned - Charles Stein shocked the statistical world in 1955 with his proof that maximum likelihood estimation methods for Gaussian models, in common use for more than a century, were inadmissible beyond simple one- or two-dimensional situations. The uptake however was slow and took 20 yrs

Kelvyn, I am not too sure about multilevel models where random slopes are being contemplated. But with random intercepts I still cannot understand the benefit - we add random effects because it helps improve error estimation (same as with meta-analysis) but this can be improved anyway using robust error variances. So why allow the random effects to alter the coefficients based on a completely fictitious distribution (of effects) that does not really exist?

The meta-analytic model is also a hierarchical or mixed effects model and if random effects fail there, it should fail in standard ML models too. Indeed, some biostatisticians have labelled the random effects variance as a nuisance variable (attached) but still go on to interpret the results of such an analysis in the meta-analytic context. 

After a quick look I don't see a measure of accuracy for the methods. Your method with the good coverage will have much wider confidence intervals and will have low power. So win something and lose something. Then there's bias.

Hi Evangelos, I am not sure I see what you mean. Power in the context of a meta-analysis depends on the within study sample size, number of studies and magnitude of the effect and the CI width only adds to the above if coverage exceeds the nominal level grossly. Estimators on the other hand are completely defined by robustness and reliability. 

Robustness is in relation to error (mean squared error) and this is a composite of variance and bias squared. The FE estimator has more relative efficiency as it has a lower mean squared error than the RE estimator despite a greater bias. If bias were the only consideration we would opt for the arithmetic mean in meta-analysis as that is clearly unbiased - but we dont because we lose in efficiency. The RE estimator fails on this criterion relative to FE.

Reliability refers to correct error estimation measured as coverage of the confidence interval. The FE that I have proposed has nominal coverage (95%) and again the RE fails as it can have coverage as low as 70-80%. Thus RE underestimates the statistical error and produces overconfident results.This is such a serious problem as to call the majority of "positive" Cochrane reviews into question.

I think this comparison can not be stated in terms of trade-off between FE and RE as the RE is clearly inferior on all counts.

Kelvyn, to add to my previous comments, under RE say we have  subjects within units that are not independent, they can be modeled explicitly as "REs". But, there are some precise distributional assumption that are made about the correlation of subjects within each unit (e.g., the within-unit correlation takes the same form for all units, that each subject within a unit is correlated equally with any other subjects in the unit). With cluster robust this is not modeled explicitly and instead allows for arbitrary correlation within units, and the form of this correlation can vary from unit to unit. RE should only be able to give more efficient estimates if the modeling of the correlation caused by clustering is correct. Since the random effects assumptions are usually fictitious, the coeffs and SEs are in many cases wrong.

Would we then not expect cluster robust to give consistent estimates across a broad range of possible forms of the correlation? They, of course, won't be as efficient as when the exact form is known – but we never know the exact form so how can we then rely on RE?

Suhail, I disagree I'm afraid. In my view the evaluation does not take into account important aspects of performance like I said. Power in this context is as useful and as important as coverage: e.g. the power to detect an effect X when one exists and decide that it is  significantly different to zero. I suspect the method you propose has very wide CIs and fails in that aspect (Type 2 error) . And the CI width is another important parameter of accuracy in this context. See one of the papers I mentioned before for more details on how to calculate these measures.

I am aware how reliability and mean squared error work.

No problem Evangelos, we will have to agree to disagree but I think I have made the particular point that change is not going to be easy

Just to illustrate the huge problem with faulty conclusions based on the failure by biostatistics, I have attached an illustrative BMJ paper which reaches the completely wrong conclusion that "Paracervical local anaesthetic injection is the best method of pain control for women undergoing hysteroscopy as outpatients." A re-analysis shows that this conclusion was reached because the biostatistics community mandated the use of RE models with consequent faulty error estimation (see attached figure). In reality, paracervical local anaesthetic pain control was useless (see the link with rapid responses from patients).

http://www.bmj.com/content/340/bmj.c1130/rapid-responses

Suhail,

Can you please post the simple steps to start meta-analysis in MetaXL. I read the user guide. It gives good overview but does not list the basic  steps on how to start the analysis.

Comprehensive meta-analysis has done a very good job on this. Perhaps you could post few youtube videos on the basics running MetaXL.

Thanks !

Hi Praveen,

Have a look at the example files on the MetaXL menu - they are as good as youtube videos.

Suhail