When you ask: Are the results comparable? I wonder: What do you want to compare?

Before seeing your answer, I would say that appropriately designed randomized trials have no bias and any error in the results is random. That means that meta-analyzing results from different trials that are designed to answer the same question, leads to a smaller error in the results than from the individual trials. (When meta-analyzing trials that do not answer the same question, the story becomes a bit different.)

In observational studies we cannot exclude bias and the more comparable the studies are that you want to meta-analyze, the more likely they have the same bias, which will propagate fully into the meta-analysis results. In other words: meta-analysis observational does not reduce error as well as that of randomized trials. If the observational studies are so small that it seems plausible that random error is larger than any possible bias, then meta-analysis may still lead to a substantial reduction of error.

According to Robert, I would add that the meta-analysis of observational studies and clinical trials are complementary and should not be compared. While clinical trials reduce research bias in most cases they report the effectiveness of the intervention (effective only when they are pragmatic trials). While observational studies as follow large populations and real condicioes often give us an idea of effectiveness of interventions

I think that from a statistical viewpoint, as model, there is not difference. The same technique can be used in the context of clinical trials or of observational studies, but the difference is in the ethics to be applied in each case.

I allways recommend to imagine a planned meta-analysis as it would have been a prospectively planned multi-centre trial/study. That might give you a better idea of how to interpret heterogeneity. With respect to RCTs, severel trials ususally don't ask the exact same question, that is the sort of methodological heterogeneity you face when dealing with observational MAs.

I have attached a paper that addresses this. The abstract states:

Some authors argue that systematic reviews and meta-analyses of intervention studies should include only

randomized controlled trials because the randomized controlled trial is a more valid study design for causal inference

compared with the observational study design. However, a review of the principal elements underlying this

claim (randomization removes the chance of confounding, and the double-blind process minimizes biases caused

by the placebo effect) suggests that both classes of study designs have strengths and weaknesses, and including

information from observational studies may improve the inference based on only randomized controlled trials.

Furthermore, a review of empirical studies suggests that meta-analyses based on observational studies generally

produce estimates of effect similar to those from meta-analyses based on randomized controlled trials. The authors

found that the advantages of including both observational studies and randomized studies in a meta-analysis could

outweigh the disadvantages in many situations and that observational studies should not be excluded a priori.

You should include both, just because something does not have a RCT behind it doesn't mean that it's not valuable in finding the true effect. (http://www.bmj.com/content/327/7429/1459) 

Your goal is to try and find all of the available studies you can.

Most of the time epidemiological studies go after real life data. Thus, these analysis reflect real life experience more than clinical data. RCT's are designed to select ideal patients with a lot of exclusion criteria and does not exactly represent what will happen in daily practice. That is mainly one of the reasons in using ITT population for analysis. I think the wiser way is combining and taking into account both epidemiological and clinical metanalysis: as mentioned both have weakness'. 

very nice point made by Rohit and I agree with comments of Hamdi. Trials and observational have both drawbacks and both convey important informations. They are basically complementary and for having participated to both, my feeling is that the difference between both is very thin. Observational studies are too often exposed to critics compared to trials and this is erroneous.

what randomised studies or experiments give you that observational studies don't is an ability to claim causality, because you are controlling the condition you are testing.  For example, if you want to compare drug A to drug B and you randomize then, if the trial is conducted well, you should be able to claim that the effects are due to the drug.  If you have an observational study and the assignment of the drug to patients was out of your control then you have difficulty ruling out alternate explanations for the effect.  A common problem in EMR studies is that the drug that is perceived to be more effective is prescribed by doctors to patients who are judged to be more severely affected and thus the better  drug may actually have worse outcomes.  OF course you can try to adjust for covariates but the relevant factors may not have been recorded.  

The shortcoming of RCTs is that patients are carefully screened to be more homogeneous than the general population and special populations such as pediatric, geriatric and patients with comorbidities may be excluded to create a more controlled experimental condition.  Furthermore, RCTs tend to more expensive so the sample sizes tend to be smaller.  Rare outcomes such a severe adverse events or cross-reactions will be harder to detect.  

 In the end the quality of the study depends on how well it is conducted and that can vary dramatically for both categories.  One of the problems with meta-analyses is that often quality of the studies is not given enough weight.  I once replicated a major meta-analysis and found that the strong effect totally disappeared when you accounted for the quality of study variables.  

I agree with Hartel that a more controlled condition in rtc allows a stronger causality element in the conclusion however it is not always the case that better drugs given to amore severely affected group would result in bias outcomes as normally baseline values and factors are taken into account. 

The point is that in observational studies you can never be sure whether there were differences between subjects that you were unaware of  or that were not measured.  You can take into account baseline measures and other covariates but that does not mean that the treating physician did not have better insight into the patient's condition that you can glean from studying what data was recorded.  As statisticians and epidemiologists we must always be aware of the limitations of our methods.  No matter how much you adjust for in an observational study there always remains the reasonable doubt that the true causal factor was perhaps not captured in the data.

I would agree with almost everything that has been said above. I understand the question in two stages: a) validity and causality epidemiologically considered different between both categories of design (higher in favor of RCTs) on the hypothetical basis that both designs are well conducted. 2) A second stage is the meta-analysis issue, what entails another different, challenging matter.

In both instances, the central point is the proper selection of articles provided the meta-analysis hypothesis and design are sound and well defined.

I would also mention that in practice the probability of flaws -when selecting the articles for the meta-analysis- increases from observational studies; these articles may be misleading as they are often published in a kind of “tricky” way that may hamper reliable meta-analysis results. An example of this are those published as a “cohort design” that actually are case-series splitted in two groups with no previous explicit hypothesis and hence, with no reliable design-related methods of analysis. This makes me to agree with Bjoern Holzhauer’s  “I have also sometimes wondered whether publication bias and lack of pre-specification of objectives & analysis methods is nowadays a more severe issue with observational studies than RCTs”.

Finally, the meta-analysis main objective (“the truth you are looking for”, as says Martin Cañon) defines which the most suitable design type is preferable.  In my experience new vaccines assessments are better achieved with randomized, blind controlled trials.

Good points above. I'm often frustrated by RCTs, as they rarely have adequate sample size to make the causality claims that they are attempting to make. As pointed out, quality is really important here. I'll take a 20,000 patient propensity score matched observational study every single day over an RCT with under 200 participants so long as the population is akin to that of my decision problem.

The other problem with RCTs is they almost never have enough follow-up for real decision making on long-term conditions. The ideal patients often found in RCTs are another significant problem, average patients are needed to make decisions based on averages. As a health economist building a model, this means that we need to look at observational data anyway.

There is also a bit of a cult of the RCT. Many systematic reviewers will set a filter that excludes studies that aren't RCTs. In a recent paediatric systematic review this led to omission of all the high quality registry data and decisions based on studies that became small observational studies after 8 weeks as randomisation was broken by complete crossover. We should use the best data available to represent a decision problem, and there are many times that RCTs are not the best studies.

While the methods of incorporating observational studies into meta-analysis are still in development, I think it is important that observational studies be included (with adjustments based on Bayesian prior belief of quality), because it is important to have as complete data as possible for good decisions for patients and health systems.

R.C.Ts are less biased if they are perfectly   Randomized an blinded so the most reliable effect sizes are gained from them.

 In reality we rarely face situations without biases and confounders. Thats why the effect sizes taken from observational studies are more generalizable.

In a meta analysis, sometimes we merge the effect sizes from different designs like cohorts and RCTs, so I think we can compare meta analysises of diffrent designes on one condition : they are less prone to random error, bias and counfounding error.

I have stumbled into the same questions time ago. I think the key point here is "treatment effect". Taking a plain frequentist view, one may always have the doubt that a clinical trial designed for one single endpoint may have no sufficient power to pick up effects on other (competing) endpoints. I think this may be a licit doubt. There is some literature on sample size computation in presence of competing risks, but I guess most tirals are designed in a classic fashion (one, ore a composite endpoint). To come to observational studies, I think they are the best source for competing risks evidence. They certainly provide an "history" of all events. Due to smaller expenses (especially retrospective studies), they also may provide longer observational time windows. Bigger time windows are particularly suited to perform event prediction, so that a clearer picture of all transition probabilities may be available....  Bias arguments shall be the usual ones. Bias effects in very big observational studies may be not so important ....

Dear Sofia.

Your questions are very relevant and leads us to some considerations.

Important to consider that to enable the possibility of using observational studies and randomized controlled trials in a meta-analysis is fundamentally necessary, identify the purpose of it, because you can identify many studies conducted in situations that can be very different and therefore impossible to they are grouped in order to perform a meta-analysis by caracteríticas intrinsic to this type of study.

Thus it is important to identify the fundamental objective of the meta-analysis to enable the inclusion of observational studies and randomized controlled trials, this is the main principle to enable the use of the above designs.

Kind regards.

Apologies for entering this conversation late on. Going back to the original question relating to meta-analysis of observational studies vs meta-analysis of RCTs - I do a lot of meta-analyses of observational studies, but I would rather have a decent meta-analysis of RCTs than a meta-analysis of observational cohort studies.

We have to remember that the meta-analysis is only as good as the studies included - and is prone to the same biases as the studies included. RCTs are less prone to bias - if the randomisation works, then it protects against confounding by things you know and can measure, as well as things you don't know or can't measure. Similarly, I'd rather have a meta-analysis of cohorts than a meta-analysis of case-control studies, becuase the case-control studies are more prone to selection and recall biases.

I'm not fussed that RCTs are often smaller than observational studies, because the power of the individual studies doesn't matter for the meta-analysis - you pool the evidence to get a more precise answer. But you do want it to be as good an answer as you can get. I'd rather be roughly right than precisely wrong. So give me the RCTs any day.

The reason I turn to meta-analyses of observational studies is only when it's very hard to do a good RCT, and so there are very few around. There are very few decent RCTs of dietary interventions to prevent cancer, for example. Partly because of the long follow-up required, and partly because of the complexity of designing a good intervention that people adhere to and that has a chance of working. So I do lots of meta-analyses of cohort studies investigating diet and cancer.

Other problems with meta-analyses of observational studies are that they are more prone to large heterogeneity. There are so many study design characteristics, different adjustments for confounding, and real-life populations differ from each other much more than the falsely homogeneous samples you get in some RCTs. So often there simply isn't one right answer from a meta-analysis of cohorts, and the focus turns to exploring that heterogeneity.

The stats get much more complicated too when doing meta-analysis of observational studies, because it's rarely just drug A vs drug B, but different levels of exposure, where those levels are different across different studies.

Some of the problems with meta-analysis of observational studies can be addressed if you have individual participant data, but in harmonizing the data, you often go for the lowest common denominator, so whilst reducing heterogeneity, you can end up with poorer control of confounding.

Because of these differences, I would not consider combining RCTs and observational studies in the same meta-analysis - at least not outside of a fairly complicated Bayesian framework. Ideally, I wouldn't want to combine results from cohorts and case-control studies in the same meta-analysis either.

I've probably left out lots of relevant points, but I wanted to give my view as someone who does a lot of meta-analysis of observational studies, and would still rather have decent RCT evidence if possible.

I will go over my warning. "Observational studies" encompass a variety of studies (patient series) not always properly defined, so besides of the inherent risk of bias of a well conducted cohorts study as compared with well conducted RCTs, the risk of erroneous figures -as results- increases when dealing with unclear designs. I would restrict the discussion over "observational studies" - when dealing about comparisons with RCTs- only to those "observational" studies whose  conceptual and operational hypothesis (and the metrical requirements that entails) are explicit and sound. Otherwise, the likelihood of “post-hoc” interpretations and statements about results are really high.

Including observational studies in a systematic review without a meta-analysis presents fewer problems than including them in one with a meta-analysis.

http://aje.oxfordjournals.org/content/166/10/1203.long

https://www.researchgate.net/post/Can_I_include_Case-Series_in_a_Systematic_Review