Suppose that we test the null hypothesis H0: theta_1=theta_2=.....theta_k = 0 and consider one of two possible cases as regards homogeneity of the treatment effects.
a) they are all equal to each other
b) they are not equal to each other
It has been suggested that when we test this null hypothesis and reject it we are entitled to assert that there was a non-zero treatment effect in at least one trial.
I agree that If b is true then H0 is false.
However if a is true then in that case the conventional fixed effects variance cannot be a valid estimate of the extent to which the weighted statistic will vary about the true mean because the variance induced by heterogeneity (systematic error) is not considered . Thus we have an invalid test of H0.
Thus either b) is true and there can be no type I error or a) is true and we will then be faced with an invalid test (when there is heterogeneity which happens in most cases)
When we test this null hypothesis and reject it we are therefore NOT entitled to assert that there was a non-zero treatment effect in at least one trial UNLESS heterogeneity was part of the variance estimate (e.g. IVhet model). The suggestion by biostatistics therefore that when we test this null hypothesis and reject it we are entitled to assert that there was a non-zero treatment effect in at least one trial IS false in the presence of heterogeneity because of faulty error estimation.
Any comments on this would be welcome
Hi there!
When performing a meta-analysis, this is an important thing to keep in mind. Check this paper:
https://www.researchgate.net/profile/Frank_Schmidt10/publication/229496828_Fixed_Effects_vs_Random_Effects_Meta-Analysis_Models_Implications_for_Cumulative_Research_Knowledge/links/541aff0f0cf2218008bffea7/Fixed-Effects-vs-Random-Effects-Meta-Analysis-Models-Implications-for-Cumulative-Research-Knowledge.pdf
My experience with meta-analyses, is that fixed effects are a particular case of random effects. Random effects allows you to generalize your results by taking into account other (additional) sources of variation different from fixed effects. Only works well when your sample of primary studies is big enough (this is pretty similar to multilevel modeling in statistics). If your sample is to small, confidence intervals for the combined estimation of the effect size will be wider (more error).
Also check for other meta-analyses that have been published in high JCR scientific journals and read about if random effects are considered or not, and how the authors explain their decission.
Hope this helps :)
Best regards,
Merche Ovejero
Article Fixed Effects vs. Random Effects Meta‐Analysis Models: Impli...
Do you take into account that H0 is not the same as biological significance? For instance if two groups are compared mean1 = 104, mean2 = 104, both have sd = 10 and n =10000, you will land with highly significant test and theoretically should reject the H0 hypothesis. But in biology very rarely this difference in means is of any value.
I would be grateful if responses could please keep on topic - many thanks
Thanks Ana. Many biostatisticians I have spoken to simply do not budge from the position that even with heterogeneous studies the FE model tests the null hypothesis that the treatments are identical. I am glad that you agree with the reasoning I propose as its not just theoretical - many wrong conclusions have been reached in medical decision making just due to this faulty position being held in the mainstream literature.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics