I think the answer depends more on other factors about the study itself, and what types of studies you are attempting to pool. Can the high degree of heterogeneity be explained clinically, rather than statistically? Are the outcomes similar between studies? is it just 1-2 studies responsible for the high degree of heterogeneity? How many studies are you pooling, and how many are 'out of line'? The answer to how you handle the analysis depends on these issues as well as others.

There is a possibility that a moderator variable is missing that does in fact explain the heterogeneity (ex. study quality, time of publication, population characteristics, etc) and it is also a possibility that even if you could collect all information on any possibile moderator variable that you will still be unable to statistically explain this heterogeneity.

I wasn't sure I understood the question about 'drawing a study time on each comparison until homogeneous subgroups'. However grouping studies based on similar effect sizes, rather than known characteristics about the study isn't something to consider. It might help you look at the individual studies to learn why they are different, but it is not the subgroup analysis you should be doing. If you are referring to a sensitivity analysis of removing a single study (or a couple studies) to create a homogeneous subgroup, again you should have a very good reason to do so - and not just to create a homogeneous sample.

A possible answer may be to simply report the pooled results using a random effects model and discuss the heterogeneity as a limitation. Real world data is rarely perfect. However if the degree of heterogeneity is really bad, then it could be an indicator of non-uniform study design, various types of publication bias, and so on.  

If I understand your question properly: you have a subgroup used to represent a wider, larger group; but, the subgroup fails to possess homogeneity... If that is so, then you have failed to properly randomize the tests groups, in other words, you need to form subplots that are designed to homogenize all the confounding factors. You say, "...drawing a study time on each comparison, until homogeneous subgroups..." I read this to mean that you will "try and try again, until the study proves homogeneous." Time is a confounding factor and care is required not to improperly fold what is essentially stochastic data into a pooled statistic, e.g. that is why ANOVAs are so concerned with how one pools together variances. 

I think the answer depends more on other factors about the study itself, and what types of studies you are attempting to pool. Can the high degree of heterogeneity be explained clinically, rather than statistically? Are the outcomes similar between studies? is it just 1-2 studies responsible for the high degree of heterogeneity? How many studies are you pooling, and how many are 'out of line'? The answer to how you handle the analysis depends on these issues as well as others.

There is a possibility that a moderator variable is missing that does in fact explain the heterogeneity (ex. study quality, time of publication, population characteristics, etc) and it is also a possibility that even if you could collect all information on any possibile moderator variable that you will still be unable to statistically explain this heterogeneity.

I wasn't sure I understood the question about 'drawing a study time on each comparison until homogeneous subgroups'. However grouping studies based on similar effect sizes, rather than known characteristics about the study isn't something to consider. It might help you look at the individual studies to learn why they are different, but it is not the subgroup analysis you should be doing. If you are referring to a sensitivity analysis of removing a single study (or a couple studies) to create a homogeneous subgroup, again you should have a very good reason to do so - and not just to create a homogeneous sample.

A possible answer may be to simply report the pooled results using a random effects model and discuss the heterogeneity as a limitation. Real world data is rarely perfect. However if the degree of heterogeneity is really bad, then it could be an indicator of non-uniform study design, various types of publication bias, and so on.  

Thank you very much for your help. Indeed I think the best option, given the heterogeneity found, will accept it as a limitation, since none of the moderator variables handled, whether or not combined with others, is allowing me to explain this heterogeneity. I have taken into account include: study design, publication year, sample size, type of instrument (self-administered, heteroadministered) and study quality. Also I used a random effects model.

A greeting.

be careful! If heterogeneous, then many of the common statistical tests do not apply and are not valid!

Look for the potential confounders. May plan a meta-regression. May also take the sensitivity analysis. try to find out the study/ies which cause heterogeneity and look for the differences there or probably recheck the extracted data.   

As a clinical analysis the 5 levels of the Social Profile* can be used by your research group or the group members themselves as participant research to supplement your statistical analysis. 

Jose, look at www.Social-Profile.com. The Social Profile is available at AOTA Press and has a flashdrive in the back cover, so you can make as many copies as you like.

If you have any questions, contact me at [email protected]

Ok, thank you.

Does your measure of heterogeneity depend on assumptions of normally distributed results? This may not be appropriate for complex interventions: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0034222

Thank you very much for your suggestion. Your article seems very interesting, and is perfectly applicable in the meta-analysis that we are carrying out, as it is complex interventions.