Hello, I would agree that predefined inclusion criteria and literature search limits (such as date and English-language) should be defined for a systematic review. Then comes a decision as to whether meta-analysis is possible and if so, which studies should be statistically combined. How similar study populations are clinically and the statistical heterogeniety are drivers to these decisions. One thing that needs more discussion in the community is our current measurement of statistical heterogeneity. According to Michael Borenstein, current measures of heterogeneity, such as I-squared were never actually designed for that purpose. He and others propose the use of the 'prediction interval' for measuring statistical heterogeneity. See Borenstein et al (2021) Introdution to meta-analysis, second edition, Wiley, Chichester.

It is important to remember that regardless of whether a systematic review is going to perform a meta-analysis or not, it is usually necessary to first analyze separately all the studies that compare the different types of an intervention to the different types of the control or comparator.

Effects will be expressed as mean differences (MDs) for continuous outcomes and risk ratios (RRs) for dichotomous outcomes. For continuous outcomes, if individual included studies have used different scales to measure the same outcome (and results could not be converted into the same scale) we will use standardized mean differences (SMDs).

After that, we will assess all the included studies qualitatively to decide if meta-analysis is appropriate for the outcome based on the degree of similarity of patient populations. Before meta-analysis, it is necessary to first assess all the included studies for clinical homogeneity with respect to type of “intervention”, and “control or comparator”, and the outcomes measured.

It is usually recommended to first assess the clinical heterogeneity and then to assess the statistical heterogeneity of all the included studies for a systematic review.

This is because if clinical heterogeneity is clearly evident, there is no point in performing a meta-analysis. Clinical heterogeneity shall be assessed by examining the types of participants, interventions for all outcomes in each study. Therefore, only if there are studies of similar comparisons reporting the same outcome measures will a meta-analysis be attempted. This also means that clinically heterogeneous studies shall not be combined in the analysis, but separately analyzed and described in a narrative summary. Therefore, if it is not possible to combine outcome data due to differences in the reporting or substantive heterogeneity, we will report a narrative summary.

However, even if all the included studies do not have a significantly high level of clinical heterogeneity, it is still recommended to check the degree of statistical heterogeneity among all the included studies by using the Higgins I2statistic.

In summary, meta-analysis will only be performed when studies are sufficiently homogeneous in terms of participants, interventions and outcomes. Both clinical (face value) heterogeneity and statistical heterogeneity will be considered, the latter by means of a Higgins I2 test.

In the presence of statistical heterogeneity (greater than 50% as estimated by the Higgins I2statistic), it is recommended to apply a random effects model for meta-analysis. This means that relative risks for dichotomous data and mean differences for continuous data will be combined using a random-effects model. It is also recommended that the significance of any discrepancies in the estimates of the treatment effects from the different trials will be assessed by means of Cochran’s test for heterogeneity and any heterogeneity will be investigated.

Finally, in the absence of both clinical and statistical heterogeneity, we will apply a fixed-effects model to pool all the data in a meta-analysis.