Without knowing the studies involved I can only assume that only population s with good cancer ascertainment, genetic analysis and confounder measurement were included in the studies in the meta analysis -- even though all continents were included. Thus the meta analysis may be said to lack generalisability (ie external validity)

Thank you James Leigh

A high-quality meta‑analysis uses appropriate methods of data collection and analysis (possesses internal validity), properly represents the literature being analyzed (possesses external validity), and provides a distinct theoretical contribution to the literature.

External validity is the extent to which research results provide a correct basis for generalizations to other circumstances. That is, the extent to which results of a research study can be applied to answer a particular question correctly, without introducing systematic error.

Therefore, the most important factor affecting the external validity of a meta‑analysis is the representativeness of the sample of studies. Ideally the sample of a meta‑analysis should contain every study that has been conducted bearing on the topic of interest.

In order to assess the external validity of a meta-analysis, it is therefore necessary to ask the following question: “Can you generalize the results to populations not included in your systematic review?” If the answer is “yes,” then the meta-analysis has achieved good external validity.

This means that for a meta-analysis to have good external validity, it is necessary to ensure that:

[1] Include as many eligible studies as possible in the meta-analysis.

The results of a systematic review with a larger sample size will typically be more generalizable than those with a smaller sample size, because using a smaller sample size for a systematic review will lower its generalizability (and thus, its external validity).

[2] Ensure that the participant characteristics of the included studies in the meta-analysis have allowed the coverage of a broad spectrum of differing participant characteristics by setting a list of appropriate inclusion criteria that will allow the inclusion of a wide variety of participants with broadly differing characteristics.

This means that if the conclusions of a systematic review are restricted to a participant population of a specific sex, age, geographic region, socioeconomic profile, etc.; then these conclusions may not be able to extend its generalizability to the other participants with a different set of characteristics. Hence, in order to improve the external validity of a meta-analysis, it is necessary to ensure that the inclusion criteria of a systematic review have been broadened to allow the inclusion of a wide variety of participants with broadly differing characteristics. By doing so, we can be more confident in ensuring that the setting and question addressed by each individual study included in a review will be able to more precisely match the setting of the review question.

[3] Ensure that the study setting that is being examined in a meta-analysis is able to be extrapolated to the other settings.

This means that if a meta-analysis is designed to evaluate different determinants of a patient-reported outcome in different urban settings, then it is necessary to ask the question “Are the same findings valid in other urban settings in a different country? Are the findings adaptable to rural settings?” for assessing its external validity.

In order to achieve high external validity for a meta-analysis, it is necessary to ensure that the setting of the systematic review question to be precisely match the real-world setting to which the review question relates, by carefully formulating the review question to appropriately justify its scientific rationale and methodology. This requires a very comprehensive and careful review of the background subject matter by performing a thorough literature review at a preliminary stage.

[4] Ensure that the comparator which has been selected to compare against your treatment of interest in a meta-analysis should ideally be a close match to real-world practice. This is another way of increasing the external validity of a meta-analysis.

This is because in order to ensure good external validity for a meta-analysis, it is necessary to ensure that the setting and question addressed by each individual study included in a review to precisely match the setting of the review question (and this can often be achieved by appropriate eligibility criteria are chosen to closely match to the real-world setting).

In conclusion, external validity requires that the setting of interest (i.e. that is specified in the review question) must be clearly defined and that the causal relationship of the setting of each study included in the review must also be generalizable to the setting of interest. External validity thus has two independent (i.e. non-overlapping) aspects, referred to as “applicability” and “transportability”, which can often be framed as general questions when assessing research studies for their external validity.

Applicability and transportability can often be assessed by comparing in detail the PECO / PICO elements of the research study against those of the setting of interest and examining whether differences exist in confounding variables between the research study and the setting of interest.

In summary, the external validity of the review question itself (i.e. whether the review question matches the real-world problem that the review intends to address) should be considered and justified during the formulation of the review question and transparently justified in the review protocol. For example, a systematic review of pesticide effects in laboratory studies would clearly not reflect real-world pesticide effects but such a review may be justified if the aim of the review is to inform a predictive model.

A useful and pragmatic way to assess the external validity of studies included in a systematic review is to consider systematically how well the key elements of the studies (i.e. the PECO / PICO elements and other relevant aspects of the study design) match those of the review question. This is the general approach recommended in the previous version of these CEE Guidelines and Standards (version 5.0, April 2018). By explicitly considering and reporting the applicability and transportability components of external validity, we can reach an overall judgement on external validity of a meta-analysis.

A final note is that we should not be confused between ‘external validity of the included studies for answering the review question’ and ‘external validity of studies included in a systematic review.

The external validity of an included study (for a meta-analysis) for answering the review question is whether the study is asking an appropriate research question. This refers to the generalizability or applicability of a study’s findings.

However, the external validity of a meta-analysis (the extent to which the answer to the review question is free from systematic error when informed by a specific study) depends both on the characteristics of the research study and the evidence review question being addressed. Thus, external validity is context-dependent.

You can refer to the following sources of information should you require any further clarifications and/or explanations on external validity of a meta-analysis.

[1] https://learning.edanz.com/validity-systematic-review/

[2]https://environmentalevidence.org/cee-critical-appraisal-tool/