It depends on the details of the studies. To judge this, it takes a thorough investigation of all the details. If you can't do that, or if the investigation doesn't show anything that would explain why these studies are "outlying", there is no justification to ignore these studies.

Note that outlier tests are tools that just help you to spot those studies (or values) that would be worth to be investigated in more detail. A "significant" outlier test is by no means a justification to ignore a study (or a value). A justification can only be given based on the subject matter, never solely by the data.

If one observes rather many outliers (or if this happens regularily), it may not be that the studies (or values) are "bad", but instead that the statistical model (its functional or stochastic part) we imply is not ideal. This often happens, for instance, when we assume a symmetric (or even normal) distribution on the values where a skewed distribution model would be more appropriate.

There are few tests for outliers. See the article 'Algorithm 974: The OutlierLib - A MATLAB library for outliers' detection' , ACM Transactions on Mathematical Software, 43(4):38, March 2017 (https://www.researchgate.net/publication/315519154_Algorithm_974_The_OutlierLib_-_A_MATLAB_library_for_outliers%27_detection).

do not remove them;

Article TO DETERMINE SKEWNESS, MEAN AND DEVIATION WITH A NEW APPROAC...

It depends on the details of the studies. To judge this, it takes a thorough investigation of all the details. If you can't do that, or if the investigation doesn't show anything that would explain why these studies are "outlying", there is no justification to ignore these studies.

Note that outlier tests are tools that just help you to spot those studies (or values) that would be worth to be investigated in more detail. A "significant" outlier test is by no means a justification to ignore a study (or a value). A justification can only be given based on the subject matter, never solely by the data.

If one observes rather many outliers (or if this happens regularily), it may not be that the studies (or values) are "bad", but instead that the statistical model (its functional or stochastic part) we imply is not ideal. This often happens, for instance, when we assume a symmetric (or even normal) distribution on the values where a skewed distribution model would be more appropriate.

Alireza, for your particular instance I refer you to Jochen's response.

Because your question is with respect to how to consider metadata, and Jochen's sufficiently provides an answer to your question, I want to provide similar rationale for the scenarios that I find myself in when considering outliers.

In short, I avoid excluding values UNLESS I KNOW (and I emphasize this) that a value was obtained under flawed conditions. For example, if I perform an experiment and I set up each condition with say 6 replicates, and I KNOW that I inadvertently but unintentionally manipulated one of the replicates in a manner different from the rest, I will make note of this. If this sample gives an outlier (usually determined by statistical tests) value and this outlier can be explained by my inadvertant manipulation, then and only then do I consider excluding it from the data set.

There is a balance here, you don't want to exclude values without strong justification, but you also don't want to keep values in that can be explained by some KNOWN factor (e.g., human error) that has nothing to do with the science you are investigating and attempting to describe.

Thank you all your answers are very helpful.

Alireza,

I'd follow the Jochen and Ricardo's advice. I'd check, though, the extent to which your results depend on these outliers. After all, I suppose that you want to explain a general pattern, instead that your results are almost exclusively generated by one or two cases. If the latter was the case, I'd devote more time to understand what is going on with the outliers before rushing to publish your findings.

Best regards from Madrid,

Andrés

Dear Santana

Thank you for the recommendation.

Hi Everyone,

I agree with the consensus to leave these studies in the analysis. If the inclusion/exclusion criteria were outlined a priori, unfortunately we cannot select studies that match our personal views after all of the effects have been calculated. What we typically do in situations like these is report the overall mean ES when ALL of the effects are included, and then re-run the analysis excluding these potential outliers. Report both results in a "Sensitivity Analysis" section of the manuscript and let the readers decide. It also provides you with an opportunity in the discussion to point out that these "outlier" studies were so different that perhaps caution should be used when interpreting their results. Or, on the other hand, maybe these results offer a much more effective treatment or therapy than what had been previously used and should be examined more thoroughly in future research.

Good luck!

Mike