See this paper

https://www.ncbi.nlm.nih.gov/pubmed/30458262

This attached section from Higgins et al., (2011) is so helpful for your question, although its more about meta-analysis but it could be applied to systematic review.

See this paper

https://academic.oup.com/bja/article/117/4/428/2410577

Sander Greenland called so-called Quality Scores 'useless and potentially misleading' (AJE 1994;140:300-302). Susan Olivo and colleagues reviewed 105 studies of 21 scales and modifications and concluded that 'most scales were not rigorously developed or tested for validity and reliability' and that 'scale scores may provide a false impression of meaningfulness' (Olivo et al., Phys Ther 2008;88:156-175). This is the major reason for a move from numeric quality scores to simpler Risk of Bias tables.

To see if your quality scores provide a false impression of meaningfulness, if you are planning to do any meta-analysis, you could split your studies at the median quality score and examine the forest plots, summary effect measure, and I squared statistics for the 'high' and 'low' quality studies.

Suppose you found a smaller and more consistent effect size in the 'high quality' studies, and all of them had adequate concealment of allocation, and the low quality studies didn't. You would have a reasonable methodological basis to trust the high quality study pooled result. But if you can see no apparent relationship between study effect size and reported aspects of study implementation, it would be difficult justifying excluding some studies on the basis of a 'quality' score.

The concept of quality scores and their utility have evolved since the studies by Greenland as well as by Olivio. See the first link I posted. Just to be clear, we agree with Greenland and all concerns he raised have now been resolved in terms of how quality scores should be used and interpreted. A median split is no longer a valid approach or for that matter any form of stratification by quality

Thank you Suhail, for sharing that excellent paper. We agree on the limitations of quality scores and you have convinced me not to use them as a stratification variable.

Thanks Mark. It's been a long time getting to this point but it seems that the relative approach resolves many of these issues.

I would recommend Suhail A. Doi comment.

Dear Zahra Nasr

The methodological quality evaluation scales are useful insofar as they allow filtering the studies. When low-quality elements are introduced into an analysis, they generate noise and their applicability is reduced. In this sense, systematic reviews only recommend including studies that exceed these standards. However, there is still a bias that evaluators may have regarding quality, which is why this is done in a blinded way and with a third party that can settle the differences.

Regards,