Risk of bias and methodological quality (MQ), in general, are thorny subjects. No perfect study has ever been done, but if you judge by risk-of-bias and other MQ indexes, many studies have achieved perfect scores. The size of literature is also a consideration: If you have a small pool of studies, discerning any trends as to whether MQ affects study results may not be possible. If you have a larger pool of studies, then there are more things that are possible to do. I blogged about it in the link below, if you are interested. (If you look in the comments, there is a preprint of an article that looks at the issue directly.)

http://meta-analysis.ning.com/forum/topics/methodological-quality

There is not other way to do it.  It is necessary to navigate through the articles many times to make a good judgement

Thanks for the quick help. The article is very interesting. We have a large pool of studies and will, as you mentioned, rate the "risk of bias" and the "PEDro score" for the RCTs. Think this will give us a good overview of the overall quality of the studies. Thanks again

Risk of bias assessments are one of the most time consuming steps of a meta-analysis, but necessary as the quality of a meta-analysis is only as good as the trials in it. Unfortunately there are no quick shortcuts. There has been much published regarding the utility of various scores, the use of which is actively discouraged by the Cochrane group, and for fair enough reasons. The Cochrane handbook's section on risk of bias assessment is a good summative discussion of this.

Risk of bias assessment is also very useful if you have a large number of studies, as it enables you to perform secondary analysis to look for any difference between high and low quality studies. The use of a qualitative tool like Cochrane's has an advantage over scoring systems as it allows you to choose which components of the risk of bias assessment are particularly relevant for your meta-analysis. Meaning that for a particular comparison, blinding might be particularly important so you can give that more weight when deciding which trials to group as low vs. high quality in your secondary analysis. Such determinations are of course best made a priori.

There maybe quicker ways but these may not be 'better' ways. The RoB assessments can be done in REVMAN which is fully/freely downloadable and you can export these as Figs or Tables. Remember one thing... that most authors carry out these assessments based on what was reported in the paper but these may not necessarily reflect what was actually done. Its an expectation that you make reasonable attempts to contact investigators to clarify these issues. For example trials may often be reported as 'double-blinded'  this means nothing as it stands and is often confused with allocation concealment or vice versa. The methods used and how successful they were is as important. Trial reports often dont include this information because they maybe removed for brevity by editors etc.

Hi

I would also recommend RevMan (http://tech.cochrane.org/revman/download). It will prompt you to describe why you judge a specific domain as low/high/unclear risk of bias. It also has a nice feature that allows you to see the risk of bias assesments in the context of each individual meta-analysis, making it easier to judge if the result is at high risk of being biased. RevMan has no tool to handle the consensus process. You could do that in Excel (https://sundhedsstyrelsen.dk/da/sundhed/kvalitet-og-retningslinjer/nationale-kliniske-retningslinjer/metode/~/media/3C0E60B708F8491D9700D781F285B3BF.ashx) or Access or you could try one of the data extractions tools that are out there f.ex. Covidence (www.covidence.org) or Distiller (http://distillercer.com) 

Cheers,

Britta

Thanks very much Britta for the links.

Also thanks to Zbys for the advice for the Revman. i just downloaded it - just need to connect to my review group. Yes, I know it takes lot of time to do the risk of bias analysis but I think it's absolutely neccessary to take this time but also have the possibility to double check different items.

BTW we are developing a ' consensus' tool  specifically for Cochrane reviews which can be adapted which we will present in Hyderabad at Cochrane Tech http://tech.cochrane.org/cochranetech/schedule 

abstrackr and Distiller are great but not Cochrane specific and have a learning curve.. Plus from experience theyre not truly intuitive.. ie if ur not using them regularly you have to re-learn. Bit like working with EXCEL.. its easy to forget how you did what you did last time you used it.  ;-)

Thanks a lot!

A little bit off-topic: a quicker way to do a systematic review might be through automation technologies. Find more in article attached.