That's really helpful, thank you both!

I'm working on a similar project, and have been now for about 2 years. Quality assessment continues to be a big thorn in my side as well, and I'd love to know what the OP (Charlotte Fry) decided to use, and how that went/is going.

This is just my two cents: One of the issues with STROBE is that it's focused just on reporting, rather than actual design, sampling/recruitment, and the appropriateness of analysis methods. My intuition is that you may get comments from reviewers about this if you use STROBE, and they'll likely say to find an assessment that taps risk of bias, or they'll tell you to contact the authors for their data. As for Cochrane, again, they're focused just on intervention studies, so much of what you'll find won't be related to your research questions, as you've found. Their handbook tries to make the case that you can apply their ROB list to quasi-experimental and non-randomized designs, but I haven't tested that quite yet, so I can't be sure if that would work in practice. 

Additionally, there have been several papers critiquing the use of list-based quality assessments, a la Downs & Black (http://ije.oxfordjournals.org/content/36/3/666.short). Cochrane lists some of the pros and cons of D&B, as well as the Newcastle Ottowa Scale in the attached link. The main issues identified are reporting vs. actual design (re: STROBE), the applicability to case/control and cohort studies, as well as what to do with quasi experimental designs. There have also been papers saying that we need to move away from numerically-based scores of quality, because we lose the detail of what these papers actually do/do not contribute with a composite score; I believe this is the justification for the "star system" and the divided categories used in the NOS. For example, if you were using a composite score like the Downs and Black, you may end up with a score of 5 out of 10: this could mean that they didn't use a control group, the analysis was inappropriate, and the sample didn't represent the population, or it could mean that they selectively reported significant findings only, they didn't blind their participants, and they didn't adjust for confounders. The two cases are both "bad news" but are qualitatively different, and we lose some of that rich information with a composite score (which could help you make recommendations about what future studies need to do to fill these gaps and correct for these issues in design/sampling). 

All that said, unfortunately NOS is not without bias either: http://link.springer.com/article/10.1007/s10654-010-9491-z 

I don't agree with the author of the last paper on some of the issues he notes, namely that representativeness is arbitrary. This is all about writing; if the authors say their sample of British physicians is representative of the population at large, that is an issue. If the authors say that their sample is representative of the population of physicians in Britain, they're fine. It's all about not overstepping bounds, but again, you have to look closely for the authors' caveats about representativeness in both the limitations section, and the participants/recruitment sections of their MS. 

My best advice would be to make an informed, critically thought-out decision, having explored and tested many options, and having documented all those tests and their various troubleshooting issues. You'll be able to justify what you do to reviewers, and at the end of the day, as long as that justification is solid and supported by your inter-rater reliability, and several cautious caveats, you will probably be ok. 

http://handbook.cochrane.org/chapter_13/13_5_2_3_tools_for_assessing_methodological_quality_or_risk_of.htm

That's really helpful, thanks Jaclyn!  I am still debating at the moment but will let you know how I get on.

Just an update on where I'm at with this, we decided to go with the NIH/NHLBI quality assessment checklists. I found these to be an adequate "middle ground" between the NOS and the old Downs and Black checklist. 

Fab, I will check that out.  Thanks again!

Hi Charlotte & Jaclyn,

I'm doing a similar dissertation with largely quasi-experimental and cross sectional studies. The NHLBI checklists seem the most approporiate. How was your experience using these? Thanks!

Hello! I completed and published this review effectively using the NHLBI checklists to assess risk of bias. They're easy to use, specific to study design, reviewers liked them, and they provide a very detailed view of the risk of bias. My suggestion would be to take care when talking about quality vs. risk of bias with these checklists, because the literature provided by NHLBI is very clear about using it for the latter over the former. I also used them more as a commentary on the state of the body of literature that I was evaluating as a whole, rather than singling out studies with high risk of bias and talking about them individually. 

Best of luck! 

Hi Swetha,

I ended up adapting the Newcastle-Ottawa scales for mine (with the help of one of my supervisors), using the case-control and cohort as they were and creating a new one for studies that did not fit those designs based on the types of items in the other two (this has been done previously by others too).  It wasn't ideal, but did give some indication of 'quality'.

Best wishes,

Charlotte