I am revising some literature in the field of knee cartilage repair and trying to apply some robust methodology for paper quality assessment. To this aim I used the modified Coleman score. Whilst I had no problem in attributing the scores for all of the 8 questions, I have a doubt: what is the cut off? In other words, when can we consider a paper as "good quality"? When it goes above a score of 70? Higher than that? Any suggestion or fixed parameter you are aware of? Thanks.
Hi Britta, I am trying to score the report rather than the bias in this specific case.
I also did bias assessment and I used a funnel plot where I inputted standardised mean differences of post operative scores accounting for the study size. This was to ensure homogeneous results because in some studies different scores were used. Any suggestion? Thanks!
Hi Federica - can you post a link to the Coleman score method? I'm not sure I will be much help (my advice will probably be not to score, or at the very least not impose nominal divisions such as "good" or "average" over the top of an ordinal scale) but I will see if there is anything I can contribute.
Hello Paul, sure.
The page 1 of the following pdf is a good representation of the scoring methodology. Thanks.
Interesting stuff! However, I would personally be very cautious about the validity of this method. If you are assessing the quality of a paper (in this case, it looks like a report of a clinical trial), then I would suggest reconsidering whether or not a risk-of-bias approach is going to work better.
One of the fundamental problems with scales is they conflate, into a single quality score, the methodological differences between studies which are important for interpreting their validity. I believe this has been shown both empirically (Juni et al., attached) and in principle (Greenland & O'Rourke, attached) to prove unreliable for predicting systematic error in studies.
I don't see anything in the Coleman method which addresses this problem. For example, it puts a lot of weight on arbitrary boundaries between scores: for example, a recruitment rate of 81% would give a study more points than a rate of 79%, when in reality there may be no practical difference between the two in terms of the error this difference introduces.
Also, continuing to illustrate the problem with scales, if you are looking for a quality cut-off of e.g. 70 for good vs. bad (or however you are doing it), how will you deal meaningfully with one study which scores 69 and another which scores 71? Is there really an important difference between the two? (Of course there might be, if you look at what the study did: but if you have to go into the detail to figure that out, then you should be questioning what value the scale had in the first place.)
I'm probably sending you back to square one, and I know scales are widely used and very popular, but I am unconvinced they are the right sort of approach - I just don't believe they actually measure what they claim to be measuring. I hope this helps a bit, anyway.
Article On the bias produced by quality scores in meta-analysis, and...
Article The Hazards of Scoring the Quality of Clinical Trials for Me...
Hi Paul, the points you raised are absolutely valid. Actually what I tried to do is to combine this (score the only score I am aware of to elucidate the "goodness" of a paper in the field of orthopaedic-related surgery clinical studies) with an assessment of bias through a classic funnel plot analysis which gave me an overall pooled effect estimate of 0.3 for the papers considered, which are fairly symmetrically distributed across the funnel. This is part of a more complex meta-analysis I am trying to carry out, but being a cell biologist this is not really my cup of tea. I can see you are knowledgeable in your area, perhaps there is room for collaborating together and spit out a publication :-) ? Anyway, I will have a look to the suggested literature and try to get my head around this. Thank you so much.
I'm very far from being an expert in this, particularly when it comes to meta-analysis. I'm also still not entirely sure of what it is you are doing (I'm not sure how your funnel plot treats risk of bias, for example), but if you wanted to send me a draft of your paper I would be happy to look at it in case I could be of help. I would treat it in confidence, of course.
- Badges
- Science topic
- Similar topics
- Chemistry
- Pharmacology
- Pharmaceuticals
- Drugs