I’ve performed a study in which 53 subjects each evaluated whether a set of 25 imaging studies was normal or abnormal. I also have the “gold standard” of whether each of the 25 imaging studies was truly normal or abnormal. What is the best way to assess the overall accuracy of the subjects’ evaluations relative to the gold standard? Can I just report gross accuracy (% correct) with a confidence interval? Can I calculate the sensitivity and specificity for each subject and then report the mean sensitivity and specificity?
I don’t want to report inter-rater reliability (kappa, ICC) because it doesn’t matter if the subjects agree with each other, only if they got it right.
Part 2 of this study involves 53 subjects that each evaluated whether a set of 25 imaging studies should be in one of three categories (low, medium, high). The gold standard for the correct category out of 3 is also known for each of the 25 imaging studies. What would be the best way to assess how accurate the subjects were at identifying which category out of 3 each study was in?
Thanks in advance for any advice.
I think Cross tabs in SPSS would be perfect for this analysis. Enter their responses on one column and the gold standard in the next column.
Run cross tabs on SPSS [check/select ‘Observed’ under COUNT; and ’Row’ under PERCENTAGEs]
If you like you can select Chi-square test to know the extent.
Good luck!
If you are looking for overall accuracy of a individual subject across the 25 images, calculate the proportion of correct evaluations relative to GS for each subject (correct/total (25). If you want an overall reader accuracy compared to GS, then pool all subjects and calculate total correct readings/ total readings (53 x 25 = 1325).
If you want reader accuracy for each image, calculate the proportion of correct readings / total readings (53)
Same for the 3 way categorisation.
Hi Rahul,
I would say that you should report whatever you think is of most interest. So: Why do you think this accuracy is of interest?
I get the impression that your issue is not just about being correct or not: results are categorized low, medium, high. That implies that there is actually a quantitative result that is categorized into three levels. If the true category is low, should evaluations be considered equally wrong if they concluded ‘medium’ or ‘high’? And if the true quantity is categorized as low but it is actually quite close to medium, should evaluations be considered equally wrong if they concluded ‘medium’ as when the true quantity is very low?
Maybe it is good to consider the consequences of incorrect evaluations and score according to the losses/damage due to inaccuracy of evaluations.
Best regards,
Rob
There are lots of arguments against using proportion accurate (this is a big field in medical diagnostics). People often use signal detection theory (e.g., the d' from that) or multilevel logistic/probit regression. These procedures are very similar (and can be the same depending on how you code your variables). kRAEMER (cited in paper below) also argues for weighted kappa. An introduction is in: Article Functions for traditional and multilevel approaches to signa...
- Badges
- Science topic
- Similar topics
- Medicine
- Epidemiology