To assess interrater agreement/reliability of a tool (containing categorical answers) that aims at screening health related issues in older persons with intellectual impairments we have to deal with the situation that we have about 30 rater pairs. Background is that the persons to be assessed are living in many but small institutions and it needs some familiarity with the person to rate these issues properly. I've screened the IRR literature which was not surprisingly scarce as this is a rather uncommon IRR situation. I would be grateful to get suggestions on the most adequate IR measure and how to get one final IR value in the end.
This may or may not be helpful, but here goes: I assume that the raters are receiving some sort of training in how to use the rating instrument. At the end of the training period (whether this is done in a workshop setting or as an on-line activity), they should all be presented with a set of "cases" to rate. Those cases (whether real or fictitious) should contain enough material to allow for ratings to be made, but in a format similar to what might be available in the small facilities (e.g., doctor's reports, bloodwork, nurses' progress notes). Then calculate the inter-rater agreement based on those training cases and use that figure. As long as you are up-front about what you did and why you did it this way, it seems acceptable. Not optimal, maybe, but since you can't have the same raters see all the cases, what more could be expected?
The ideal situation would be that all cases would be examined by the same raters. Sometimes it is difficult because people who are being assessed are in differente places. One possibility would be to videotape the assessment session and have some raters do the assessment of all cases.
Thanks, Stephen and Claudio. We have decided to work with real cases and the staff's notes are (as quite often in those institutions) not very reliable. Thus, the assessment has to be done by staff members who know the person to be assessed for quite a time. So, the problems remains and is now more a statistical problem about which Interrater measure and which procedure (e.g., calculating the mean of all rater pairs) seems to be most appropriate.
About what I thought. By fictitious cases, all I meant was ones that can be used to establish inter-rater reliability. Those pseudo-cases could be documented well. They wouldn't be part of the study sample.
If you don't have a subset of cases (real or otherwise) rated by everyone, then you have no way of knowing whether some raters are systematically more lenient or more stringent than the rest. Again, those can be training cases you make up. But there's no statistic I ever heard of that can tell you whether two raters agree when they're rating entirely different samples.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology