We conducted a study, where three investigators with different experience levels measured joint range of motions. We calculated the ICC an 95% for each combination of the there investigators (s. table). We found significant differences when looking at the 95% CI. (do you know a test to calculate?)
We now want to calculate norm values. Should we use the mean values of the three investigators with different skill sets or the mean of the two investigators with the highest ICC scores (All vs. HP-SFB)?
Thank you very much for your help!
To calculate norms, you want to use the most accurate scores. In your situation, these would be the scores from the experienced tester/s. If you have gross differences between testers, was this random error or systematic error (did you use 1-way or 2-way ICC? Did you compare means with an F-test?). One almost has to assume, barring evidence to the contrary, that disagreement was due to the inexperienced tester.
Was the highest ICC between the two more experienced testers? I'm not sure about your last question - what is HP-SFB?
If there are significant differences between scorers I'm not sure that you can justify creating norm values as your data suggests values change according to the measurer. If you can argue that experience leads to more accurate measurement then you must exclude the data from the inexperienced rater, although you have a very small number of raters and so I do not feel you can really make this claim. I would suggest that if you want to create norm values, you need more raters in order to ensure that all raters score in agreement
I agree with Dave.
I also wonder to what extent judgements of either of the three investigators were actually consistent (consistency meaning that an investigator doesn't change his/her frame of reference for judgements during the judgement process, or across repeated judgements).
Wouldn't you want to check this first, before only thinking about using an investigor's test scores for establishing norm values?
I can't tell if you ran any test on consistency by the data you provided.
- Badges
- Science topic