Hi and thank you to everybody.
I have data regarding a training that aimed to improve the agreement among raters. Raters gave a dichotomous judgment (presence/absence).
The study is multi-centric: several different centers recruited different number of subjects. Each subject in each center was evaluated by three judges before and after the they attended the agreement training. Judges were the same for all subjects of a certain center, but vary among centers.
I'm interested in providing evidence that the training was useful in improving the agreement.
i have two questions:
1) it appears to me that I'm in a borderline situation between ICC two-way random (every subjects is evaluated by the same random sample of judges, that is not the case) and one-way random (every subject is evaluated by random judges, that is also not the case as there are groups of subjects evaluated by the same groups of judges). is there a way to handle groups of judges instead of opting for the more "conservative" one-way random strategy?
2) Regarding the pre-post training difference, I'm considering to use a bootstrap resample approach. is it correct to resample subjects, calculate the pre-post ICC difference in every resample and consider a significant improvement if the lower 2,5th percentile of the bootstrap ICC difference distribution is higher the 0?
Thank you very much!
sorry I do not have the expertise to answer this question.
Regards
You have to acknowledge that your sample is not random. You deal with multilevel cluster sampling. The first level is your center. Participants within the center will have non-trivial characteristics that will differentiate them from participants from other centers. This is one thing; the other is that as long as your raters share a lift with each other when they go to work in their center, they are less independent from each other than from the raters in another center whom they never met (yet another level will appear if you bring raters together from several -- but not all -- centers, for induction training.) Finally, your rater will rate more than one participant, which makes their results less independent among themselves, compared to participants rated by other raters.
Non random sampling affects measures of variability (and hence statistical significance of tests.) The best source that I know on theory of non-random sampling and its application is Lohr, S.L. (2010). Sampling: Design and analysis (2nd ed.).
Thank you Oleksly. I do understand your point and agree with it.
Do you have any suggestion on how to proceed with the analyses? I know that the one-way random is not a perfect strategy but it is the best solution i came out to by now.
- Badges
- Science topic