Hello,
I am conducting a research degree and for my ethics I am required to provide an sample size for participants.
In one study I will look at the reliability of a screening tool(s) where two (potentially groups) of raters watch a performer conduct an action and give it a score.
I therefore need to work out how many raters I need and how many participants I need.
For example, it will be easier to get more participants rather than raters, so could I have say 100 participants and 2 raters, as this would give more data points than 10 raters and 10
participants?
However, the issue is…I do not know what screens I will be using as of yet, as these screens will be decided as part of the research (a focus group with experts). Therefore, I do not know if the scoring system on one screen might be 0-10, and on another screen it might be 0-3. I believe that this might affect any calculation?
Any advice on where to start would be great.
Hello Alex,
Since you don't as yet even have a handle on what the final measure(s) will be, perhaps you should wait until your experts attain consensus, then, as Natacha Heutte 's reply suggests, try a quick pilot study. That will give you valuable information on how much of the score variation is a function of respondent, item, and rater, respectively. The general principle is, the more variance a source adds to the mix, the more levels of that source you'll need in your study. This is what can help you to determine the best possible allocation of your valuable resources and time to gather information to best address your research questions.
I recognize that deferring the sample size declaration may be a bit frustrating, but otherwise you're forced to: (a) possibly mis-estimate the appropriate sampling plan, resulting in either less precision or needless data collection; or (b) try to persuade the IRB that your resources limit you to X observations total, and just go with that.
Good luck with your work.
Hello,
I advise you to consult the book of David Machin, "Sample Sizes for Clinical, Laboratory and Epidemiology Studies" (ISBN-10: 1118874927). I think the solution is there. There is a small utility to simulate the number of subjects needed.
Sincerely
Hello Alex,
Since you don't as yet even have a handle on what the final measure(s) will be, perhaps you should wait until your experts attain consensus, then, as Natacha Heutte 's reply suggests, try a quick pilot study. That will give you valuable information on how much of the score variation is a function of respondent, item, and rater, respectively. The general principle is, the more variance a source adds to the mix, the more levels of that source you'll need in your study. This is what can help you to determine the best possible allocation of your valuable resources and time to gather information to best address your research questions.
I recognize that deferring the sample size declaration may be a bit frustrating, but otherwise you're forced to: (a) possibly mis-estimate the appropriate sampling plan, resulting in either less precision or needless data collection; or (b) try to persuade the IRB that your resources limit you to X observations total, and just go with that.
Good luck with your work.
Next to the great advice, I can't help but stating the obvious: you need larger sample of raters than 2 for any meaningful estimate of inter rater reliability.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size