I have classified a set of code review comments by myself and need to check the inter-rater reliability between me and my 2 supervisors. Obviously, they cannot categorize all 417 comments I categorized for this. How can I determine a suitable number of comments for them to categorize so that I can calculate the inter-rater reliability?
Dear Sanuri Gunawardena
Generally, For reliability analysis the standard advice is to have at least 10 participants per item on your scale. However, this should be regarded as the bare minimum.
Kind Regards
Qamar Ul Islam
Dear Qamar Ul Islam
the 10:1 'rule' is outdated and conservative. Modern sampling methods consider simulation studies in order to determine sample size and statistical power. Also, this was not the question Sanuri asked - to which I do not have a qualified opinion, sorry.
Hopefully, someone else can help you out.
Best
Marcel
Thank you both. After looking at more literature and the web, I found that Intraclass correlation coefficient can be used to calculate the sample size. Yet, it is used in literature for all data types except nominal that I'm working with. I will keep looking. Hopefully, someone will be able to help me.
An article reading can give best idea on the requirement of minimum sample size for performing reliability test: Guo, H., Pohl, E., & Gerokostopoulos, A. (2013, January). Determining the right sample size for your test: theory and application. In 2013 Annual Reliability and Maintainability Symposium (pp. 1-9).
Manojit Chattopadhyay Thank you! As I understand that paper, you determine the sample size based on the confidence interval and bounds ratio you need but on what do we determine the confidence interval and bounds ratio?
- Badges
- Science topic
- Similar topics
- Comment