Hello,
I wish to calculate inter and intra rater agreement between 2 observers who are gradings Xrays for degree of bony fusion into 6 grades. In all there are 1750 xrays. Which statistical test would be most appropriate for the same? Is there an excel sheet template available for making this calculation for 1750 xrays. If not excel is there any online platform for this calculation apart from SPSS??
Key phrase here is "agreement between". Cohen's Kappa method is a strong suggestion for comparing the categorical decision scoring of two comparable "judges". Available in the SPSS Analyze-Descriptive Statistics-Crosstabs-Statistics-Kappa menu.
Shown here: https://statistics.laerd.com/spss-tutorials/cohens-kappa-in-spss-statistics.php#procedure
Cohen's Kappa gives you the strength of proportion of agreement (outside of chance agreement) between two independent observers (judges) as well as the probability that this agreement is statistically significant compared to zero (0). This is "Inter-rater" reliability NOT "Intra-rater" reliability (oh what a difference one little letter makes!).
On the other hand for the key logic phrase "agreement among" (aka "Intra-rater" reliability). Use the old reliable Cronbach's Alpha test. SPSS Analyze-Scale-Reliability Analysis-Model Alpha.
Understand that all models have specific mathematical assumptions, and that the data being tested must meet or exceed those assumptions in order for the test results to be considered mathematically valid and publishable in peer reviewed scientific journals.
Key phrase here is "agreement between". Cohen's Kappa method is a strong suggestion for comparing the categorical decision scoring of two comparable "judges". Available in the SPSS Analyze-Descriptive Statistics-Crosstabs-Statistics-Kappa menu.
Shown here: https://statistics.laerd.com/spss-tutorials/cohens-kappa-in-spss-statistics.php#procedure
Cohen's Kappa gives you the strength of proportion of agreement (outside of chance agreement) between two independent observers (judges) as well as the probability that this agreement is statistically significant compared to zero (0). This is "Inter-rater" reliability NOT "Intra-rater" reliability (oh what a difference one little letter makes!).
On the other hand for the key logic phrase "agreement among" (aka "Intra-rater" reliability). Use the old reliable Cronbach's Alpha test. SPSS Analyze-Scale-Reliability Analysis-Model Alpha.
Understand that all models have specific mathematical assumptions, and that the data being tested must meet or exceed those assumptions in order for the test results to be considered mathematically valid and publishable in peer reviewed scientific journals.
Although Cohens' kappa has been widely used for inter-rater reliability, it can have distinct problems due to the differences in the marginal distributions. Here is a discussion of kappa, along with a brief consideration of "issues" in its use.
https://online.stat.psu.edu/stat504/node/99/
A more recent competitor for kappa is Krippendorff's alpha.
You can perform this test on Microsoft Excel using Data Analysis with the appropriate test sample.
- Badges
- Science topic