Hi,

Gwet's AC1 is appropriate for skewed category prevalence, in contrast to Cohen's Kappa. Assessing agreement on individual symptoms provides a more detailed understanding. T-tests aren't suitable for nominal data; consider using non-parametric tests or Chi-square tests.

Hope this helps.

Thanks Hsin-Yuan Chen ! Yoyr reply was very helpful, I finally retained Gwet´s AC1 as my official agreement coefficient for my work. However, I have pretty low agreement (34 % agreement 24 % expected, ac1 .14) but significant in the overall set of symptom. I don´t understand if this is an effect of the categories, the raters or the symptoms. Anyway, I obtained some weird values when I tested rater´s agreement over a single symptom (all non-significant and pretty low values), so I have to check this in more detail (probably by hand calculation and checking each step). with regard to the t-test I finally get convinced that t-test is suitable because it compares two quantitative coefficients (as I guess Gwet himself demonstrated in Gwet (2016) -- Gwet K. L. (2016). Testing the Difference of Correlated Agreement Coefficients for Statistical Significance. Educational and psychological measurement, 76(4), 609–637. https://doi.org/10.1177/0013164415596420

Fernán Arana

Hi,

It's heartening to see you've adopted Gwet's AC1 for your work. Low agreement values can arise from factors such as categories, raters, or individual symptoms. A closer look at each symptom might illuminate the sources of disagreement. Your understanding of the limitations of the paired t-test with nominal data, and the suggestion of non-parametric tests or Chi-square tests, demonstrates a deep appreciation of statistical analysis. Well done!