I have 3 raters and 3 categories of behavior (Looking at partner's face? Yes, No, undetermined), with a code given every second for 150 seconds for each of my 20 subjects.
I have tried calculating Fleiss' Kappa and Gwet's AC1, but 1) that gives me separate reliability results for each participant and I'm not sure how to get an overall measure of reliability - Do I average the reliability results across participants?, and 2) I'm not sure if these statistics assume independent ratings, in which case they would be inappropriate for time series data.
.
if i understand well, "subject" can be understood as a video or film ; there are 20 of them and they are rated by 3 raters (150 measures per film for each rater) on a set ot 3 (nominal, ordinal ?) categories
applying Fleiss'Kappa for each subset of measurements related to a film leads to 20 values of Fleiss'Kappa
as a simple approach, i would suggest to combine them with the "stratified alpha" approach
http://mjesec.ffzg.hr/revija.psi/vol%2021%20no%202%202014/Wahyu%20Widhiarso%201.pdf
.
from my understanding, Fleiss'Kappa requires independence of the raters and assumes the priors on each rater are the same ; i do not remember any independence assumption between measurements (but if you wanted to go beyond the index itself and the "rules of thumb" for its interpretation, for instance calculate confidence intervals and the likes, the question might resurface ...)
"stratified alpha" combination assumes independence of measurements between "subjects"
.
.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics