We have 1000 pair of sentences annotated by 8 annotators. Each sentence must annotate by 3 annotators only. Therefore, each annotators may annotate with different annotators for each sentence. My question here how to compute the kappa for each annotators?
Any comments and recommendations will be highly appreciated.