Dear All,

Basically, I developed an ergonomics risk assessment tool (instrument), for evaluation of work task. This tool have 10 items, with 2 of the item has 4 categories (low, medium, high, and very high) while the balance items have 3 categories (low, medium, high).

I need to do inter-rater and intra rater reliability to know the reliability of each items. Thus, I conducted data collection to test the tool. Total respondent was 32 people where they watched 3 videos of case study and fill up the assessment tool/form. Each rater will need to assess 3 case study and 1 form for each case study.

I performed the data collection 2 times because wanted to test for test re-test (Intra reliability).

Based on literature, I need to use Cohen kappa for intra rater (test and retest) and Fleiss Kappa for inter rater.

I am stuck with data analysis where I could not figure out how to analysis, since there are some items with 4 categories and some with 3 categories.

Many of article online showing step by step analysis for questionnaire reliability study and not specific for assessment instrument.

The question is :

1. Can I perform analysis using Fleiss Kappa or Cohen Kappa for each item? How to do it?

2. Is Kappa analysis suitable for my case, or any other suggestion?

Many thanks.