Shoukri has a pretty good book on the subject.

Hi Fatimah Petra,

in the articles

McGraw, K.O. & Wong, S.P. (1996a). Forming Inferences About Some Intraclass Correlation Coefficients. Psychological Methods, 1, 30-46.

McGraw, K.O. & Wong, S.P. (1996b). Correction to McGraw and Wong (1996). Psychological Methods, 1, 390.

you will find an overview over techniques for computing so called intra class correlations (icc) as measure of interrater reliability. in my opinion the best approach to handle the problem of analyzing interrater reliability.

KR, CR

Dear fatimah,

I first admit that I am not an expert to give you any advice or opinion on the question you asked. Upon running a quick internet search I found some resources on Inter-rater reliability that could be helpful to you. I combined those information in a single pdf file and am sharing with you.

Hope that helps.

I'll look for you in my files

In order to offer you what you need

I also have to admt I'm not an expert at this but I have also been looking into this for a research I am doing, in my case I had to stay with a qualitative analysis. Here is waht I have found out:

From a quantitative standpoint: The usual inter-rater reliability is done on a statistical analysis; this means that your inter-raters must yield points to be analyzed. The analysis can be done through the SPSS statistical program. Using what is called an interclass correlation coefficient (ICC) this measures the degree of consistency or agreement between the elements of a scale, according to the judges or evaluators. The procedure computes on one hand, the individually ICC (ICCj), which is conceptually similar to the individual reliability of each of the elements and, on the other hand, the average ICC (ICCk), which is an estimate of the reliability all the scale. In the review I did with some of my statistical colleagues, there should be more than one judge.

Now if you do it on a qualitative standpoint: you have to see, how much percentage of agreement there is between what judges/evaluators have seen on the video, or identified on the text they may have analyzed.

E.Cronbach’s alpha, c (Formula 1)

c= ___k2s2p____

k2s2p + ks2res

Where:

k= number of items in the group

s2res = variance of residual components which cannot be controlled

s2p = variance of the component for person

Illustration:

Illustration 1

Thirteen people were asked to rate a pair of questions on a 7-point scale. The pairings look different but are testing the same items, e.g. “How much do you like the weather today?” and “How do you feel about the weather today?”

People Observed score Total, yi

Question 1 Question 2

1 7 6 13

2 3 5 8

3 3 3 6

4 4 3 7

5 5 5 10

6 5 3 8

7 6 6 12

8 5 3 8

9 5 5 10

10 1 2 3

11 7 6 13

12 2 2 4

13 4 5 9

Total yj = 57 54 y =111

s2p = MSp – MSres

k

where k = number of questions

MSp = SSp

a-1

MSres = __SSres

(ab-a-b+1)

SSp = ∑yi2 -y2

bab

where a = 13 (number of items in the yi column)

b = 2 (number of items in the yj row)

= (132+ 82+ 62+ 72+ 102+ 82+122+ 82+102+ 32+132+ 42+ 92) –1112

13 13(2)

= 169 +64+ +36 +49+ 100+ 64+ 144+ 64+ 100+ 9+ 169+ 16+ 81 – 12321

2 26

= 1065 - 473.8846 = 532.5 – 473.8846 = 58.6154

2

Ssres = SS total – SSp – SS question

SSp = 58.6154

SS total = ∑yij2 - y2

ab

= 72+62+32+52+32+32+42+32+52+52+52+32+62+62+52+32+52+52+12+22+72+62+22+22+42+52 - 1112

26

= 49+36+9+25+9+9+16+9+25+25+25+9+36+36+25+9+25+25+1+4+49+36+4+4+16+25 – 473.8846

= 541- 473.8846 = 67.1154

SS question = ∑yj2 -y2 =572 + 542 - 1112 = 3249 + 2916 - 473.8846

aab 13 26 13

= 474.2307 – 473.8846 = 0.3462

SSres = SS total – SSp – SS question = 67.1154 - 58.6154 -0.3462 = 8.1538

MSp = SSp= 58.6154 = 58.6154 = 4.8846

a-1 13-1 12

MSres = __SSres= 8.1538 = 8.1538 = 0.6795

(ab-a-b+1) (26-13-2+1) 12

s2p = MSp – MSres= 4.8846 – 0.6795 = 4.2051 = 2.1026

k 2 2

s2res = MSres = 0.6795

c= ___k2s2p____ = _____(22)(2.1026)_____ = ____4(2.1026)___

k2s2p + ks2res = (22)(2.1026) + (2)(0.6795) 4(2.1026)+2(0.6795)

= ___ 8.4104____ = 8.4104 = 0.8609 (very good internal consistency)

8.4104 + 1.359 9.7694

Thak you, thank you, thank you so much all.. I'm clipping this thread in my note, so I won't lose any of your feedbacks. I need to digest everything before I start doing the rating (if I can call it as 'rating'). Really appreciate your responses, guides and information.

Fatimah,

you might also find some valuable input in:

Weber, R. P. (1990). Basic content analysis. Sage university papers series (2nd ed.). Newbury Park, CA: Sage Publications.

Krippendorf, Content Analysis: An Introduction to Its Methodology, has a rather technical discussion of this issue, with a comparison of several of the most commonly used coefficients.

The basic message is that there are several different ways to calculate inter-rater reliability, and all of them have their strengths and weaknesses. Since there is no one correct answer, my advice would be to find out what other people in your own field do, and use that as a standard.

So, look over some published articles in your field of studies, and if they use (for example) Cohen's Kappa, then that is probably what would be expected of you if you were to publish.

Krippendorff's alpha is usually a good and safe option. This manual may be helpful.

http://www.polcomm.org/wp-content/uploads/ICR01022012.pdf

If you have a complex rating process, then there is a lot to be said for Krippendorff's alpha -- e.g., if you have multiple raters and either ordinal or interval level scales on your codes. But if you are in the most frequent set-up where you have two coders who are being assessed on an agree-disagree basis (e.g., a 2 x 2 table), then a simpler approach will yield essentially equivalent information.

To my mind, the great advantage of Krippendorf's approach is that it yields one coefficient that applies to almost all rating situations. The downside is the complexity of a system that can handle so many variations.

You can also try the "Reliabilitätskoeffizient nach Holsti". I have an instruction text but only in German. 

I have tested the interrater reliability of my scoring rubric (scale) for assessing EFL learners' speaking skill by doing a Cronbach alpha test though SPSS, 

Please check my MA thesis, practical part III