You might look into Krippendorff's alpha, which is a generalized statistic for handling a wide variety of situations involving inter-rater reliability.

Here is a link to a paper by Krippendorff himself that gives more details:

http://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&context=asc_papers

Hey Pedro, good question! Obviously, inter-rater reliability is the level of agreement of the raters ( assessors) on each and every items. So, you can correlate their responses and see the existence of the consistence, based on level of the indices.

Good Luck,

Berhanu

So. When I run the matrix for inter-rater, raters are columns, items are row. So whenever I ask for intra-class correlations, what I have is an output related to raters, not items. The opposite, would turn raters into a categorical variable and items would be columns. That'd sound like a normal correlation and the software wouldn't actually run with the rater ids. Am I.missing anything? 

You might look into Krippendorff's alpha, which is a generalized statistic for handling a wide variety of situations involving inter-rater reliability.

Here is a link to a paper by Krippendorff himself that gives more details:

http://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&context=asc_papers

Thank you gentlemen. I've used krippendorff,  but in stata its not able to provide item specific inter-rater reliability.

Take a look at the attached.

I can send code or run it for you if the data suits. I'm not 100% sure from your description.

Research A GEOMETRIC APPROACH TO CONDITIONAL INTER-RATER AGREEMENT

just use an ICC  inter-class correlation

The item by item evaluation can be done through simple observation

hi In SPSS u can go to anlyse then scale and then reliability analysis. u will get interrater reliability.

 I have a same question with this. Did you find a way to calculate inter-rater reliability for each item? If so, could you share it with others? Thanks,

Maybe it is also an option to calculate the coefficient of variation, which is normally presented as ratio of the standard deviation to the mean. Nevertheless, as your are dealing with a Likert scale (ordinal value), I am not sure this will be statistically right. The Krippendorff's could be better for this.

제 생각엔 설문조사 결과를 구조방정식 모형을 통해서 분석해보면 어떨까 싶네요~

Hello Pedro, item-level reliability is in my opinion an important and often overlooked aspect of psychometric measurement. Unfortunately, it is not apparent to me that calculating an intra-class correlation coefficient  (ICC) is possible given your study design. In particular, it does not appear that you have more than one target being rated by your 15 raters. ICCs can only be calculated if multiple raters (either the same or different raters per target) are rating multiple targets. This is because between-target variation is an essential value for the estimation of the ICC, and with one row containing 15 column values for your rater scores (or vice versa) you do not have any degrees of freedom with which to calculate between-target variation because there is only target (where df = n - 1).

If you would like to get a better understanding of ICCs I suggest reading Shrout and Fleiss' (1979) classic paper on the subject, which is available at the link below.

http://www.aliquote.org/cours/2012_biomed/biblio/Shrout1979.pdf

Here is a brief, open-source article on three methods of assessing inter-rater reliability on a single item:

• https://odajournal.com/2016/11/11/novometric-vs-oda-reliability-analysis-vs-polychoric-correlation-with-relaxed-distributional-assumptions-inter-rater-reliability-of-independent-ratings-of-plant-health/

Pedro,

I am looking at a similar issue.

You might find this helpful: Article Interrater reliability: The kappa statistic

Kappa has limitations which are overcome using the methodology I cited earlier. Here are some open-source manuscripts addressing this issue.

· https://odajournal.com/2016/11/04/oda-vs-%CF%80-and-%CE%BA-paradoxes-of-kappa/

· https://odajournal.com/2014/03/29/unioda-vs-kappa-evaluating-the-long-term-27-year-test-retest-reliability-of-the-type-a-behavior-pattern/

· https://odajournal.com/2014/03/28/unioda-vs-weighted-kappa-evaluating-concordance-of-clinician-and-patient-ratings-of-the-patients-physical-and-mental-health-functioning/

Yes, indeed, Dr. Yurdabakan.

Unfortunately, arbitrary weighting schemes used to apply kappa to ordinal data are unmotivated statistically and have no basis in measurement theory. Furthermore, forcing a single model to "best fit" the data of multiple (samples of) raters has the potential to induce confounding attributable to Simpson's Paradox.

https://odajournal.com/2015/04/21/estimating-inter-rater-reliability-using-pooled-data-induces-paradoxical-confounding-an-example-involving-emergency-severity-index-triage-ratings/

Yarnold, P.R. (1996). Characterizing and circumventing Simpson’s paradox for ordered bivariate data. Educational and Psychological Measurement, 56, 430-442.

Hello Pedro,

I think You can use Kripendorrff's alfa or log lineer models.

Good Luck!