Hi, I am doing the content analysis of qualitative and quantitative data in 2200 financial reports. I will statistically analyse the data. In that case, how can I confirm inter-coder reliability in PhD thesis being a single coder? Is it possible to use any software if I being the single coder, is coding the content? or otherwise can I take the service of any other coder? or can I use two coding software or is there any alternative to inter-coder reliability in quantitative/statistical inferences? Can I justify that I am using latent content for analysis? Coding sentences and paragraphs based on judgement and then assigning them the pre-defined codes seems giving some space not to test inter-rater reliability. As, I am also conducting 10-15 interviews for the same research questions using thematic analysis, can it justify the no use of inter-rater reliability? Any suggestions please!
There are a number of statistics that can be used to determine inter-rater reliability. Different statistics are appropriate for different types of measurement. Some options are joint-probability of agreement, Cohen's kappa, Scott's pi and the related Fleiss' kappa, inter-rater correlation, concordance correlation coefficient, intra-class correlation, and Krippendorff's alpha.
The definition of inter-rater reliability requires more than one rater.
Dr. David L Morgan Should it be ok to engage other coders for a pilot sample? This is not possible to engage a full-time coder for all 2200 reports. In this case, what should be the sample size for pilot testing?
One common strategy is to do the inter-rater reliability on a smaller subsample of the original data. A related strategy is to begin by working with two coders and establishing a high degree of inter-rater reliability, and then going on to work by yourself.
First of all, you need to develop the rating scale and identify the number of rater. Correlation coefficient can be used or inter-rater reliability test can also be deploy.
- Badges
- Science topic