What is the best accepted statistical formula to calculate inter-coder reliability in a content analysis study?

More Ruchi Jaggi's questions See All

Are there any specific guidelines to write the theoretical framework of a research study?

01 February 2016 7,417 2 View

Can the sample used for pilot testing be included in the final research sample?

I am using group interviewing technique for a research study. I conducted a pilot study on few respondents. Can I use the results of the pilot study also to write my findings and conclusion? So if...

01 February 2016 8,482 21 View

How long does it take for Google Scholar to track a citation?

01 February 2016 3,239 13 View

What are the key features of good academic writing ?

Any specific guidelines or references?

01 February 2016 7,145 1 View

How can I transfer my articles on researchgate to my profile on Google Scholar?

01 February 2016 4,314 0 View

Can I write my research design in first person or should I write in third person?

01 February 2016 6,756 3 View

In a content analysis study, can the data be transferred directly from the coding sheets to SPSS?

Can the data on coding sheets (MS Word document) be directly copied on to SPSS or does the data have to be typed in SPSS? Can you advise me on how coding sheet data should be recorded so that it...

31 December 2015 7,559 3 View

Is it okay to base your sampling design on the findings given by another researcher in a similar study?

31 December 2015 6,686 8 View

Is there any other way to track one's citation count besides Google Scholar?

11 December 2015 2,213 10 View

How to compare two groups with only two measurements?

Hiiiii everyone! I have an enquiry on statistical analysis. I was looking for many forum and it's still cannot solve my problem. I want to compare means of two groups of data but only with two...

03 March 2021 8,796 3 View

Guidelines for survey (question type) analysis?

Hi. Please tell me what guidelines should i need to follow for questionaries' type research work in India. It is not hospital based work, we are conduction basic institutional based qualitative...

03 March 2021 2,037 3 View

Enhanced Yellow Fluorescent Protein (Aequorea victoria) DNA sequence?

I am on the lookout for the Enhanced Yellow Fluorescent Protein (Aequorea victoria) DNA sequence. Does anyone know where I can find it? Thank you in advance

03 March 2021 3,568 1 View

What kind of fluid could I use for a pitfall trap that also does not invalidate molecular testing?

Hi, I want to start testing pitfall trap to obtain ants samples, but I need to conduct molecular analysis on those insects. So, what kind of fluid can I use? Ethanol expires too early and I need...

03 March 2021 5,978 5 View

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10?

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10? Since, the growth curve from day 2 to 10 won't be like the "Logistic curve" it might not follow logistic...

03 March 2021 1,401 3 View

Do we need to write protocol for a second study of a systematic review that has been registered and published?

I have conducted and published a systematic review and meta-analysis research with the topic related to public health and health pomotion (protocol was registed in PROSPERO). Now we want to...

03 March 2021 8,920 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

How to access MSCI KLD Stats and Sustainalytics?

Hello all, I'm currently working on my undergraduate thesis and I'm having difficulties trying to access KLD Stats and Sustainalytics because my campus Airlangga University and apparently other...

02 March 2021 8,525 1 View

Do I need to do linear regression before running mediation analysis using PROCESS MACRO by Hayes ?

I just wanted to check if I need to run a linear regression separately if I am using PROCESS MACRO to run mediation analysis. Thank you.

02 March 2021 4,359 3 View

Ideas for Phd's Thesis on pricing (Marketing)?

Hi everyone, I'm studying Marketing and I would like to write my PhD thesis on the topic of pricing. Any specific ideas?

02 March 2021 9,706 5 View

Muayyad Ahmad Popular answer

Hi,

The following book is excellent in answering your question. if you dont have access, I may find the exact page with the answer and send to you. good luck

Waltz C., Strickland O., Lenz E. (2010; 4th ed.). Measurement in Nursing and Health Research Fourth Edition. Springer; New York.

Muayyad Ahmad

Rubén Daniel Ledesma

Dear Ruchi,

I recomend you Dr. Uebersax's website: http://john-uebersax.com/stat/agree.htm It provides a nice summary of agreement analysis.

Hope it helps,

Ruben.

David L Morgan

Take a look at Krippendorff's, which is a general approach that includes most specific cases.

Paul Yarnold

The methodology described in the following free, brief articles remedies the shortcomings of all alternative methods.

No alternative methodology provides this set of features...

Can be applied to ratings that are categorical or ordered.

Specifically identifies points of convergence as well as of divergence between raters or methods (i.e., systems or devices).

Explicitly identifies the most accurate model for relating ratings made by raters or methods.

Models are expressed in the original measurement metric, and they are transparent and intuitive.

Provides a chance-corrected and maximum-corrected index of strength of effect: this facilitates direct assessment (comparison) of reliability between (different) pairs of raters.

Provides exact p-values.

Supports multiple methods of assessing potential cross-generalizability of the models if the raters or methods were used to assess an independent random sample of observations.

Prevents paradoxical confounding attributable to combining data from raters (methods) that provide disparate responses (e.g., see http://optimalprediction.com/files/pdf/V4A6.pdf)

Pin-points the areas of reliable agreement and disagreement between pairs of rates, facilitating efficient calibration and potentially leading to refinement of training methodology.

Which of these features is not important in research?

***

Inter-rater:

http://optimalprediction.com/files/pdf/V3A15.pdf

Inter-method:

http://optimalprediction.com/files/pdf/V3A16.pdf

Noor Saadah Zainal Abidin

Well, perhaps these references could be of assistance to you:

Bennett, W. L., Foot, K., & Xenos, M. (April 01, 2011). Narratives and Network Organization: A Comparison of Fair Trade Systems in Two Nations. Journal of Communication, 61, 2.)

Hayes, A. F., & Krippendorff, K. (2007). “Answering the call for a standard reliability measure for coding data.” Communication Methods and Measures, Vol. 1, No. 1, 77-89.

Krippendorff, K. (2011). “Computing Krippendorff

’s alpha-reliability.” Philadelphia: Annenberg School for Communication Departmental Papers.

Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Thousand Oaks, California: Sage.

Krippendorff, K. (2004) “Reliability in content analysis: Some common misconceptions and recommendations.” in Human Communication Research. Vol. 30, pp. 411-433.

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). “Content analysis in mass communication: Assessment and reporting of intercoder reliability.” Human Communication Research, 28(4), 587–604.

Van Selm, M., & Jankowski, N. (2004). “Content analysis of internet-based documents.” In M. van Selm & N. Jankowski (Eds.), Researching new media: An advanced-level textbook. Thousand Oaks, CA: Sage Publications.

good luck.

Antal Wozniak

Depending on the scales of measure and the number of attributes and the variance of attributes you might want to use more than one reliability measure. Krippendorff's Alpha is great but it is also very sensitive when it comes to variables with low variance. So you might want to add percent agreement and Cohen's Kappa.