There were 8 interviews, each interview had 9 questions, and each transcript was coded independently/concurrently by two coders (same Coding Scheme, 9 different themes). Do I demonstrate inter-coder reliability for each interview, each theme or each question?
The basis of the research is I developed a questionnaire to assess disaster preparedness in a population with chronic medical disease. I then preformed 8 cognitive interviews that examined each question in the questionnaire, essentially assessing if what I think I'm asking is the same thing the participant thinks I'm asking. These interviews were transcribed, a coding scheme was developed with 9 codes/themes.
Coding Scheme :
A) Would like question to be elaborated on, and/or more information to make questions easier to answer
B) Asks for clarification
C) Gives improper, incorrect or inadequate answer
D) Has a different understanding of what the question than the one desired
etc.
Then two coders went in and coded the interviews based on the above coding scheme. I'm trying to demonstrate intercoder agreement but I'm not sure how I should assess the agreement, by question, by interview, by theme.
Thank you for any help.
My own experience relates to the use of thematic analysis and content analysis rather than the use of cognitive interviews. In these cases I examined agreement on themes. However, this was used to try to demonstrate reliability - and it sounds like this might not be your goal.
You state that you are not trying to demonstrate intercoder agreement. For me this raises the question of why you had two coders involved. If you can answer that why question, you may be able to identify what you should be doing.
Hi Maddy, I applaud you for taking the long journey to develop and validate your own questionnaire. To your question, first, identify the main themes and subthemes (therefore, not by interview or by question) from both coders that are most relevant to your study goals. Those are the ones that should acquire your attention. However, since there are only 8 interviews, the transcript may not be too long to try and discuss the less relevant or smaller themes with your second coder.
Intercoder reliability assessment for content analysis is one of the most common method to establish reliability however there is no consensus of a single, best way to do this (To the best of my knowledge). Authors such as Creswell (2014) and Silvermann (2008) proposed the commonly used percent arithmetic where agreement at 80% of major themes and subthemes are acceptable. Others include Cohen's kappa, Holsti's method and Krippendorff's Alpha. Lookup their pros and cons and determine which is suitable for your project. Since you are asking on reliability, I assume validity assessment is done and I presume quantitative validation is next, so...hope this is helpful and All the best!
It seems to me that where your 2 coders agree on themes by question, then your questions will be reliable too.
I sounds you are dealing with the issue of "unitization." To do any assessment of inter-rester reliability, there has to a "unit" that each coder has marked, and then you need to use statistics (such as Krippendorff's Alpha) that work with their degree of agreement.
In your case, since you are working with cognitive interviewing, the most obvious unit would be each respondent's reaction to a question.
- Badges
- Science method
- Similar topics
- Teaching