Hi, Jennifer,

to share my perspective as a single coder. Not having a team of researchers does not preclude one from being able to assess intra-rater reliability (though you would definitely need someone to help you with establishing inter-rater reliability). As Mackey and Gass (2005) explain, firstly, the researcher codes all the data. Then, after some lapse of time (few weeks or months) he or she would need to re-code the data or some part of it. The scores achieved by the same researcher but at different points of time (hence, "intra-rater") can be compared through standard inter-rater reliability check procedures.

Content analysis is a data driven process, so your data will determine your codes or labels. To ensure validity of you coding process, there are such procedures as "external audit" aka "peer checking" (Cresswell, 2012). Writing an article about your study and submitting it to a reputable scholarly journal counts as such an approach. In the article, you would need to describe your coding process (and the logic behind it) as well as your findings. Based on the feedback  you will received from the reviewers (hence, "peer checking") you can make a judgement about the validity and, if needed, improve you coding approach. The bonus is that you can get a publication too!     

Hope this helps. Good luck with your studies!    

References 

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4 ed.). Boston, MA: Pearson Education, Limited.) .

Mackey, A., & Gass, S. M. (2005). Second language research: Methodology and design. Mahwah, NJ: Lawrence Erlbaum Associates.

The following is an older article (1999) but does have some useful focus on specifics of reliability and rigour relating to content analysis - Potter & Levine-donnerstein (1999) Rethinking validity and reliability in content analysis.  Journal of Applied Communication Research 27 258-284.

Are you not able to provide some evidence of inter coder reliability through reference to the supervisors of your PhD and their take on the data interpretation? This would be the usual resource for a lone PhD researcher.  Other possibilities would be to look for evidence of confirmation of findings through triangulation, as this would provide evidence of validity.

My first advice would be to look at publications in your field which have done similar work, to see what standards they have used.

With regard to inter-rater reliability, the popularity of this approach varies from field to field. It is very common to see it reported in the literature on communication or media studies, but the more qualitative your research area is, the less often it is used. The main thing it accomplishes is to demonstrate that you can train two people to see the data in the same way, which is of little value if your primary goal is to do interpretive work, generate hypotheses, etc.

With regard to validity, in quantitative research this primarily amounts to whether your "variables" relate to each other in specifiable ways. So, unless you are trying to measure things, that doesn't really apply. In qualitative research more generally, the preferred terminology is "credibility," following Lincoln & Guba 1985, "Naturalistic Inquiry." But most of the criteria they offer for credibility apply to observation and interviewing, not content analysis.

You can find a useful discussion of these issues in Phillip Mayring's new book, "Qualitative Content Analysis" which available online at:

http://www.google.com/url?url=http://files.qualitative-content-analysis.webnode.at/200000075-82241831d6/Mayring(2014)QualitativeContentAnalysis.pdf&rct=j&q=&esrc=s&sa=U&ei=M_aiVOaPMNH8oATQ1oG4BA&ved=0CC4QFjAE&usg=AFQjCNFrQixLHdxD7dm0jMYu6GLdE70-fA

Good to see that David is 'covering most bases'. I, personally, would associate content analysis with qualitative enquiry. In quantitative research, the basic principle for assessing the quality of the research is that of ‘rigour’ through the concepts of ‘reliability’ and ‘validity' - as Jennifer alludes to her in her study. For many qualitative researchers, the process of rigour is linked instead to trustworthiness - which includes 'credibility' as mentioned by David. Reliability and validity are generally not held to be applicable to evaluate the findings of qualitative research. There has been considerable debate among qualitative researchers for decades, but a common and unified approach to describing the criteria for quality has not emerged clearly. It may be that there is no one method that can be identified as the nature of qualitative research makes it difficult to reach agreement on criteria for assessing its quality. Due to its complexity, authors such as Schou et al. (2012) have sought to develop additional tools to offer further clarity. Their VAKS (Danish acronym for Appraisal of Qualitative Studies) tool is a new tool based on Guba’s (1981) seminal four criteria for assessing the trustworthiness of qualitative studies. Currently, there are six broad positions, which can be adopted with respect to the criteria for trustworthiness. A researcher will generally select from one or more of these positions in making the claim of trustworthiness.

Yes I did some work on this earlier. You need to apply Kapa test to check inter coder reliability and the result should show more then 70% to proceed further.

I find Johnny Saldana's 'The Coding Manual for Qualitative Researchers' to be concise and helpful. 

I'd go back to Krippendorf's Content Analysis Reader. There are a number of places in the text which deal with validity and reliability. For a more critical perspective, Patti Lather's work on context validity in qualitative research was very helpful for me. 

Hi, Jennifer,

to share my perspective as a single coder. Not having a team of researchers does not preclude one from being able to assess intra-rater reliability (though you would definitely need someone to help you with establishing inter-rater reliability). As Mackey and Gass (2005) explain, firstly, the researcher codes all the data. Then, after some lapse of time (few weeks or months) he or she would need to re-code the data or some part of it. The scores achieved by the same researcher but at different points of time (hence, "intra-rater") can be compared through standard inter-rater reliability check procedures.

Content analysis is a data driven process, so your data will determine your codes or labels. To ensure validity of you coding process, there are such procedures as "external audit" aka "peer checking" (Cresswell, 2012). Writing an article about your study and submitting it to a reputable scholarly journal counts as such an approach. In the article, you would need to describe your coding process (and the logic behind it) as well as your findings. Based on the feedback  you will received from the reviewers (hence, "peer checking") you can make a judgement about the validity and, if needed, improve you coding approach. The bonus is that you can get a publication too!     

Hope this helps. Good luck with your studies!    

References 

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4 ed.). Boston, MA: Pearson Education, Limited.) .

Mackey, A., & Gass, S. M. (2005). Second language research: Methodology and design. Mahwah, NJ: Lawrence Erlbaum Associates.

I would argue against the usage of the terms "reliability and validity" in qualitative inquiry. Assessing the rigor of qualitative data is akin to assessing the trustworthiness of the data, otherwise known as data verification in qualitative research.The  Guba (1981,  as cited in  Kreftin 1990) model of trustworthiness is applicatble to content analysis. It includes assessing the credibility, trnasferability, dependability and confirmability of your data.  Check out the following references:

Krefling, L. ( 1009) Rigpr in qualitative research: The assesment of trustworthiness. American Journal of Occupational Therapy, 45(3), 214-222

Lincoln, Y. S., & Guba, E. G. ( 1985). Naturalistic inquiry. London: Sage

Many good points have already been raised about assumptions of rigor (and tests for them) in qualitative analysis. However, here's an idea if the norm in your field is for multiple coders: ask a colleague or hire someone to code a small portion of your documents. Use this sample to gauge what biases you may have in your coding scheme relative to the other person's and include it as a sensitivity test in your methods section. Then you don't have to fund someone to code everything, but readers will have a sense of what the implications of your coding might be.

First, content analysis exists in both quantitative and qualitative forms. I'm presently using discourse-historical analysis which is a form of content discourse analysis, a thoroughly qualitative method. I assume I am subjective and I am indeed not worried about inter-coder reliability.

But the quantitative paradigm is different. It's striving for objectivity (which is mythical, but let's set that aside for now) and repeatability. If you're doing this, then you absolutely need to establish inter-coder reliability. Larisa's approach here is interesting. I think I would prefer to actually have distinct individuals doing the coding, but we are not talking about an ideal case. So, as a last resort, remembering that none of us here are likely to be the ones actually reviewing your work, you might follow her approach. That is, if you have the time.

Hello Jennifer,

If your content analysis scheme allows it, you might consider using an automated system to the actual coding using dictionaries and/or rules. This solves the reliability issue and can greatly increase the scope of an inquiry.

If you're really lucky there may be an existing coding scheme that can easily be modified to your requirements.

-Michael

This is probably one of the most debatable issues: On the one hand you might argue, that within the qualitative paradigm a different logic applies than in quantitative research. In that reading, there would be no need to employ any "quantitative help" like the calculation of the intrarater/interrater-reliability.

On the other hand you might argue, that "proper" coding is vital to a good analysis, and that the quality of coding might therefore be tested by measuring its reliability.

In the end, it is probably a question of individual taste, but as much of where you want to publish and what is required. Even though their authors are heavily in favour of checks on the reliability, you might find the literature below useful in order to form your own opinion. The abovementioned option of intra-rater checks might be one option. Regarding interrater reliability checks, taking segments of your sources might be an option to still check for interrater reliability without too much workload for others. Based on my own experience, this might prove a somewhat frustrating experience depending on the number of your codes. For me a rather time-consuming and very detailed explanation of the various codes in writing and in person yielded an acceptable result.

Good luck!

Krippendorff, Klaus (2004a). Content analysis: An introduction to its methodology. 2nd ed. Thousand Oaks, CA: SAGE.

— (2004b). ‘Reliability in content analysis: Some common misconceptions and recommendations’. In: Human Communication Research 30, pp. 411–433.

Lombard, Matthew, Jennifer Snyder-Duch and Cheryl Campanella Bracken (2002). ‘Content analysis in mass communication research: An assessment and reporting of intercoder reliability’. In: Human Communication Research 28.4, pp. 587–604.

— (2010). Practical Resources for Assessing and Reporting Intercoder Reliability in Content Analysis Research Projects. URL:http://matthewlombard.com/reliability/index_print.html#How%20should%20content%20analysis%20researchers % 20properly % 20assess % 20and % 20report % 20intercoder % 20reliability (visited on 09/02/2015).

Scott, William A. (1955). ‘Reliability of content analysis: The case of nominal scale coding’. In: Public Opinion Quarterly, 19.3, pp. 321–325.

You can apply Alpha test to check the relaibaility of the tool of data collection. If you are collecting qualitative data, let me know to suggest you some other test. 

Habib

Thank-you all for your advice - much appreciated!

Stemler, S. (2001). An overview of content analysis. Practical assessment, research & evaluation, 7(17), 137-146.