For my thesis I am developing and validating a questionnaire to asses with people with COPD (a chronic respiratory diseases) are engaging in disaster preparedness. The data is all categorical (you either have a flashlight or not sadly can't Likert it). I am moving into the test-retest portion. I will have to use Kappa stats to examine it's test-retest reliability. I have no clue how to determine sample size and I'm having a hard time finding any literature that tells me how to do it. Any help greatly appreciated!
The difficult part is deciding what the minimum shift you would realistically like to be able to detect is. In some ways you would like to be able to detect super small differences, but this requires huge samples. That's the hard part and it is context dependent so you probably won't get useful answers here, but will need to think about it and discuss it with your colleagues. Once you have that, and decided what design (for example, is it just one hypothesis, the flashlight [I thought Canadians said "torch" like the Brits], or do you have multiple hypotheses), then you may want to do a power analysis. At that point you could post the more specific question and likely get useful answers.
Hi Maddy,
Sim & Wright (2005) The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements
And there is already a question thread regarding kappa in more general terms with more references:
https://www.researchgate.net/post/What_level_of_cohens_kappa_interrater_have_you_seen_published
Best wishes, René
Hair et. al. (2006
Green (1991)
Cohen (1968)
Field 2012
can be helpful
Hi, Maddy,
In addition to the suggestions above and the link provided by René, please note there is another question thread regarding kappa, which you may also find informative:
How to calculate the sample size for performance the Kappa coefficient? https://www.researchgate.net/post/How_to_calculate_the_sample_size_for_performance_the_Kappa_coefficient [accessed Mar 7, 2017].
As Prof Conroy aptly points out there:
The sample size depends on three factors:
Sample size formulas are given in:
Cantor, A. B. (1996) Sample-size calculation for Cohen's kappa. Psychological Methods, 1, 150-153.
There is an R package called irr which has a command called N.cohen.kappa that does the work for you.
Warm regards,
Deborah
https://www.researchgate.net/post/How_to_calculate_the_sample_size_for_performance_the_Kappa_coefficient
Thank you everyone for all the help and yes we say flashlight in Canada not torch :)
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire