We are developing and validating a questionnaire regarding chronic disease and disaster preparedness. The questions are all categorical in nature ie:
1) Which of the following items do you have as part of your household’s disaster preparedness?
Check all that apply
Water, two liters of water per person per day for at least three days
Food, at minimum a three-day supply of non-perishable food
First-aid kit
Working flashlight
2) Do you require supplemental oxygen?
Mark only one box
If no go to question 5
Yes
No
3) Do you have 72 hours of supplemental oxygen cylinders (compressed gas, not liquid)?
Mark only one box
Yes
No
I am struggling to find the best way to examine test-retest. I think I should be using Kappa Stats but I'm a bit confused on it and there is little literature on the subject matter.
Hello,
I assume that you are asking about how to test change after some intervention as a means of assessment. In other words, you aren't using test-retest to measure instrument reliability or response stability over time. Kappa is used for reliability or stability.
If you are interested in the increase of proportion from pretest to post-test, you would use repeated measure ANOVA. The time x group interaction is what you are looking for for change (growth) over time. If you want to know if one group is larger while controlling for where they started (pretest answer) then you would use ANCOVA.
I hope that I am understanding the question properly.
Steve
Hi Maddy
Steven is right in that the correct method of assessing test-retest reliability for categorical variables is kappa for Yes/No and nominal variables, and weighted kappa for ordinal variables such as Likert scales. You would like kappa to be at least 0.6 for each question.
Adrian
You might want to use structural equation modelling with the wlsmv estimator and modell e. g. the latent change or longitudinal measurement invariance. If you only have some single items, however a cross table (chi square or exact test) will give you significant differences in the distribution between the measurement points. As an effect size, you could use point biseral correlations on the cross tab (the higher the more stable the results of each item). This approach does not require distribution assumptions.
I share the views of earlier contributors. Weighted Kappa is most appropriate for the analysis. Alternatively, I think also that the researcher could arrange all options for each test (test-retest) and rank the responses. In this case Spearman's Rank correlation coefficient could help in first ensuring that there is a relationship between the two test results before considering the ANOVA or ANCOVA for further analysis.
- Badges
- Science method
- Similar topics
- Mathematical Sciences
- Statistics
- Applied Statistics
- Survey Design