A questionnaire with a Likert scale rating was pretested among the same respondents on two separate occasions. How to assess the test-retest reliability of the questionnaire? Which statistics has to be calculated to determine the reliability of the responses?
The simple answer is to compute a Pearson Product-Moment Correlation Coefficient between the total scores. A square of this result will provide the percentage of overlap. If you wanted to look more closely at the reliability of items, I would suggest using computing Spearman Rank-Order correlations between each matching item. This would also help to identify low performing items.
When James E. McLean says "between the two total scores" he is describing the case where you have a scale composed of several Likert-scored variables. If you have only a single such variable, then you should do the ordinal equivalent of a correlation.
Cronbach's alpha is also a widely used test for internal consistency estimation. Particularly useful when the items in the questionnaire are not very many or too few; a flexible measure.
James E. McLean and David L Morgan, sir, my questionnaire have several Likert scale dimensions. Thank you for the suggestions. Helpful. Shall I also compute the intraclass correlation coefficient. Some papers also reported Cohen's kappa, Kendall's W, so I am a bit confused.
Joseph Kipkorir Cheruiyot, thank you, sir, for the suggestion. I have to report the alpha value obtained in both test and retest? Do we need to compare the two values obtained?
Dear Nidhi,
After seeing the additional information you added, you only need to compute the reliabilities for each dimension. I would suggest that you use the Cronbach's Alpha for each dimension in the pre and post measures and then compute a test-retest reliability for each dimension as I suggested earlier.
Good luck,
J. McLean
It sounds like you should proceed in two steps. First, determine coefficient alpha for each time period, to establish that your scales have sufficient reliability. Then assess the test-retest reliability with a Pearson's correlation.
Try to do a reliability test on using R programming or SPSS and check on the Cronchbach alpha value
Nidhi Joshi, as some contributors above have indicated, it is probably wise to check the internal consistency of each dimension within each of your scales before assessing test-retest reliability - and it could well be valuable to do that for each measurement occasion. I do, however, have reservations about the point(s) at which you might need to be worried by the results you obtained. For example, when is a coefficient alpha (AKA Cronbach's alpha) too high or too low (in my view, that's quite fuzzy), and what difference in alpha values would raise concern? Maybe you could report your findings primarily by way of demonstrating that you left no stones unturned in your analyses.
As you have acknowledged, there are several ways in which to assess test-retest reliability. To a large extent, what method you should aim for is dependent on your data. If you have continuous data that can be regarded as at the interval level for statistical purposes, I think you would be better off using the intraclass correlation coefficient (ICC) rather than Pearson's correlations - though there are some contexts in which Pearson's correlations fit the bill nicely.
If you use ICCs, it's important to use the appropriate ICC because there are 10 kinds of ICC and use of an incorrect version can result in researchers deceiving themselves (and others).
In case it's helpful, you might like to read an article about ICCs that I wrote around 4 years ago. Here's the reference:
Trevethan, R. (2017). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology, 17, 127–143. https://doi.org/10.1007/s10742-016-0156-6
All the best with your research.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Probability
- Probability Theory
- Reliability