I am currently conducting a cross-sectional study investigating the impact of different orthodontic appliances on oral health-related quality of life. The study is divided into four groups: two different orthodontic appliance groups, a non-treatment group, and a completely healthy patient group. The outcome is based on questionnaire scores, which are continuous variables.
The study has been submitted, and we have received a major revision request. One of the reviewers commented: "Sample size calculation formation is not suitable for a comparative study among >2 groups."
In the initial design, we planned to use One-way ANOVA for result testing, considering the outcome as continuous. However, since the questionnaire scores did not conform to a normal distribution, we eventually used the Kruskal-Wallis (K-W) test. The formula currently used for sample size calculation is: n=(2(z_α+z_β )^2*σ^2)/δ^2.
I am seeking advice on how to address the reviewer’s comment concerning the sample size calculation in a comparative study among multiple groups with a non-normally distributed continuous outcome. Any suggestions or references to guide the appropriate sample size calculation method or statistical approach for this study would be greatly appreciated.
Thank you in advance for your valuable input!
It seems to me that your four groups contain a number if variables.
First there is a binary variable that identifies "needs treatment" or not.
Then there is the "treated or not" variable, and finally there is the "treated with which appliance" variable.
You need to think carefully about hypotheses here. It seems to me that you have several:
1. Untreated have worse quality of life than normal controls
2. Treated have better quality of life than untreated
3. Treated have the same quality of life as normal controls
and
4. One of the appliances is better than the other.
I don't know what your specific hypotheses were, but you need to cone in on them and test them. The ANOVA is pretty pointless : why would there be no difference between any of these groups in quality of life? I don't see a credible null hypothesis.
Yu Chen, you said that the study has been submitted, and that a reviewer was unhappy with the sample size calculation you did (if I follow). You also wrote:
In the initial design, we planned to use One-way ANOVA for result testing, considering the outcome as continuous. However, since the questionnaire scores did not conform to a normal distribution, we eventually used the Kruskal-Wallis (K-W) test.
There are multiple issues here.
1) There is no point in estimating the needed sample size after data collection has ended. See Section 3 (retrospective power) of this short note by Russell Lenth: https://homepage.stat.uiowa.edu/~rlenth/Power/2badHabits.pdf
2) Your comment about switching from the ANOVA you intended to use to the Kruskal-Wallis test makes me suspicious that you relied on a statistical test of normality. Generally speaking, that is a very bad practice--especially if you carry out the test on the DV rather than on the residuals. But even if you carry it out on the residuals, tests of normality have far too little power when n is low, and have too much power as n increases. IMO, you would be much better off asking yourself if it is honest, fair, sensible, and defensible to use means and SDs for description of the DV (by groups). If the answer is yes, then ANOVA will likely be just fine. This issue has been discussed many times on RG. And I summarized my thoughts on it in a short conference presentation several years ago. You can view the slides here:
Presentation Silly or Pointless Things People Do When Analyzing Data: 1. ...
I doubt you can share your raw data, but can you at least share the n, mean and SD for each of your groups? Some kind of distribution plot (by group) would be helpful too--e.g., box-plots. Thanks for considering.
I hope this helps.
Bruce Weaver makes an excellent set of points and I definitely should have noticed the “read more“ tag to discover that this was a completed study.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Applied Statistics
- Survey Design
- Questionnaire