This question is about a reviewer’s comment on a paper that I sent to an appropriate journal. I first describe the research in the paper. It had three research questions. The first question asked whether an intervention affects significantly the outcome (dependent) variable. To answer the first research question, we ran ANCOVA test that takes into consideration the differences of students’ scores before the experiment in the three participating classes. We did that, though we had taken account of these differences by running ANOVA for the scores of the three classes before the experiment. In addition, when getting significant differences, to find out which pairs of groups differ significantly, we ran Scheffe post hoc analysis.
The second and third research question asked whether the interaction of the intervention with gender (two values) and ability (three values) affected significantly the outcome variable. To answer the second and third research questions, we ran two way ANCOVA. The two independent variables, in our case, are the class type (three classes, one experimental and two control) and student’s gender (or ability). The covariate is the value of the outcome variable at the beginning of experiment.
Sending the paper that included the research to an appropriate journal, I received a comment from a reviewer that 2x3 ANCOVAs to test the interactions actually make the one-way ANCOVAs redundant. In addition, the reviewer requested more information about the size of the samples (three samples: one experimental and two control) and cell requirements for your analyses (2x3 and 3x3). The reviewer said that the sample sizes meet the 10 subjects per cell statistical requirement, but likely put the robustness of the 3x3 analyses in question. Can anybody explain the reviewer’s comments please? Why 2x3 ANCOVAs to test the interactions actually make the one-way ANCOVAs redundant? How to compute the robustness of the analysis? What is the importance of this robustness?
Hello Wajeeh,
It seems to me that all three questions are answerable from a single analysis: a 3 (class, which apparently is the treatment dimension) x 2 (gender) x 3 (ability) ancova (where, we presume, some sort of baseline score is the covariate). Since you didn't indicate sample size, it's impossible to comment on whether cell size is adequate (and, for the record, a blanket assertion by reviewer that 10/cell is sufficient, could be wildly misleading).
This set-up lets you test:
1. main effect of class (question 1), holding other effects constant
2. class x gender interaction (question 2), holding other effects constant
3. class x ability interaction (question 3)....
4. gender x ability interaction (even though you don't declare that as a question...
5. class x gender x ability interaction (even though...)
6. main effects of gender and ability, holding other effects constant
In this framework, the ancova (and anova) for the sole effect of class is redundant with that of the full, 3-way model analysis.
Good luck with your work!
Hello David,
I really thank you for your clear and detailed response. The sample size was (number of participants in the experimental group=32 , number of participants in the first control group=33, number of participants in the second control group=34). Can you elaborate on the reviewer's saying "but likely put the robustness of the 3x3 analyses in question"? I mean you addressed the redundancy of the one way ANCOVA. It would be great if you could address the robustness of the 3x3 ANCOVA.
Best regards,
Wajeeh
Hello Wajeeh,
Still not sure I understand what is meant by "...robustness of 3x3 analysis" being "in question." So, I don't have a glib response for you. If what the reviewer is asserting is that running a one-way ancova, then a two-way ancova on a data set that would have most logically been handled as a three-way ancova is potentially problematic then I partially agree. But, my agreement is only because the two analyses would have: (a) different error terms; and (b) ignored plausible and possible systematic sources of variation.
If the reviewer's concern is that having more cells means smaller cell size, then that really only impacts higher order interaction terms (and those not as much as you might imagine). The main effect of treatment, for example, has exactly the same number of cases (per treatment) regardless of whether you analyze it as a one-way, two-way, or three-way design. The statistical sensitivity would actually tend to be higher in the three-way design, since you'd expect error term to be smaller by having identified more systematic sources of variation to be separated (from what otherwise would have gone into the 'error' pile).
Hope that helps.
Hello David,
I really thank you for great help. Another reviewer suggested using MANOVA to study the interaction instead of ANCOVA. What do you think? I really appreciate your help and clear reasoning, so I am encouraged to ask in order to discuss the reviewers' comments. Doing that, I will be able to do better the revisions.
Thanks in advance.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology
- Research Papers