One of the reviewers questioned the rationale of 3-way interaction in my article.
In the article I predicted that reporting one’s gender before some test completion and having an opposite gender test administrator would activate stereotype threat for women and that women would perform as well as men only in a condition when women would report gender after testing and they would be paired with a woman experimenter.
I run a 2 × 2 × 2 ANOVA with participant’s gender (man vs. woman), experimenter’s gender (man vs. woman), and location of the gender question (before the test vs. after the test) as between-participants factors.
Results of my study confirm my assumptions (significant 3-way interaction): men performed always better than women with only one exception, where the group of women who reported gender after testing and were paired with a woman experimenter outperformed the group of men assigned to the same condition. Among men, there were no statistically significant differences in scores across all conditions; whereas women achieved the best in the condition where they reported their gender after the test and were paired with women experimenters respect to all the others; no differences emerged between the other women groups).
I got also one significant main effect (participant’s gender: women achieved lower scores than men) and one 2-way interaction (participant’s gender x location of the gender question: when gender was reported before testing women yielded lower scores than men, whereas they perform exactly as men when gender was reported after testing). 2-way interaction participant’s gender x experimenter’s gender was insignificant.
I thought that my prediction that women would perform as well as men only in one condition and that in three other conditions women will underperform men will fully justify 3-way interaction. However, for my reviewers this rationale isn't sufficient. I'm not sure how else could I justify this 3-way interaction or why my current rationale isn't enough.
I would be very grateful for any help!
Reviewers are concerned about fishing for results and then presenting as if you have not.
I think that complex interactions are entirely plausible if you explain what is going on - voting behavior is now affected by gender by age by class and not to model this in full is missing out on important relationships.
This paper
https://www.researchgate.net/publication/269409538_Modelling_Interactions_A_Random-Effects_Approach_to_the_Analysis_of_Large_Tables_of_Counts_illustrated_by_two_Welsh_Referenda
has some arguments at the beginning for the importance of interactions - it also presents a model that automatically guards against over-interpretation as differential effects are shrunk back to no effect if they are based on unreliable estimates.
But we will probably have difficulties getting accepted too!
Article Modelling Interactions: A Random-Effects Approach to the Ana...
Three way interactions are of special interest to developmental researcher. An example: There can be an effect of age on accuracy and RT that interacts with both the number of items in a display and and how long the display is presented. For example, five year old children are correct re the N in all displays, no matter how long they are shown; 4 yr olds need more time for set sizes > 3 but not for N's < 3 and 3-yr-olds need the greatest n of time for all set sizes. It is only in the latter condition that 3,4 and 5's give the correct answer for all set sizes up to 5.
So, overall we see two major developments over age: a probable shift in strategy and the related variable of time need per strategy.
I hope this helps RG
Reviewers are concerned about fishing for results and then presenting as if you have not.
I think that complex interactions are entirely plausible if you explain what is going on - voting behavior is now affected by gender by age by class and not to model this in full is missing out on important relationships.
This paper
https://www.researchgate.net/publication/269409538_Modelling_Interactions_A_Random-Effects_Approach_to_the_Analysis_of_Large_Tables_of_Counts_illustrated_by_two_Welsh_Referenda
has some arguments at the beginning for the importance of interactions - it also presents a model that automatically guards against over-interpretation as differential effects are shrunk back to no effect if they are based on unreliable estimates.
But we will probably have difficulties getting accepted too!
Article Modelling Interactions: A Random-Effects Approach to the Ana...
Was the reviewer's question whether you did indeed have a 3-way interaction, or whether you had a prior hypothesis about this interaction? In the latter case, I certainly agree with Kelvin Jones that reviewers are suspicious of "fishing" for higher-order interactions, so you would need to make it clear from the beginning that your experimental design was intended to capture such an effect.
With regard to the presence of a 3-way interaction, your tests confirm that it exists and your post-hoc analyses show that it takes the predicted form, so I'm not sure why the reviewers would question the existence of such an effect.
Thank you for your helpful suggestion.
One of the reviewers questioned whether I had a prior hypothesis about 3-way interaction.
Before my research I have hypotheses that pattern of results will look like this (I tested 8 groups, 2 x 2 x 2):
Men > Women
Men > Women
Men > Women
Men = Women
In the same time I assumed that two 2-way interaction also will be significant. All this hypothesis I justify by previous studies.
I think that for my reviewer assuming this pattern of results and in the same time assuming significance of two 2-way interaction is not clear from framing the hypotheses purely in 3-way fashion (maybe there is something like inherent contradiction between hypothesis).
In addition to the above I would add 2 comments. First, 3-way interaction terms are not frequently reported, difficult to present graphically and not everyone understands their meaning. Your peer review team may not be as facile with linear regression and interaction terms making them uncomfortable with the result.
Second, i didn't see anything in your explanation concerning literature review. Were your hypotheses well supported by existing literature? If there were logical leaps that were either too many or too large that the hypotheses were not well supported, then I might agree with the reviewers. On the other hand, assuming the hypotheses were well supported with existing literature and/or published theory, then as stated by others above, the review team should demur and allow the results. If this latter context is the case, perhaps you could engage the editor for assistance.
Good luck, and please post the conclusion as it unfolds.
I agree with the general idea suggested in all the of the comments above. That is, it is probably less about your statistical analysis and more about your hypotheses. If you can show prior research that supports or leads to your questions, I don't see a problem with your analysis. As was mentioned, people are leery of fishing. Statistical analyses are simply the tool for answering the question...I wonder if the researcher is less critical of your analysis and more concerned with whether you should have asked the question in way in which you did in the first place (i.e., they may not believe you have sufficient justification in the literature to test whether there is a three way interaction). If your question is based on the literature and conforms/confirms prior work, I think you are correct and they may be off a little. Best of luck, sounds like a very interesting paper and from what I know of stereotype threat, it seems to fit with the overall theory. I hope it works out well for you.
In general , the factorial designs are useful when complex situations , in which various factors act simultaneously on an observed fact investigated. Therefore it is important the analysis of the interaction of these variables. If you can analyze the effects of each variable separately , analyzing correlations would be sufficient
I think as long as none of the assumptions for ANOVA are violated and that there is random assignment to the cells, your methods should be okay.
For example, I can imagine if a researcher was interested in predicting the propensity for grade inflation among college teachers who encounter problematic male or female students, a three-way ANOVA would be a good statistical tool for the researcher to use (a 2 x 2 x 2 factorial ANOVA).
Knowing the professors' gender (sex at birth), students' gender and the third factor the nature of the relationship (antagonistic or cooperative) would be independent variables that influence each other and could effect the dependent variable of the assignment of course grade (number of As, Bs, Cs, Ds, and Fs assigned), if the hypotheses has to do with predicting grade inflation for combative students.
If the researcher was interested in predicting that:
Ho1: antagonistic male and female students will received better course grades than cooperative male or female students when the course instructor is male.
A three-way ANOVA would be a good tool to use to test the hypothesis for all levels of the main effects and interaction effects.
In your case, Professor Dariusz Drążkowski, I think you are correct in your statistical methods.
Suppose you have three binary predictors, A, B and C, if the model with main effects only, no interaction is used,
The regression coefficient (say for A) describes the main or marginal effect of the individual factor of A after controlling B and C. The coefficient describes the difference of the outcome means of two levels of A. For your case, the gender difference, if A is gender
If you put two way interaction A*B in, it means you think the outcome means of A varies by the two levels B, the coefficient of A*B describe double difference: The difference of the differences of A by The levels of B. If A*B is significant, the difference of outcome by A change significantly by the levels B. You need find four means outcome for A1B1, A2B1, A1B2, A2B2, reject the Null hypothesis
H0: Mean of A1B1= Mean of A2B1=Mean of A1B2= Mean of A2B2
Ha: not all of them are equal
Similarly, the three –way interaction, A*B*C , describe the changes of A*B by the two levels of C. It is triple difference. If it is significant, you need find 8 outcome means.
Three things should be mention,
1. there are two types codding used in modeling: 0 and 1 codding, the regression coefficient is related the outcome difference. 1 and -1 codding, the regression coefficient is related the an half of outcome difference.
2. Usually, the model is hierarchical, it means if the high level interaction in , all the low levels of interaction should be also included in the model,
Say if A*B*C is included in the model, then A*B, A*C, B*C, A, B, Care also included in the model.
3. Before perform the analysis, you need check the sample counts for the contingency table: AxBXC, 8 cells, if the counts are not balance, 0 counts appears, it means collinearity exists. You need re-design the model. The estimation with collinearity is biased. The three way interaction should not be used.
I have seen books on statistics and papers that prescribe rules of thumb for minimal cell counts, is five per cell and some say 10 observations per cell is ideal for factorial ANOVA tests.
Though I am not as familiar with social cognitive theory, perception, gender and social attribution theory as I am with my own field, I am fairly certain that there is adequate background literature to support your hypothesis. Without that, you are fighting a battle on two fronts: 1) complexity and general lack of experience (among peer reviewers in general) interpreting multilevel interaction terms and 2) defending a complex hypothesis without extraordinary literature support. As 3-way interactions are infrequently reported, they may seem a bit exotic to some reviewers. If that is true among those reviewing your manuscript, then remember, extraordinary hypotheses require extraordinary support (in this case background literature). I would be happy to discuss further via email or telephone.
Thank you for your kind words about my study, for your good wishes and for your help! I thoroughly analyzed what you wrote and texts which you propose to read. With this new knowledge I returned to my article and the comments of reviewer and I understood where were lacking of my justification for 3-way interaction.
Based on previous studies I made a justification for the two 2-way interaction, but not for 3-way interaction. Now I see difference between two 2-way interaction and one 3-way interaction:
Pattern of results of two significant 2-way interaction in eight groups will look, for example, like this:
Participant gender’s
Women Men
Gender reported before the test
Men Experimenter 5 15
Women Experimenter 10 15
Gender reported after the test
Men Experimenter 10 15
Women Experimenter 15 15
However, I predicted 3-way interaction and this pattern of results:
Participant gender’s
Women Men
Gender reported before the test
Men Experimenter 10 15
Women Experimenter 10 15
Gender reported after the test
Men Experimenter 10 15
Women Experimenter 15 15
In other words, I do not justify why there is a limit to how much damage can be done by stereotype threat - so if a women is already suffering in terms of performance because they are exposed to a man experimenter, they won't suffer more if they also have to write their gender
I will write that the reason I had expected that two factors that lead to stereotype threat will not result together with more strong effect is that the key mechanism for both factors is the same: priming gender identity. I also did not expected that women’s performance in that group where two factors lead to stereotype threat will be significantly lower than in two other conditions where one factor primes gender identity, because it would mean really low women’s performance in that condition. Results of metaanalysis (Nguyen & Ryan, 2008) do not suggest that stereotype threat will have such huge effect.
Thank you RG community for your support!
- Badges
- Science topic