I am constantly frustrated by my colleagues and students getting their manuscripts returned either rejected or with suggested revisions based on inappropriate statistical advice. I consult across the full range of 'health research' and have noticed that particularly in some disciplines (e.g. pre-clinical) the advice reviewers give is either inappropriate or often wrong. How do we deal with this problem? Should the student just 'roll-over' and take this inappropriate advice, or should they 'fight it out'?
Let me give you an example: assumption checks. In many clinical studies, particular care is given to appropriately powering the study. That is, calculating the appropriate sample size to detect a particular clinically meaningful effect (e.g. between two means). I recommend that researchers use their eyes to gauge the (approximate) equality of variances between two groups (i.e. variance homogeneity is an assumption of the standard independent sample t-test), but many reviewers (no doubt SPSS users) will recommend the researcher perform a Levene's test for variance homogeneity. This is pointless for two reasons:
1. The study has been powered to detect a particular level of difference between two means which may or may not mean it is adequately powered to detect a difference between two variances.
2. If there was a 'statistically significant' difference between the two variances, does it follow that this difference is sufficient to cause problems for the (generally robust) t-test, or conversely (when we have small samples) does a non-significant result for the Levene's test mean we are safe.?
This is just one example. The main problems is that reviewers are often context experts who may have poor research methods knowledge and/or experience. There is clearly a major shortage of methodologists out their. How do we solve this problem?
Is it not sufficient to respond to incorrect advice with a thorough explanation of why the advice is inappropriate, at least when the paper isn't rejected? A good editor should be able to objectively evaluate such responses, then either ignore the review or request an additional reviewer with more appropriate background.
As Marty Kardos has suggested, one approach is to respond to reviewer comments with a thorough explanation of why one believes the advice is incorrect. But for things where inappropriate advice can be anticipated, perhaps a more proactive approach should be taken. E.g., one could add a footnote explaining why one is not testing for normality prior to doing a t-test, etc (with supporting references when possible). Even if those footnotes end up being removed for the final version of the ms, they might serve the purpose of stopping some reviewers from giving bad statistical advice.
p.s. - Re the homogeneity of variance example you gave, I like this well-known quote from George Box:
“To make the preliminary test on variances [before running a t-test or ANOVA] is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port!”
- George Box, Biometrika 1953;40:318–35.
Marty Kardos and Bruce Weaver both make good points. Where an article hasn't been rejected but a "major revision" is requested, there is the opportunity to respond refuting the errors. (I agree it's more difficult if it is rejected, other than to try another journal.) If the confusion is expected, a footnote (as sugested by Bruce) is probably more effective than a statement in a covering letter that the reviewers may not see.
It might also be pertinent for supervisors whose students repeatedly experience this problem from particular journals to write to the editors suggesting that at least one reviewer with the appropriate skills set needs to be allocated. It would be a good idea if they offered themselves as future reviewers, except where there is a conflict of interest. Journal editors struggle with reviewer pools that are too small.
Besides statistical issues, reviewers can err by having a personal agenda and missing the points being made. As one example, I had the opportunity to respond and defend my approach when a reviewer with a particular agenda made recommendations for adding chunks of text that would have changed the nature of the article (from history to advocacy).
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology