Hi Matt

This is a tricky one, and I am not sure that there is any consensus yet among statisticians. There have been a couple of good papers that discuss the issue, and based on these I usually advise  researchers to NOT undertake adjustment for multiple testing, but instead discuss the issue in the Discussion section.

Here are the papers:

Ronald J Feise. Do multiple outcome measures require p-value adjustment? BMC Medical Research Methodology 2002, 2:8.

Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990 Jan;1(1):43-6.

Adrian

Question 1: My first reaction is that the reviewers are correct to say that when alpha = 0.05, if there are twenty items and one difference between treatment and control is significant, then it could have happened by chance. But isn't that always true with inferential statistics? So I agree with your understanding of inference and its role under certain circumstances.  In other words, it really should not matter how many outcome variables you tested, if the variables are independent. 

Question 2: I do not see how an adjustment for multiple comparisons is responsive to the reviewer comment. This is normally done for changes in model specification. The only thing this can do is make your one significant variable not significant. 

But I have several more fundamental questions for you to consider keeping in mind that I have no idea about the purpose of  your research.

1) Did the literature or pilot data support theory to show a  treatment impact on 20 independent phenomena?  In other words, do you understand the basic science behind the treatment mechanism well enough to justify the statistical properties of Independence that you observe empirically? I can tell by your write up that you were wrong 19 times out of 20. We need to this published so that others know not to use these precious degrees of freedom. The current ethical consensus is that negative findings need to be published--especially in intervention research. 

2) Why are you not setting alpha = 0.01? I assume it is because your sample size is too small and to get a larger question would be too expensive. Alternatively, did your a priori power analysis have estimates of the hypothesized effect size? This gets back to my first question. 

3) I hope you know where I am going with this. I think the best literature to bring into the paper is the one on pre-study odds of having a finding. I don't know how much of this you did, but I think you can re-frame the paper as primarily one of negative findings. Perhaps there is a creative way to do some pre-study odds calculations post-hoc and then let the reviewers know what you did or would have done to give you confidence in your one significant finding. I've attached a link.

http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124

This conclusion is not statistically right. For making fun of it,  let say 20 percent of world population are Chinese.  So if you chose a new born infant randomly out of new infants population,  it is 20 percent probable that the kid has Chinese nationality.  Then, if your nigbour are going to give birth to her 5th child, you can expect a Chinese kid !

Although such argument may seems right psychologically,  but logically they are false. This type of inferences are called fallacy. 

Probability laws are only correct for huge sample sizes.  If the probability of car accident is 1 percent, you wouldn't think that if you take your car out of the garage 100 times, you will have an accident. 

In your case, the reviewer can not conclude such reason, but practically speaking, this amount of difference might be neglectلable, so you can pass presenting it in abstract and do not pay any words in discussion. 

I agree with Richard. Is it really so that you have 20 hypotheses to test?

If that is the case and you have no knowledge of how large a relevant effect is I would recommend adjustment. For this one may use Bonferroni's methods or the less conservative method by Holm-Bonferroni. The adjustment scheme used in Holm-Bonferroni's method gives you a correct significance level for a family of tests, thus resolving the problem of mass-significance.

I have read papers with extensive testing (+50 tests) in what seems to be a search for significant results. I don't recommend it. It resembles data mining, which (though interesting in itself) is another brach of statistics.

It is a rather strict method to use, but when screening data sets for correlations or associations I think that's what you should do. Every data set points in some direction and if you don't know what to expect it is difficult to interpret the results. Where to draw the line is another question. Should an adjustment be made for all p-values within a paper or just within a certain analysis or table? I have no answer to this. The question of "study-wise error rate" is an interesting one.

I think the bottom line is that you should be careful when selecting what to test. Base the choice of tests on the currect litterature or you own hypotheses. Maybe you will have to do som extra analyses to explain strange results or to further illuminate the research question. In those cases I seldom do any adjustment for mass-significance. In my experience those extra analyses are limited in number.

I have no comment. As a reader I just want to thank all the discussants for enlightening insights and reflections. Thank you!

I would tend to agree with the reviewer, that 1 out of 20 (or 5%) items being significantly different between experimental group and controls could be expected by chance alone.  As I understand it, adjustment for multiple comparisons basically has the same effect as lowering the alpha level, so that your p-value would be equivalent to, let's say, .02 instead of .05.  

You say your questionnaire questions are independent from each other. I don't know exactly what your research project is but have you considered using some kind of factor analysis to see if some questions are related to each other?    And maybe that one question that is different between the experimental and control groups reflects an important domain.  Similar to questions on smoking addiction: one single question for smokers, "do you smoke within 30 minutes of getting up in the morning?", identifies who is strongly addicted with a very high predictive value. 

Another thing to consider is that your lack of strong results is not because of an effect size (i.e. a difference in scores between the two groups) but is because of a small sample size.   I recommend doing a power analysis to estimate if having more subjects in each group would make the results more statisticallyl significant.  It is very common in small studies to reject findings as insignificant when it is really a sample size issue, and not an effect size issue.  I believe some authors even recommend declaring a p-value of .10 or below as significant in small studies for this reason---although a p-value of  .09 would still be considered only "potentially" significant and should be addressed in the discussion along with sample size issues. ( In that case the familar statement is used: "additional studies are needed." ) 

Think of it conceptually this way:  Should you reject your results because five times out of 100 they would occur by chance if there is no relationship (i.e., if the experimental treatment really has no effect you would see these results 5 times out of 100 anyway)?   But if you get these results 6 times out of 100, then they should be accepted?    Remember that the .05 cutoff comes from a historical tradition from looking up the .05 level in paper booklets of values well before the age of computers or even calculators and that page became dog-eared.   It is arbitrary---although defensible---in medical and social research to use .05.  However, in legal circles a p-value of .49 is enough to establish liability---i.e., if there's a 51% chance that implants cause medical problems, then the makers of implants are liable for damages.     Science, however, is very cautious.  The analogy is more like deciding on the guilt of a person who would go to prison if guilty:  you want to be very sure of their guilt (be sure you have good reason for rejecting the null), beyond a reasonable doubt.  If there is less than a 5% liklihood of them not committing the crime (i.e., it's >95% probable that they did the crime), only then can you convict them (or reject the null hypothesis) and send them to jail.  Some guilty people will go free, no doubt, but the innocent will rarely go to jail: the wrong conclusion (Type I error) will seldom be made but somtimes a Type II error (setting guilty people free) will be made.   

However, in some medical research the cautionary principle is more important.  If there is only a 1% chance that children could be harmed or killed by some vaccine, should the vaccine be considered "safe"?    Probably not.   But if you might save five lives out of 100 who would die, is that worthwhile?  Probably so. 

So a lot depends on how you set up your hypotheses and explain your results.  As others have said, negative findings are important to know (although hard to get published).  But in your case I would investigate sample size first.  The free software G-Power is excellent at computing sample sizes for different experimental designs and statistics (although the user manual is very minimal).  The more subjects you have in each group, the easier a small difference will be found to be statistically significant.    I hope this helps. 

20 times, and one p=0.05, precisely (0.05=1/20); reviewer is right

Just a comment to precise that working with effect size does not completely avoid the problem of « multiplicity corrections », at least when working with confidence intervals. In fact, the matter is the same: if you do 20 95 % CI of something without effect, you expect in average 1 that will not overlap with 0...

However, confidence intervals are indeed much more informative than p-values and much easier to interpret. Sadly, multiplicity correction may be more difficult if you want joint 95 % intervals.

My last comment would be that indeed, with 20 variables tested, multiple testing problems arise; however, it depends what you are trying to answer.

If you conclude that something as an effect (in general for the study) if _at least_ one of your test is significant, you definitly must correct for multiplicity to keep a whole 5 % Type I error level.

If you conclude that something as an effect (in general for the study) if _all_ of your test are significant, you do not have to correct; you could even work with higher individual risks...

If you do not draw general conclusions but simply look at each test without trying to draw any general conclusion, then debate is open; I think you should at least use something related to « false discovery rate control » (FDR).

This is a very interesting and controversial topic. It is also affected greatly by the philosophy of statistics and how people interpret probability. I for one do not subscribe to this notion of family-wise error (the multiple comparison problem) because of several strong reasons. However, it is an extremely common experience in medical field and I as a reviewer always request for it and as an author always correct for the multiple comparison problem...

The theory behind this thing relies upon the alpha and beta values. However, most statisticians do not even approve the notion of statistical significance, let alone silly alpha and beta values set at values determined stupidly.

Plus, there is absolutely no way to identify a family (as in the family-wise error). While your reviewer said you had 20 comparisons, another person can count 50 comparisons and another person can count 2000000 comparisons for the same report, based on the way each person defines the "family".

So this  "multiple comparison problem" is absurd. However, I always ask for it, and always do it! Because it is too difficult (actually impossible) to change such a huge common error. I also always report P values, while completely believe that it is absolutely absurd! this is the unfortunate part of science.

The best available approach for this thing is to switch to Bayesian statistics. However, in my field, people even have difficulty understanding the difference between McNemar and chi-square tests, let alone Bayesian statistics! Once a reviewer asked me to omit most of my (really basic) stats, as (s)he considered them difficult for the readers... Yes, this is science!

Note that a single p-value is often misleading.  Please see attached. 

Article Practical Interpretation of Hypothesis Tests - letter to the...

Interesting how often this topic is debated. More interesting that books are starting to arrive that refer to these "New Statistics." In large samples the effect size is the where the action is. Please see http://www.amazon.com/Understanding-The-New-Statistics-Meta-Analysis/dp/041587968X

These responses have been really helpful, and recommended readings insightful. Thanks so much for helping me develop my library on this. The topic doesn't come up a lot in my research, but when it does it's helpful to have some good resources handy. 

Matt - 

You might want to check out the question link attached, which may be somewhat related to yours, as a starting point anyway, for continuous data. 

Cheers - Jim 

https://www.researchgate.net/post/Is_Effect_size_or_CI_a_more_reliable_as_an_alternative_to_P-value

Matt -

Back to your original question: If you consider a simple null hypothesis, it will never be exactly correct. How low the p-value might tend to be depends upon two things: (1) How far from true is this value?, and (2) How large is my sample size? The further from 'true' the hypothesis, the smaller the p-value will tend to be. Also, the larger the sample size, the more even a small discrepancy between 'truth' and null hypothesis will tend to drive down the p-value. So, i would expect a p-value will tend to be smaller than the uniform distribution of p-values obtained when the null hypothesis is always true. Thus if your 20 tests could be treated independently, it would seem that the expected number of cases with a p-value less than 0.05 would generally be greater than 1.

Considering sample size alone, for a given test, as sample size goes up, observed p-value would tend to go down.

So only getting one p-value below 0.05 in your example does seem to indicate that in an overall sense, null hypotheses may appear to be close to true, assuming such an overall sense has any practical meaningfulness. But on an individual basis, you would still want to see if a 'low' p-value meant something. To do that, for each case you need a power analysis or other sensitivity analysis.

Setting a level of 0.05 in each case to test 'significance' would be undesirable, by the way. The power analyses, considering sample size, would possibly dictate different levels be set for each question.

Just because you might expect one of 20 cases, where a null hypothesis is basically 'true,' to have a p-value less than 0.05 does not mean each test should not be evaluated independently, with a power analysis. You likely will have some p-values that are substantially smaller than should be expected, and some larger. Overall this may 'wash out' to a reasonable overall set of results, but that is not of much interest.

For the reviewer to say "You have tested 20 items and only found one to be significant, which is the number expected by chance at alpha = 0.05. I don't think these findings are very strong or reliable," seems to only mean that overall you may have had not many questions with lower than expected p-values, given the null hypothesis, but what really counts is what happened for each of the 20 individual tests, and in each case, the "significance level" should be set based on sample size.

The more basic problem is that what really matters for each question is not "Is this hypothesis right or wrong," but actually " How wrong is it? Is it close enough for practical purposes?" It will not be exactly "right."

So I think the reviewer should not have made an overall statement such as she/he did. It appears to serve no purpose.

Therefore, if the survey items are independent, it would seem they should just be considered/treated independently.

Perhaps I misunderstood your question, or otherwise botched this analysis, but I think the reviewer was making a true, but for all practical purposes an irrelevant, observation. Well, I mean "true" in the sense that the collection of results was unusual overall, but that does not justify the statement that "I don't think these findings are very strong or reliable." It was just an irrelevant observation, without practical meaningfulness.

Cheers - Jim