Hi Aurea,

That is a good question, and there are conventions in each field. I'm in public health/epidemiology, which is interdisciplinary, so I never know the answer because different fields have different conventions!

Here's what I do. First, addressing the penalty for multiple comparisons only comes into play if you are posing an a posteriori (post-hoc) question. In other words, before you obtained or collected your data, did you write down the hypothesis you are analyzing and share it with at least one person (e.g., on a grant, research team, with a statistician, etc.)? If you did that, you formed your hypothesis a priori (pre-hoc), so you do not need to pay the penalty for multiple comparisons - so you get out of even having to use a correction!

(A tip I give to those into genetics/cytokines/proteomics/things that produce boatloads of correlated data on each experimental unit: hypothesize a few profiles that look supported in the literature in your grant application. That way, we get a few pre-hocs we can do without making adjustments for multiple comparisons later when we get the grant!)

But assuming it is a posteriori, what I do is take my analysis, and first, I just do it as if it is a priori, and see what I get. Then I apply a correction (Bonferronni is the easiest to explain/understand in a paper, but Scheffe and others offer corrections. Choose what you see others doing in your field in papers similar to yours, and make sure you can understand and explain it). If I get the same answer as I did before the correction, the issue is settled!

But as you can imagine, we have at least 2 comparisons on our hands, and maybe more (you mention genetics - lots!). Once we "Bonferronni" or whatever we do to our p-values, some that were significant before the application of the adjustment will "drop out" of significance. Then I talk about what got knocked out when I applied the correction, and possibly why (obviously, low p-value, but why I think there was a low p-value that didn't survive the correction).

This is a way to present your results in the literature without actually committing to a particular interpretation in your results (although you should do that in your discussion and say why you choose that).

As an aside, like you said, it's fine to do 2+ multiple comparisons, but it's really hairy to do many (e.g., 5+), especially if we are talking about t-tests with a following ANOVA. The interpretation will tie your brain in knots! Multifactor dimensionality reduction (MDR) (http://en.wikipedia.org/wiki/Multifactor_dimensionality_reduction - not that I normally reference Wikipedia but they have good links/explanation) is non-parametric and has been used (with mixed results) to reduce variables in genetic approaches. This can help reduce number of comparisons.

Then, of course, our old friend factor analysis along with principal component analysis could be used to reduce many continuous correlated data points into "factors" to (http://en.wikipedia.org/wiki/Factor_analysis). Reducing the number of variables in your multiple comparisons will make the whole adjustment thing less cumbersome (but of course, harder to explain, so keep track of what you do and explain it well!).

I hope this helps you! The answer is not straightforward! Good luck.

-Monika

Dear Monika,

thank you for the discussion of this issue.

My brain is just in knots...!

Can you please tell me if when you apply MCC in codominant model, enter all p-values?

Since MCC is applied for univariate analyses, is the multivariate analyses and/or haplotypes analysis approaches more important/valid than MCC?

Best regards,

Áurea