Hello,

I am struggling with which multiple comparison correction is the best for my data. Any help would be appreciated.

I have 4 variables(let's call them a,b,c and d) with 6 distributions (let's call them D1, D2, D3, D4, D5 and D6). For variables a,b and d I have one distribution (different in each case) that is not normal, the rest of the distributions are normal. For variable c all the distributions are normal.

I performed paired t-test or Wilcoxon t (depending on the combination of distributions) and now I am trying to figure out how to perform the multiple comparison correction.

For the t-tests I was thinking about doing the Benjamini Hochberg procedure. It was originally created for independent variables in 95, in 97 they showed that the same procedure can be applied for dependent variables. In 2001 they showed that the same procedure can be applied only in positively dependent variables and that for other dependent variables some corrections are needed. I could easily check the linear dependence of my distributions with a Pearson correlation but the absence of positive Pearson correlation is not equal to absence of positive dependence. For this reason I am thinking about using the classic BH formula without checking dependence.

For the Wilcoxon tests I was thinking of doing Bonferroni correction. I searched many sources but I couldn't find any procedure except Bonferroni that does work for both non-normal and dependent variables.

Questions:

1. Do you think that it's a good plan? I actually had a quick look to my results using those and even though Bonferroni is very conservative I didn't lose a great deal of significant results.

2. If I apply what I described above how am I supposed to do the calculations? I guess for number of tests I would use 15 (all possible combinations of 6 distributions) for both BH and Bonferroni, the problem is about the ranking. BH needs the ranking of the p-values, should I include the Wilcoxon t-test values as well or just the regular t-test values when I am making the ranking?

Thanks for reading all this :)