If you have multiple p values from the same family of test, you can adjust them to take into account that there are multiple tests. There are different tests, I'll let other responders deal with these, but a simple way to address this is the p.adjust function in R. If your p values are .01, .02, .03, .04, .05, you can do

p.adjust(c(.01,.02,.03,.04,.05))

and you get

[1] 0.05 0.08 0.09 0.09 0.09

(the default is Holm's method)

ps. There are issues just worrying about Type 1 errors and not Type 2.

If you have multiple p values from the same family of test, you can adjust them to take into account that there are multiple tests. There are different tests, I'll let other responders deal with these, but a simple way to address this is the p.adjust function in R. If your p values are .01, .02, .03, .04, .05, you can do

p.adjust(c(.01,.02,.03,.04,.05))

and you get

[1] 0.05 0.08 0.09 0.09 0.09

(the default is Holm's method)

ps. There are issues just worrying about Type 1 errors and not Type 2.

Thanks a lot! But I have some questions. Is the p.adjust function also applied to ANOVA (overall effect, main effect, or interaction effect)? How to define that a series of tests belong to the same family? Or in other words, is it true that if tests are from the same family, they use the same test techniques on the same variables? Besides, in some robust statistics in R, such as sppba, yuenbt, ydbt, p-values are not provided but only confidence interval, so how can I avoide the Type 1 error when multiple dependent varibales are to be compared? Thank you!

ANOVA is not controlling the family-wise error rate (FWER). ANOVA is used to compare two nested linear models, what is a bit a different question than a "family" of questions if all additional coefficients in the larger model are all zero. These tests can refer to coefficients from one or from several different predictors. Since all these tests are based on the same set of data, the complete information from the whole data can be used to get better estimates of the residual variance and thus of the standard errors. An ANOVA is only made to calculate these interim statistics, but it is not the ANOVA that is interesting. The tests on individual coefficients are t-tests, and these tests use the residual variance estimate. They can account for multiple testing to control the FWER. So it's not about ANOVA, it's only about using a testing procedure that controls the FWER.

The "family of tests" is not given by the analysis technique you use. It is the intellectual question behind your analysis. You can see tests of different predictors as independent or "unconnected", or you can see them as a family. Whenever you would say that "if at least one of these tests is significant, then ..(bla bla)" then you have a family of tests. An example: given you have treated cells with some cytokine and you test several different inhibitors, it may be the aim to show that at least on of the inhibitors shows an effect (because you know that not each inhibitor must really work; but if you could show it for at least one, you would see your task accomplished). In this case you have a family of tests and you should check the FWER. If, in constrast, you test the inhibitor at several different times, it is more a consecutive series of tests and you would stop testing after the first time-point where the inhibition is "significant". There is no family and thus no need to control a FWER.

So the definition what a family of tests is and what not must be decided by you. It is sometimes difficult and it may be that different people have a different opinion on that. That's science. And as Daniel wisely noted: the type-I error is not the only thing to consider.

Within a linear model, the FWER of tests can be controlled by neat functions like Tukey's HSD, Dunnett's procedure and so on. Otherwise, methods like that of Bonferroni and Holm have to be used. This has nothing to do with what method the data was analyzed. It is only about the p-values. They can be from "parametric" or "non-parametric" (rank-based) tests, from "robust" or "sensitive" tests.