Hello Tomohiro,

The basic argument comes down to how many tests are required to evaluate the hypothesis(es) implied by the research question. If a RQ was, "Do men and women differ in any combination of Spatial Memory, Procedural Memory, or Map skills?" Then running three anovas would not only increase the aggregate risk of type I error, it would fail to answer the "...in any combination of..." portion of the RQ. One manova could address the RQ.

The issue of follow-up tests is another matter altogether. Choosing to run univariate tests as a follow-up to a significant manova is a common practice, but not very insightful (why? because the real difference might be that of profiles, not of individual DV differences). A better option would be to run discriminant analysis (or elicit the root parameter estimates from the manova), to evaluate which combinations of DVs best differentiate among the groups.

In any event, if the decision is made to run additional univariate tests (following a manova), the concern over familywise type I risk can be addressed by invoking any number of methods (e.g., Bonferroni, Benjamini-Hochberg) to constrain the aggregate risk at what the researcher considers to be a tolerable level.

Good luck with your work.

Hi David,

Thank you very much for the great answer.

"Choosing to run univariate tests as a follow-up to a significant manova is a common practice, but not very insightful" This comment basically solved my concern. My confusion mainly came from that many studies ended up discussing follow-up ANOVA results and not much from MANOVA itself. I am not familiar with discriminant analysis, but will go through some papers on that topic and try it on my data together with MANOVA.

Thanks again,

Tomo

The claim that MANOVA reduces Type I error risk is generally false. This is largely for reasons David Morse covered. People don't in general correct univariate tests and univariate tests are nearly always what is interpreted when people use MANOVA.

The basic issue is that the MANOVA omnibus test will only protect against the omnibus null hypothesis (not differences on any DV) and not any partial null hypotheses.

See:

Huberty, C. J., & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105(2), 302–308. https://doi.org/10.1037/0033-2909.105.2.302

Thanks for the input Thom. It sounds like MANOVA is not used in a way that maximises the benefit of the method in many cases... Also, thank you for the article information. I will go through it.

Thom Baguley beat me to it in recommending Huberty & Morris (1989). Their classic article is also available via RG:

Article Multivariate Analysis Versus Multiple Univariate Analyses

Cheers,

Bruce

More generally MANOVA is useful in theory for multivariate research questions, but in practice people hardly ever have such research Qs.

I also think there is a subtler problem which is that MANOVA creates an outcome variable that in my view is usually uninterpretable (at least for scientific work). Basically it creates a weighted linear function of the DVs that maximises variance explained in the sample. This means that two studies with the same set of DVs will have different sets of weights (because of sampling error and sample differences) so effectively near-identical studies end up with different outcome variables in practice.

A better approach is to set up a proper multivariate regression model or rely on the univariate analysis that you'll end up using as follow-ups anyway.

Thanks for the article information Bruce Weaver. I have just downloaded it!