Dear all,
I recently conducted a 3 (between-subject: two intervention groups and one control group) x 5 (within-subject: 5 dependent variables) field experiment in which two behavior interventions (indenpendent variables) are investigated in terms of their effects on 5 pro-environmental behaviors (dependent variables). In this experiment, I also measured two psychological variables (proposed mediators), socio-demographic variables (proposed moderators) and several covariates. The experiment aims to examine (1) the intervention effects on pro-environmental behaviors (main effect analysis); (2) the psychological mechanisms underlying intervention effects (mediation analysis); and (3) the boundary conditions of intervention effects (moderation analysis). We established 7 hypotheses regarding the main, mediating, and moderating effects before the experiment began (this is a preregistered experiment). Specifically, there are 2, 2, and 3 hypotheses concerning the main, mediating, and moderating effects that need to be tested.
I have a few questions as to multiple-testing correction.
1) As there are multiple dependent variables (5 pro-environmental behaviors) and our hypotheses apply only to the general pro-environmental behavior, I need to test each hypothesis for each dependent variable. Hence, there are five exploratory analyses for each hypothesis test. I decide to use the method proposed by Benjamini and colleagues to control for the false discovery rate (FDR). Should I apply the FDR procedure to the 5 tests regarding each hypothesis respectively (i.e., using FDR procedure for multiple-testing concerning H1, H2, ... H7, seperately) OR should I apply the FDR procedure to all 5*7=35 tests?
2) I used OLS regression to test each hypothesis. When testing the main effects, the coefficients of the intervention dummies are of interest, but there are also several covariates that need to be controlled in the regression. Should I apply FDR procedure only to the p-values of the variables of interest (i.e., theoretically meaningful) or to the p-values of all variables included in the regression (intervention dummies plus covariates)? Similarly, when testing the moderating effects, the coefficients of the interventionXmoderator interaction terms are of interest. Should I apply FDR only to the p values of these interaction terms or to the p-values of all variables included in the regression?
3) Assume that I conduct bootstrap method to test the mediating effects, as bootstrap concerns the product of two coefficients (a*b) instead of individual coefficients, how should I use the FDR procedure in this case?
I appreciate any help for my questions.
Hello Maoliang Ling
one question
Given the explanations you wrote, why do not you do this study in the model of Two-way MANOVA?
I would caution against MANOVA because it doesn't do what you think it does. There is an ongoing discussion here https://www.researchgate.net/post/Hi_I_am_currently_studying_the_effectiveness_of_activated_carbon_generated_from_rice_husk_ash_for_water_purificationWhat_data_analysis_should_I_use
In it I wrote
A lot of people assume that a MANOVA works like an ANOVA but with several outcome variables at once. This is not in the least bit true. MANOVA mashes your outcome variables together into composite outcome variables. And furthermore, it does it in a way that you cannot control. Instead of giving each outcome variable equal weighting, or even allowing you to decide on the relative importance of each variable, it basis the weighting on the relationship of each variable to the predictor variables. As Kendal Smith puts it ANOVA and MANOVA do not analyze the same variables and, thereby, address different research questions.
See
Huang FL. MANOVA: A procedure whose time has passed?. Gifted Child Quarterly. 2020 Jan;64(1):56-60.
Smith KN, Lamb KN, Henson RK. Making meaning out of MANOVA: the need for multivariate post hoc testing in gifted education research. Gifted Child Quarterly. 2020 Jan;64(1):41-55
And Bruce Weaver made this additional important point:
MANOVA "mashes" your outcome variables together in a way you cannot control. What may not have been clear is that the manner in which it does that varies from sample to sample. In other words, if you did an exact replication of your study but with a new sample, the weights used to mash together the outcome variables would be different in the two studies. I.e., you would be analyzing different outcomes in the two studies.
I'll also add the 1989 classic by Huberty & Morris to his list of articles.
Article Multivariate Analysis Versus Multiple Univariate Analyses Article Multivariate Analysis Versus Multiple Univariate Analyses
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression