Suppose I have one data set with 4 outcome measures. I also have 6 predictors and then I can create 9 interaction terms. I run a model with all main effects (6) and 9 interaction terms for each of the outcomes. So in total there are 4 models with 15 predictors each. All of this again is done on the same data.
Are these considered multiple tests and do I need to adjust alpha levels? Or is it correct to say that since I run a model on each individual outcome, no multiple testing is taking place?
Thanks
Bogdan
You have multiple tests whenever you run multiple models on the same data, because each model may turn out to be significant. You need to adjust your alpha, probably by four. Depending on your outcomes, you may actually try to do multiple regression with all outcome measures in one model. This may end up being more meaningful -- provided you can find an interesting interpretation of the results (and the relationships between dependent variables.) In this case, this will be a single model with a single test of significance, and thus no need for alpha adjustment.
Thanks, well one of the outcomes is actually a scale with 6 subscales so I run a multivariate model on that one. But the other 3 are not really outcomes I can combine so stuck with 4 models.
Bogdan
If you run the models separately for each of the outcome measures, you will need to adjust your alphas. However, this method would ignore the relationships between and among the four out measures. I recommend that you consider running it as a multivariate regression or as a canonical correlation. That way, you take into account all of the information your data might provide.
James, thanks. The errors of the outcomes are not correlated so four models are appropriate. But I guess the real question is what to adjust? If there are four models is the alpha inflation due to those four models, in which case it's just a/m? Or alternatively is the error rate inflation due to 15 terms in each model, in which case the adjustment is probably an FDR, which is less conservative, for those 15 tests in each model?
How did you decide that you can only test 9 interactions among your 6 variables? I count 15 interactions.
Is this from a designed experiment using factorial designs or observational data? This will help determine how you analyse your data. If your data is observational, and you make a lot of interactions, you will run into issues where the VIF among terms is high. This might fool you into thinking a significant term really is insignificant, and vice versa.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression