Hello! I want to give a brief overview of my research (it's a thesis) for a better understanding of my "problem".
I have three research questions:
RQ1: it holds two hypotheses both using the same set of independent variables. The difference is in the dependent variable. Hypothesis 1 is using the main score of a published scale (planned analysis ANOVA). Hypothesis 2 is using the four sub-scores of the same published scale (planned analysis MANOVA) because the literature suggests testing for differences in BOTH: the main score AND the four subscales.
RQ2: has one hypothesis using partially the independent variables from RQ1 but a new dependent variable (planned analysis logistic regression analysis)
RQ3: has one hypothesis using partially the independent variables from RQ1 & RQ2 but again a new dependent variable (planned analysis multinominal logistic regression)
In addition, I have to conduct two manipulation checks (Chi²-tests) and a "Nonresponderanalysis" (planned analysis t-test) as well as some exploratory analysis (mostly descriptive but with one planned t-test and with 3 planned correlation analyses).
As one can easily see, a lot of testing was conducted on one data set, but unfortunately, I wasn't allowed to narrow it down. I'm concerned about the Typ1 Error.
I am confused since I found sources saying it's only considered multiple testing if the tests use the same dependent variable. Other sources mention that Bonferroni should be used if the same set of independent variables is used multiple times. And others are saying that if multiple tests are "done" to the same data set there needs to be a correction of the alpha level according to the total number of tests conducted with the same data set.
Depending on which of those "explanations" of multiple testing I consider it makes a huge difference for the number used to divide the alpha-level i.e. using Bonferroni-correction.
My question is: WHEN do I correct for the multiple testing in my case? Overall at the very beginning or for each research question? Do I only correct if the same DV is tested multiple times? Or if the same set of IVs is used (even if the DVs are different)? Or is it really about the total number of tests conducted with the same data set with no relevance what DVs/IVs are used and how many times they were the same?
I looked at similar theses from study colleagues but most of them are not correcting the alpha level at all even though they have multiple research questions with multiple hypotheses, sometimes testing the same DVs multiple times and often using the same IVs in multiple hypotheses. In most cases the alpha level was set to .05 and IF they conducted i.e. a MANOVA they would select Bonferroni or another for posthoc analysis - the overall consensus seemed to be that in multivariate testing the computer program "did the work" because "we were told to click on Bonferroni". But that there have been multiple tests on the same data set never lead to a corrected alpha level.
Can someone please help me out? I'm probably just standing in my own way because I've read so much about the pros and cons and so many different "interpretations" of multiple testing and when to actually calculate the corrected alpha level that I feel like I don't understand it at all anymore. I don't want to follow the procedure of my study colleagues simply because it is "what everyone seems to do" because in my mind there IS an issue with Alpha-Error-Cumulation if I perform that many tests on one data set.
I'm sorry if this is confusing and a lot to read - I feel like I would have reached a level of confusion where I would only get it if someone explains it to me if I would be 5 years old. The more articles and book chapters I read, the more opinions I seem to get.
Thanks in advance to everyone willing to take time, share knowledge and help me to find a solution for my thesis because I strongly dislike the feeling of just doing "something" without reasoning and being able to discuss WHY and WHEN I chose to correct (or not correct) the alpha-level within my thesis.
Melissa
It is confusing because there isn't in my view a clear consensus in the literature. Generally we correct for a family of tests (hence correction for familywise error). However, a family can be defined in a broad or narrow way and that gives a fair bit of leeway and room for fudging (especially if not set out in advance).
In general I think its OK to separate out tests for things like manipulation checks as being in their own family (one family per check). I'd also note that for a manipulation check you might well interpret it more qualitatively.
For each hypothesis I'd argue that one could treat any tests for that as a single family even if using different DVs. I might even pool linked hypotheses if for example you'd treat a test with p < .05 as evidence supporting a theory one could argue that one should correct for the whole set.
For post hoc or non-predicted effects one could also reasonably group them as a single family. I think this can be a sensible strategy for things like large correlation matrices or interactions in multi-factor ANOVAs.
Last but not least I'd really recommend not to use Bonferroni. Particularly for large numbers of tests it is very conservative and you lose a great deal of statistical power. Depending on context I'd use a more powerful test like the Holm test (which is uniformly more powerful than Bonferroni for any situation) or something like the Hochberg correction which is even more powerful than with Holm or Bonferroni and appropriate whenever a set of test statistics are positively correlated (e.g., a correlation matrix or a set of pairwise comparisons). For very large numbers of tests you may want to control the false discovery rate rather than alpha.
e.g., With the following 6 p values here are the adjusted values (equivalent to testing by correcting alpha, but more convenient to compute):
uncorrected p: 0.098000 0.023400 0.021100 0.005670 0.000345 0.000201
Bonferroni: 0.588000 0.140400 0.126600 0.034020 0.002070 0.001206
Holm: 0.098000 0.063300 0.063300 0.022680 0.001725 0.001206
Hochberg: 0.098000 0.046800 0.046800 0.022680 0.001725 0.001206
FDR: 0.098000 0.028080 0.028080 0.011340 0.001035 0.001035
For Holm and Hochberg the can be done by hand and just requires the original p values. If you use R use can also run these easily from a set of p values with the p.adjust function.
This isn't exhaustive - there are other approaches - especially where the are logical relations between the tests. For example the Westfall procedure capitalises on the correlations between non-independent tests and their logical structure (but is very computationally intensive and isn't always superior to something like the Hochberg correction).
This blog is also useful:
http://daniellakens.blogspot.com/2020/03/whats-family-in-family-wise-error.html
and I discuss the topic and various options in my book:
Baguley, T. (2012). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave.
- Badges
- Science topic
- Similar topics
- Bioinformatics
- Bioinformatics and Computational Biology