When we teach about family-wise error rates, we usually use straightforward examples such as:
I conduct an ANOVA and obtain a statistically significant result. Next, I conduct post hoc comparisons to determine which cells significantly differ. However, I need to adjust alpha since I'm doing multiple hypothesis tests.
...but are family-wise error rates an issue only in these narrow situations; namely, when multiple hypothesis tests are done on the same datum and all related to a single underlying "question", or "inference".
For instance, if a study contains 3 experiments, all slightly different in design, but all intended to address a single question or phenomenon - is a statistical adjustment required? Why or why not?
*My point in asking this question is that what exactly constitutes a "family" is far from clear (at least to me).
This is a good question. I don't know if there is a consensus among statisticians.
To me, tests belong to a family when and only when the test should identify "hits" among several possible candidates, like a screening. This is not about the "sam underlying question" - it is about a whole family of related, but individual questions (is the evidence from the data sufficient to demonstrate an effect of drug A?, of drug B?, of drug C?, ... )
The example with three experiments addressing the same underlying question is actually the opposite. They all contribute evidence to the same question, and instead of correcting for multiple testing I would rather even combine the p-pvalues (Fisher's method) to proxy the combined evidence that all the estimated effects are in the correct direction.
Jochen Wilhelm This is an interesting take. I want to make sure I understand this idea of screening among possible candidates. By "candidates" do you mean something along the lines of "alternative explanatory factors" that could account for the pattern in the data - something like this?
If I'm conducting multiple hypothesis tests, and they're all comparing the effects of drug A, drug B, and drug C on some DV...these tests should be considered as belonging to a family.
That's correct.
Another prototypical example for a "family" is the screening for a correlation between some of the variables X, Y, Z, ... (e.g. between BMI, blood pressure, education, income, sex, age, mobile usage, household size, etc.).
I am also confused a bit by this correction. I understand why one would do this regarding (Neyman-Pearson ideology) in the same dataset but, it also seems like Fisher was a bit warry about these error claims (considering the scraps of his work I get fed).
However, if they are not from the same dataset, then not??? I assume there is some rationale on this, but have not found it. I have seen in some studies where they the do multiple t-test or mw-test, but not sure if they then should or should not correct (they also do not address this). I mean if we would correct for it this should also be the cause in SEMs or multiple correlations, which we don't. What if we apply it a bit more loosely and realistic and follow in the Philosophy of Fisher. I'am curious to more response and views on this topic! good discussion.
Dear Wim Kaijser
The multiplicity problem is not depending on the fact whether the tests are done on the same set of data (like in oneway ANOVA designs) or on different sets (like obtained from different assays and/or from different experiments).
It depends on if the different tests are in support of the same underlying entity. If they are, they "cooperate" and pile up evidence. If they are about different entities, they "compete with each other" (-> error inflation).
Comparing the effects of a set of different drugs leads to competeing tests, because the drugs are different entities. Drug A may have an effect but not drug B. This also applies when the effects of some drugs are measured in different labs. When lab A tests drug A against placebo, lab B tests drug B against placebo etc., they may even use different assays - and still these are competing tests and a correction for multiple testing is required to find out if the data from all the labs about all the drugs is sufficient to identify one or more effective candidates.
In contrast consider a lab that tries to show that some cell-physiological signalling pathway is involved in the reaction of cells to some environmental stimulus. To show this, a series of different experiments is performed, blocking different molecules within the signalling pathway. The corresponding tests are cooperative, they add evidence that the pathway may be involved. There is surely no need to correct for multiple testing. If and how the evidence could be combined is a difficult question and depends on the complexities of the studied pathway.
Did that help a bit? Or could I manage to increase your confusion even more?
Personally, after reading familywise and 3 experiments, I wonder if maybe we are not overlaying two different things, namely per-experiment and experiment-wise error probabilities.
Correcting for multiple comparisons though makes pvalues lose their intuitive meaning as evidence against the null hypothesis, so if one does not have a large number of comparisons, I would keep the raw pvalues and use common sense rather than an arbitrary threshold.
Blaine Tomkins please note that the comparisons can be performed either if the ANOVA (even though other F statistics perform better in certain conditions†) is significant or not: these tests are called protected and unprotected and different tests (e.g. Tukey's HSD, Dunn-Bonferroni) have different performances on per-experiment and experiment-wise error rate which change depending on the nature of the test (protected or unprotected).
It is clear that picking one single method and using a threshold of 0.05 can be extremely restrictive, and two people analyzing the same data could get two different set of results.
Ultimately it really depends on how you weigh sensitivity/sensibility.
† https://psycnet.apa.org/record/1979-30184-001
Stefano Nembrini ,
I find the terms "per-experiment" and "experiment-wise error probabilities" extremely confusing (and not heplful).
I suppse an F-test comparing a linear model with an intercept plus one or more further coefficients to an intercept-only model is something that gives an "per-experiment" p-value that is compared against the "experiment-wise error probability"? The whole story about protected and unprotected tests to control (or fail to control) the family(!)-wise error rate (FWER) is unneccesary in this context. Dunnett controls the FWER for "multiple-to-one" comparisons, and Tukey for "all-pairwise" comparisons in oneway-ANOVA like models. There is no need to think about "F-test protected" procedures, and designs in which such protected methods are available are not congruent to scenarios where a family of tests exists that requires considering or controlling the FWER.
To my understanding, only the distinction between "test-wise" and "family-wise" error rates (TWER vs. FWER) is relevant. The important point there is the question Blaine Tomkins asked: Who's in the family?
Jochen Wilhelm still somehow confused, does this example makes sense? I want to compile evidence that a particular drug with similar "structures" (e.g., A, B, C) and provide evidence for an "effect". Than no correction would be need as A, B and C are sharing the entity "structure". But, when I ask does drug A, B or C - same as before, but with fixed "normal" environmental concentrations - at this particular dose has an "effect" then it would be competing, as A, B and C are treated as separate entities. The issue that I have too find a valid argumentation that these are separate entities. Perhaps in some cases this is very clear, but not sure if this is for all cases which makes it very complex.
I'd say yes, testing the same "structure" using three different drugs (assuming that there are no other effect interfering with the action of that structure), the evidence from the experiments should pile up. As an extreme case you may think of using three drugs with minimal differences, actually with no difference at all - so essentially you use the same drug in three experiments. In each experiment you make 5 independent measurements. Due to the large variance all the p-values are rather large. "Piling up evidence" means that you could pool the three times 5 measurements and so have a higher power (i.e. you get a lower p-value for the same "true" effect). This should be (more or less) equivalent to combining the three p-values.
To the second part (dose-effect of at least one of the drugs). I also agree here. This is a screening. The difference to before is that here you assume that the drugs work at different doses because of some feature of these drugs (independent of "structure") which are all different between the drugs. You are testing a family of different features (or feature-combinations).
And I also finally agree that this can be a rather complex question in practice. Most authors don't seem to think about such things at all. They just always use some correction for muliple testing when doing some kind of ANOVA and otherwise they completely ignore the topic (except in meta-analyses, where evidence is purposefully combined).
Before starting the analysis you may choose how many comparisons you will make. For instance simply comparing to a single outcome designated as the control will require a very much smaller number of comparisons and thus less adjustment than will comparing every treatment to every other treatment. The latter is usually not practical because the large number of comparisons tends to require an adjustment that requires a tiny individual 'p-value' for each one. That would, in turn, usually require a huge sample size and might be hypersensitive anyway.
- Badges
- Science topic