At university, we only do post hoc t-tests for main effects in an ANOVA for significant factors with more than 2 levels. When there are interactions, we only look at the profile diagrams and don't run any further analyses. Now I found a paper with the authors testing (ordinal) interactions with two t-tests to conclude that the effect vanishes in one condition. Is this a valid procedure? They write nothing about correction of the p-level because of alpha inflation. Would it further be permitted to run post-hoc ANOVAs for interactions on factors with more than 2 levels?
The main advanatge of post-hoc tests is that they use pooled estimates for the standard errors, what is more robust than estimating individual standard errors for each possible pair-wise comparison. Many post-hoc tests also control the family-wise error-rate. I doubt that this is always sensible.
And this leads to the answer to your question: it will be reasonable to use pooled estimates for the standard errors, and it remains the question how to control for multiple testing. What to chose depends on the experimental context and the design of the experiment. A strict control makes sense only if the whole experiment was designed for that purpose (planned contrasts, a given utility function, sample size calculation). This is very rarely the case in (basic) research, where experiments are just performed the way they are feasable (w.r.t. time, money, equipment, experience) and utiliy functions are not available. There simply is no control of error-rates in a strict sense anyway, so any strict control is not very sensible. The problem here is that many researchers interpret and report their results as if the experiment was planned to control error-rates, and this causes a lot of fuzz.
What would I do? - When the experiment is planned to control error-rates, all thinking has been done in the planning phase. A procedure to adequately control the desired error-rates on the interesting contrasts must have been considered. So such a question would then actually not arise at the time of the data analysis. When the experiment was not planned to control error-rates, there is no sensible way to make up such a control after the experiment. This is a logical misconception (related to the illogical confusion of hypothesis tests and significance tests).P-values can be calculated and they need to be interpreted in the context of the whole experiment anyway (simply saying that any p
The main advanatge of post-hoc tests is that they use pooled estimates for the standard errors, what is more robust than estimating individual standard errors for each possible pair-wise comparison. Many post-hoc tests also control the family-wise error-rate. I doubt that this is always sensible.
And this leads to the answer to your question: it will be reasonable to use pooled estimates for the standard errors, and it remains the question how to control for multiple testing. What to chose depends on the experimental context and the design of the experiment. A strict control makes sense only if the whole experiment was designed for that purpose (planned contrasts, a given utility function, sample size calculation). This is very rarely the case in (basic) research, where experiments are just performed the way they are feasable (w.r.t. time, money, equipment, experience) and utiliy functions are not available. There simply is no control of error-rates in a strict sense anyway, so any strict control is not very sensible. The problem here is that many researchers interpret and report their results as if the experiment was planned to control error-rates, and this causes a lot of fuzz.
What would I do? - When the experiment is planned to control error-rates, all thinking has been done in the planning phase. A procedure to adequately control the desired error-rates on the interesting contrasts must have been considered. So such a question would then actually not arise at the time of the data analysis. When the experiment was not planned to control error-rates, there is no sensible way to make up such a control after the experiment. This is a logical misconception (related to the illogical confusion of hypothesis tests and significance tests).P-values can be calculated and they need to be interpreted in the context of the whole experiment anyway (simply saying that any p
There are such post hoc tests, for example Dunn's Multiple Comparison Procedure. It tests pairwise differences and the advantage is that you can determine how many differences should be tested. You can also use post hoc tests like S-N-K or Tukey, but the problem with these tests is that they test any possible pairwise difference and there are a lot of them when looking at the cell means. The more differences you test the more the test has to correct against alpha-error inflation and so it becomes conservative.
I think to run two t-tests is a problem, because of alpha-error inflation. And of course, in my opinion, if you play the p-value-game you have to stick to the rules. Anyway, I agree absolutely to what Jochen Wilhelm wrote: never rely solely on p-values. I contradict in the point that the estimated effects have to follow well-known facts. This would hinder innovation and new insights ;)
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA