If you find a significant interaction in, for example, a 3 x 3 Anova and the interaction plot (with 9 data points) suggests it might be due to just one data point, would it be good practice to examine all possible 2 x 2 interactions to find which are significant?
The thing is, I have never seen this reported in a published paper. I have wondered about this before, but co-collaborators have shied away from looking deeper and editors/reviewers have never asked when we have said there was a significant interaction of two factors. The bottom line is that I don't really know and have never been challenged on it - but I'd like to learn.
I would be grateful for any suggestions - or any instances where it's been mentioned in a published paper. (I'm guessing that inflated risk of Type 1 error is going to be a problem)
I think your best bet would be to run what are known as "post hoc tests." I found quite a few references when I did a web search for:
post-hoc tests anova
absolutely, if you apply one way ANOVA for examine whether more than two groups differ significantly, it is not enough to do so. because, overrall result of Overall ANOVA is significant if only one couple of groups deffer. therefore if we have more than two groups it ought to apply Post hoc test.
I suggest having a look at some of the literature on decomposing ANOVA interactions (if you google that you'll find some good advice by people like David Howell as well as some SPSS syntax for helping you do it http://www.tqmp.org/RegularArticles/vol08-1/p001/p001.pdf). You're wanting to do simple (interaction) effects analyses by the sound of it.
Firstly, you must see if simple interaction effects are significant. If yes you must investigate second-order simple effects, and if they are significant, then you can run pos-hoc analysis, when the variable has more than two levels.
I've got no quarrel with anything said by our colleagues, but I think that post-hoc analysis alone is limiting - especially when the interaction involves a time variable - that is it's a repeated measures analysis. In such a case, asking the statistical package to print and plot the marginal means gives you not only a numerical value for each variable at each point in time, but perhaps more importantly gives the researcher a graphical representation of the shape of the data over time.
The vectors of the interaction lines should either cross within the frame of the plot, or be seen to be on a course to cross outside of the frame.
Remember, there are 9 2x2 variations in a 3x3 plot - conducting that many individual comparisons inflates your type 1 error - be sure to use a test which controls for this - not just repeated t-tests (everyone has their favorite post-hoc test) or specify a stricter alpha level for these analyses.
I'm going to take a different approach. Was this three-way interaction predicted? Are there competing models that did or did not predict the interaction? I ask because post hoc tests of an unexpected interaction can be dangerous. In my experience, a lot of these unexpected high level interactions won't replicate unless they are pretty strong. So, be careful!
What are generally referred to as "post-hoc" tests typically use the marginal means which are not appropriate to locate which levels of one factor are interacting with which levels of the second factor because the marginal means themselves are affected by the interaction.
The process you mention where the larger design is broken down into all possible 2 x 2 designs is called "simple interaction contrasts" or "pairwise interaction contrasts" in most experimental design texts. You are exactly right that content researchers tend to shy away from implementing these in empirical studies, likely because of the large number of contrasts that must be tested.
Nonetheless, there has been some semi-recent research in this area and ways to effectively control for Type-I error rate for the multiple contrasts. Klockars & Hancock (2000) developed a modified Scheffe prodcure, Bird & Hadzi-Pavlovic (2008) outlined a maximal root procedure, and I have an in press paper with Greg Hancock (available on my RG page if you're interested) on a modified Bonferroni approach that was shown to generally be more powerful that the previous two methods (if you thinking of taking the plunge and using simple interaction contrasts!).
- Badges
- Science topic