I ran ANOVA test to examine the mean differences among four groups and results were found significant at 0.05 Alpha level. Then, I ran Scheffe's post hoc test to see the between-group differences but none of the group showed significant differences between them.
Is it common? If yes, how should i present my results?
Dear Harmandeep,
Yes, it is quite possible for this to happen. It should be written up exactly as you have explained it.
A couple of additional points:
1. Make sure the post hoc test you have chosen is appropriate for your data. Andy Field provides a useful summary of how to do this.
2. If the significance value you obtain lies between 0.05 and 0.1 you can report on it informally as weak evidence to reject the null hypothesis.
Hi dear friend Look at one important point. Scheffe's post hoc is used when we want to make double comparisons simultaneously and together. It is recommended to use the LSD and Bonferroni tests.
The ANOVA is helping you answer the question is there any effect at all? The multiple comparison method (Scheffe in this case) is asking is there a difference in specific pairwise comparisons. There is a problem in looking at pairwise comparisons in that a large number of statistical tests will result in one or more showing a "significant difference" even if no such significance exists. Remember that the p-value is simply the probability of getting an outcome as extreme or more so given that the null hypothesis is true.
Now think about the number of statistical tests: I have an experiment where I have 5 treatments. I can test any one pair of these treatment. I could test up to 5!/(3!*2!) pairs. I might then rerun the experiment several ways and have a manuscript with ten such tests. Each additional test improves the probability that I get at least one significant outcome (1-(1-alpha)^n where n is the number of tests, alpha is the level where you declare significance). So statisticians have devised ways to correct this problem. Bonferroni is one such, Scheffe is another, and there are several dozen more options. However, the corrections are not "free." The Bonferroni approach is to divide alpha by the number of test that are performed. So now alpha is 0.05 for my experiment, but for individual pairwise comparisons alpha is now alpha/n.
So the ANOVA asks if there is a difference and the answer is yes. However, the p-value is only 0.0427. I have an experiment with six pairwise comparisons and so my Bonferroni alpha for each test is .0083. No individual pairwise test achieves this and I have your result.
Some things to try. I have not seen Scheffe used in a while. Try a different method. Try thinking about what you need. Do you really need all pairwise comparisons, or only a subset? If you will only discuss the outcome from a few then only test those few. Do not test more than you need to. Consider increasing replication. Be prepared to accept that there is not as much difference in your treatments as you would like. Sometime using this outcome and redesigning the experiment is useful. There may be an effect, just not the way you were asking the question.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA