What F test are you using? There are different tests for different purposes that use the F statistic.

Dear Emily,

It is probably impossible for me to provide a definitive answer to all your questions without more knowledge of your study and measures. However, I think I know what is happening in general. The ANOVA procedure is significant when their is at least one significant difference between groups. However, ANOVA is based on an experiment-wise error rate as is the Tukey Honestly Significant Difference Procedure (that is why they use the word, honest). However, when you perform pair-wise comparisons, they are based on pair-wise error rates. For example, if you performed the pair-wise tests at the .05 level with four groups, you would be preforming six pair-wise tests and your experiment-wise error rate would be much higher being approximately .25.. Another observation is that based on the description you provided of your data, it appears that the ANOVA assumptions may have been violated to the extent that the Kruskal-Wallis approach is more appropriate. KW is a less sensitive procedure.

One final observation. The purpose of research is to discover truth. A non-significant finding is not bad if you can estimate the Type II Error as no relationship is also an important finding. I imagine that thousands if not millions of non-significant research studies have been replicated because journals seldom publish non-significant findings.

Good luck with your work!

J. McLean

Thanks for your answer, James.

I know that non-significant findings can be just as important and telling than significant ones. It's just that looking at my data, there are clear differences. I'll add an example of two of the graphs so you can see it.

Thanks for describing the differences between ANOVA and pairwise comparisons. So since both the ANOVA/Kruskal Wallis are the appropriate tests to use, it would be inappropriate to use pairwise comparisons since there would be different error rates? Since ANOVA/KW are saying n.s., is it okay to state that they found them n.s., but note that pairwise comparisons found significant differences? Or should I just give up on the pairwise comparisons? haha

Thanks for your help!

Emily

Hi Salvatore, I used the F test with the R code of res.ftest.

Maybe I should've been using T-tests instead of F-tests? Not really too good with stats knowledge!

• I don't see any R function res.ftest . I suspect you are using an F test that compares variances of groups, not the location (like the mean or median). So that's testing something totally different than anova, t-test, Kruskal-Wallis and so on.
• Based on your graphs, it does look like the variances among groups are different. But that the locations (mean, median, and so on) may not be meaningfully different. How you proceed really depends on what you want to know.
• If you conduct a test for equal variances, be sure to choose a test that can handle non-normal data. Perhaps Brown–Forsythe, robust Levene’s test, or Fligner-Killeen.