I have an animal experiment with 4 different groups:
A) control diet + vehicle
B) control diet + drug
C) experimental diet + vehicle
D) experimental diet + drug.
We then measure the expression of several genes.
The specific question that we want to ask is:
Does the dietary intervention affect gene expression independently of the drug?
Therefore I performed a two way anova (usign graphpad prism) and the results table states that there is no significant interation (P=0.5), there is a significant effect of the diet (p=0.027) and no effect of the drug (p=0.4398)
When I then perform a tukey comparison of all the values, there are no statistical significances in any of the groups.
I also did a statistical comparison of just the row factor (diet) with a
Sidak post-hoc test and again, there is no significant difference.
How would you then intrepert this data for publication purposes? On one hand, the anova tells us that a significant portion of the deviance in the groups comes from the diet, but without differences in any post-hoc test, I do not know how to write this up.
Thanks in advance to anyone that can help!
All you really need to write is that "there is no significant interation (P=0.5)", as this is about the main research question. I am not sure how Prism codes the factors. This needs to be clear when one tries to interpret the other coefficients. If it is simple treatment coding ("dummy coding"), then the coefficient for "diet" is a simple effect and gives the diet-effect in the reference category of the treatment (usually one would chose "vehicle" as reference here). If it is contrast coding, then the coefficient is a main effect and refers to the mean change in diet over all categories to treatment. What you usually want is the simple effect (did the diet work under control conditions?). The full model uses the full data and has the higest power to allow you interpreting the diet effect.
Post-hoc tests are a bad choice here, as they usually control a family-wise error-rate (FWER) of an utterly uninteresting "familiy of tests" (you don't do a screening here).
This part in your post is semantically wrong: "no effect of the drug (p=0.4398)". A large p-value is not indicating the absence of an effect. It only tells us that the data is insufficient (too less, too noisy) to be confident about the sign of the coefficient.
Jochen, thanks for the response! I will check the simple effect vs main effect and get back to you
However, in the meantime I have some follow-up questions.
1) can you further elaborate on your comment on post-hoc tests? I am not very familiar with this and i do not know what FWER is. If it is too cumbersome to explain here (which i fully understand) can you provide some literature (post-hoc tests for dummies, if that exists) I could read to get a better understanding of this?
2) Though I understand your point about semantics and the fact that "no effect of drug" could be incorrect, couldn't the same then be said for the interaction value? Maybe if I had more points, or less noisy points, the interaction value could be significant. Why then focus on the semantics of the "effect of drug" but not on the "interaction". Is there a mathematical reason for this?
3) Finally, the data IS a bit noisy and, at this stage, it will be nearly impossible to add more data points (experiments too costly and time consuming). I was wondering if it would be statistically sound to do something like:
a) run the anova
a1) If no significant interaction is found and only one of the factors shows a significant difference (for example, only diet in the original case) and the P value for the other factor is high, pull the groups together (that is, in the original example groups A+B vs groups C+D) and then perform a simple t-test on them
a2) if there is a significant interaction found, perform post-hoc tests, to verify which groups are generating these differences (possibly a tukey test)
I found the attached paper and, if I am reading it correctly (again, i am not an expert, so I could have easily misunderstood it) it seems to be state that, option a1 is a sound hypothesis, but I want to be sure.
As I understand it, if one of the factors is stated to not have a significant effect (with the caveats of low signal or noisy signal you mentioned) then we could assume that particular factor does not contribute significantly to the observed variation any more than random noise does. When I combine the groups and perform a T-test, I do get statistical significance, which, to my understanding comes from the fact that I now have more points per group and that the t-test has more power.
Thanks once again for the help with this.
1) FWER = family-wise error rate = Pr(at least one rejection | H0)
2) Absolutely. But here you wrote that the interaction was not significant. Tha't ok. To conclude that there is no interaction would be wrong. To conclude that the data is not sufficient to interpret the sign of the interaction (synergism vs. antagonism) would be ok.
a1) what should that help? But if you do that, then don't pool because of non-significance. Rather check the confidence intervals and try to find out if they cover some values that might be biologically/technically relevant. You may pool only if the confidence intervals are completely within a range that is irrelevant.
a2) It's trivial to say that is something is not significant, then it's not significant. Significance is a stochastic property of the observed data in relation to a model and a null hypothesis. The argument that non-significance indicates absence of an effect (and thus allows pooling etc.) is clearly wrong.
Article Absence of Evidence Is Not Evidence of Absence
Article Nonsignificant P values cannot prove null hypothesis: Absenc...
I don't wnat to make this more complicated, you can ignore the following, what I only add for the sake of completeness:
If you do an A/B test (Neyman/Pearson theory) rather than a significance test (Fisher's theory), you can control the confidence in deciding each of two alternative hypotheses (A or B). If A relates to "no effect" and B to "a minimally relevant effect", and the experiment is designed to give acceptably high confidences in either decision, rejecting B means accepting A (with appropriate confidence), so that one can then conclude that the effect is too low to be of any relevance.
However, ideally one would like to have some estimate of the effect, but that is possible only through Bayesian statistics and that would require a statement about a prior probability distribution over the effect (that is then conditioned on the observed data, to get a posterior distribution, which can then be used to judge if we find the effect being relevant).
- Badges
- Science topic