I observe that the interaction effect of A ~B, and the effect of B are significant (P
"Significance," even in the best of situations, is easily misinterpreted. It may be too misleading here.
You might try comparing model validation results with competing models, using test data. You might research "model validation." - You might also research the term "model selection." There are various ways in which independent variables might interact.
You could compare sets of estimated variances of prediction errors. (Note, for example, STDI in SAS PROC REG is the square root of that statistic.) They are designed to estimate variances, but because of the way sigma is estimated, bias also influences the outcome and they can be good general estimators of accuracy.
Graphical comparisons might also be useful, as in Figures 5 and 6 in the file at the following link: https://www.researchgate.net/publication/313925533_Comparing_Model_Performances_Graphically
There you do a kind of residual analysis or analyses for competing models on the same scatterplot.
Method Comparing Model Performances Graphically
The easy example. I have a chemical reaction. I find that as the concentration of chemical 1 increases in the reactor, I get increased yields. I now add some powdered platinum. No matter how much platinum I add, nothing changes. However, platinum is a catalyst. It does not consumed, it only facilitates the reaction of interest. Thus I expect that there is a significant effect of chemical 1, and that platinum is not significant. However, there is a huge interaction effect. Biological examples are more messy.
One problem is interpreting a non-significant statistical outcome as support for the null-hypothesis. It isn't. Please do not do this. Failing to find a statistically significant relationship may only mean that your sample size was too small.
Another scenario: Treatment A and treatment B are applied to improve crop growth. The effect of treatment A is similar to treatment B, but they also act synergistically. Thus, either treatment A or treatment B will be significant, and the one that is significant depends on which one enters the model first (Type I sums of squares). If you want to use Type III SS, then suppose that A and B are correlated, and that your sample size of 20 replicates results in a power of 0.9 for treatment A, while it is only 0.3 for treatment B. If treatment B enters the model alone, then it is just barely significant. Once treatment A is included, the correlation between the two knocks B out of the model. However, the interaction term is still significant. The solution is to keep models hierarchical: If the AB interaction term is significant, then B is kept in the model regardless.
Timothy states: "If the AB interaction term is significant, then B is kept in the model regardless."
Note that Fox, in Applied Regression Analysis and Generalized Linear Models, 2nd ed, 2008, John Fox, Sage, agrees on page 325, in his section (13.2.2) on "Variable Selection" in his chapter on collinearity (Chapter 13).
.
usual suspects : confounding variable or moderator variable
https://en.wikipedia.org/wiki/Confounding
https://en.wikipedia.org/wiki/Moderation_%28statistics%29
difficult to go further without any knowledge of the problem since the discussion involves a causal model
.
- Badges
- Science topic