In a pre-post design with two intervention groups, there was a difference between the groups A and B in "pre", but not in "post", as seen in preliminary results examined via t-tests. Also, within group A, there was a significant difference between pre and post.
Later on, a more sophisticated (and complex) repeated measures ANOVA did not reveal a time*group interaction. However, the graph to be included in the paper (with offset 0!) makes one wonder if there might be a true difference, as group A showed markedly different values in the pre-test. Looking at pairwise comparisons in the rmANOVA, they reveal the significant differences seen earlier in the t-tests.
So I wonder... is it a legitimate way to basically say "ok, this interaction was not significant, but going from the graph, we would be remiss not looking into the numbers behind it and report them" (and of course discuss the while issue properly, limitations and all!)? I, as a reader, would certainly wonder about that graph, if presented with no further examination.
PS: Yes, it is odd to have a pre-difference, but it is what it is. Plus, the results would still make sense theoretically (supported by literature). N=139.
I personally put as much faith in the interaction plot that showed lack of parallelism as I do in significant cross product terms.. Best wishes, David Booth
I don't see the problem. Your data is sufficient to conclude the sign of the effect in group A. It is not sufficient to conclude the sign of the effect in group B. And the non-significant interaction also tells you that it is not sufficient to conclude the sign of the difference in effects between both groups. This information is important, because this is not easy to see from the graph alone.
Hello Nina,
So, while group A shows a statistically discernable increase from pre to post, group B does not, nor is the interaction (group by occasion) distinguishable from zero (as both you and Jochen note). To me, that means exactly what it says.
If you've got a good reason (e.g., theory) to posit that there should be an interaction, then clearly the magnitude of the effect wasn't big enough to be statistically salient given your sample size. Remember that plots of cell/group means do not show the variation, which is important. As well, the visual scale of plots generated by statistical software may make differences appear to be large visually, but in absolute measure terms, they may be minor.
Good luck with your work!
As I understand it , David is saying no if you don't have a theory that gives strong reason to expect an effect. My suggestion is to to investigate until you understand the effect if any. David Booth
Hi everyone, and thank you for your answers!
Indeed, when standard deviation was included in the graph, it looked a lot different. Very small bars and huge deviation "T"s. Thank you for reminding me to do that!
Plus, although the effect in general was "nonsignificantly present", in lack of better words, it were only a few people who put it within the level of significance. While trimming those few people made the effect disappear in the t-tests too, winsorizing them did not. So there was kind of a tendency... but really not enough to assume it was an actual effect.
Also, calculating change scores (post - pre) and examining those did not show signficant results .
Therefore, I concluded that the lack of an interaction effect was correct and there was indeed no significant effect.
So, if anyone reads this and wonders the same thing... Learn from my agony over this:
1. Look at the variance in your data.
2. Look at outliers (boxplots are an easy way to start)
3. Try to look at your pre-post design by calculating change scores and examining them.
Thank you!
Nina
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA