I've conducted two-way ANOVAs and I'm trying to clarify what my supervisor means when he says:
"When you analyze components of an interaction you do not have to adjust for multiple testing when it is predicted but you do if it is not predicted"
I asked him for clarification, but his answers were even more confusing.
I ran 3 two-way ANOVAs.
ANOVA 1)
DV: Preparedness for employment scores (PES-PD)
IV 1: Employment status: Employed vs Unemployed
IV2: Personality Disorder Characteristics: PD group vs Non-PD
Results:
- there was a significant interaction
- and so I performed a simple mains effect interaction for Employment Status was performed with statistical significance receiving a Bonferroni adjustment and being accepted at the p < .025 level.
There was a statistically significant difference in mean PES-PD scores for PD participants in either employment status level, employed or unemployed, and also Non-PD.
I than rain pairwise comparisons for each simple main effect
All pairwise comparisons were run for each simple main effect with reported 95% confidence intervals and p-values Bonferroni-adjusted within each simple main effect. Mean PES-PD scores for employed and unemployed PD participants were 58.10 (SD = 14.86) and 43.55 (SD = 16.95) respectively. Employed PD participants had a statistically higher mean PES-PD score than employed PD participants, 12.37, 95% CI [9.60, 15.84], p =0.01. Mean PES-PD scores for employed and unemployed Non-PD participants were 70.84 (SD =15.44) and 76.61 (SD =12.88). Unemployed Non-PD had a statistically significantly higher mean PES-PD score than Non-PD unemployed participant, 33.05, 95% CI [30.23, 35.74], p = .001. [PF1] [SL2]
Does he mean that I didn't need to run the pairwise comparisons for each simple main effect because I found a significant interaction? Did I do what he said in bold?
ANOVA 2)
DV: Preparedness for employment scores (PES-PD)
IV 1: Employment status: Unemployed vs Looking for Work
IV2: Personality Disorder Characteristics: PD group vs Non-PD
Results:
- no sig. interaction
- so I performed a simple main effect for Employment Status (Unemployed versus Looking for work) was performed, which indicated that the main effect was statistically significant, F (1, 464) = 20.542, p < .001, partial η2 = .0.42.
- I also ran an analysis of the main effect for subsamples (Non-PD versus PD) was also performed, which indicated that the main effect was statistically significant, F (1, 464) = 236.050, p < .001, partial η2 = .0.33.
- I then ran an all pairwise comparisons were run where reported 95% confidence intervals and p-values are Bonferroni-adjusted
- i also then provided weighted means
He also said that, "If the interaction is not there than you are doing a post hoc analysis when you are looking at the mean differences between any subgroup and should adjust for multiple comparisons" .
My questions are did I not do the comments in bold in the second ANOVA?
Any answers are welcomed, along with references for further reading.
He means that you adjust on the x's but not on the y's, usually Y=DV, X= IV. See the attached which may be helpful: as well as:
https://www.amazon.com/Design-Analysis-Experiments-Douglas-Montgomery/dp/1119589061/ref=sr_1_2?keywords=montgomery+design+and+analysis+of+experiments&qid=1566300383&s=books&sr=1-2
Best, David Booth
"adjust for multiple testing" probably refers to the Bonferroni adjustment. Beyond that, I'm not sure what your supervisor intends.
I agree that this is probably a issue of Bonferroni adjustment. In particular, if you began with a hypothesis for each of your effects, you do not need to make such an adjustment, but if you ran every possible test to find out which effects were significant then you should make an adjustment to reflect this kind of "guessing game."
After reading the above comments, I withdraw mine. I don't know either. In any case, what I said was wrong. I should have said betas not x's.I still very much like the booklet I attached, Please read it in good health. Best, D. Booth
another name for a Bonferroni "adjustment" is a Bonferroni "correction."
Chapter 15 Two-Way Analysis of Variance
bcs.whfreeman.com › webpub › statistics › companion_chapters
Two-Way ANOVA. Two-Way Analysis of Variance. Introduction. In this chapter, we move from one-way ANOVA, which compares means of several populations ...
- Badges
- Science topic
- Similar topics
- Geoscience
- Atmospheric Sciences
- Atmosphere