Hi
Im working on data form an experimental study. I have a categorical independent variable with three levels, a continuous dependent variable and a continuous covariate.
An ANOVA checked the assumption of independence of covariate and treatment effect and found that the three experimental groups differ on the covariate. I´m not entirely sure in what way this affects my inferences.
Now in some ways it is expected that the groups would differ since the covariate is how long the intervention took and one of the conditions took longer to complete than the others. I want to know if the longer time spent in that condition impacted the dependent variable. This is why I was thinking about ANCOVA so I would be able to separate variance associated with each variable. However, with the violation of this assumption I got stuck since I don´t think I can trust the numbers coming out of the analysis.
The univariate ANOVA (i.e., without covariate) generates a significant difference between the three groups. Adding the covariate removes these group differences. It would appear as if the time spent administering the intervention (i.e., the covariate) is what drives the mean differences in the dependent variable. But I´m not confident I can trust the results due to the covariate/treatment association.
What is happening is that the covariate "takes" variance explained from the independent variable rather than from the unexplained variance. This reduces the impact of the independent variable rendering it non-significant when the covariate is in there. My thinking is that this shows that the covariate and the independent variable are sharing variance and this is a consequence of them not being independent, as the ANCOVA assumes.
Any helpful thoughts out there?
Hello Daniel Pettersson. I can't believe I'm writing this, because I am considered a mediation skeptic by many of my students & colleagues. Nevertheless, what you describe sounds to me like a mediation model, with X = your categorical treatment variable, Y = the DV, and M, the suspected mediator = length of time to complete the task. Here is an example (using Stata) that is structurally the same as your data, but with X = 3 SES categories that are observed, not manipulated:
https://stats.idre.ucla.edu/stata/faq/how-can-i-do-mediation-analysis-with-a-categorical-iv-in-stata/
If you tell us what software you use, someone may be able to point to examples using that software.
If you do use mediation analysis, I urge you to read Klaus Fiedler's 2018 article shown below. I consider it essential reading for all mediation analysts. (Perhaps that is why I am considered a mediation skeptic.) HTH.
Article Unwarranted inferences from statistical mediation tests – An...
PS- Here's another article you may find useful.
Article Design approaches to experimental mediation
Regardless of the method and analysis of this study, this was interesting to me
"However, with the violation of this assumption I got stuck since I don´t think I can trust the numbers coming out of the analysis."
Which assumption is violated? You mean that?
"An ANOVA checked the assumption of independence of covariate and treatment effect and found that the three experimental groups differ on the covariate."
This is not assumption analysis of covariance
See the link below. Especially Assumption #1 to #9
https://statistics.laerd.com/spss-tutorials/ancova-using-spss-statistics.php
Daniel Pettersson , it sounds like you have diagnosed the situation. You have two independent variables that are correlated. If they are strongly correlated, then it may not be useful to include both in the model. ... A potential approach is to use Type I Sum of Squares, with the covariate entered into the model first. But this would yield what you have reported: when you account for the length of the procedure, there's not enough "signal" left in the data to find a significant difference among the treatments.
Thanks for all the answers!
Salvatore S. Mangiafico This is actually what is happening. There is not enough signal left when the covariate is in the model. This is because the covariate and the IV share too much variance; the covariate end up explaining variance associated with the experimental effect instead of the error variance, thereby reducing the experimental effect.
As Abolfazl Ghoodjani pointed out this is not about a statistical assumption but rather an interpretational issue. Bruce Weaver I use SPSS and I have attempted a mediation and a moderation with the PROCESS tool.
I have also tried dummy coding the experimental groups to run a regression. I have also run ANOVA and ANCOVA. All these linear methods reach the same conclusion - there is some effect of group without the covariate but no effect of group with the covariate. This is actually what I am hoping to test (i.e., does the effect remain or disappear when accounting for the covariate?). The question is can I make such an interpretation?
I ask this question because I have read that ANCOVA cannot be used to answer such questions (i.e., ANCOVA cannot "control" or "adjust" for real group differences on the DV). ANCOVA is instead a technique that can be used to reduce error variance and thereby increasing the experimental effect (i.e., a noise reduction technique). In a way I might be asking how the heck to account for a variable if this is not what ANCOVA does?
One possible option is the moderation/interaction approach. If there is an interaction between the IV and the covariate on the DV, then I might be able to say that the effect of the IV is dependent on the COV. If there is no interaction then I can say there is no dependence. With no interaction I would look at the main effects. Significant main effects with no interaction might lead me to conclude a "real" experimental effect independent of COV. Any thoughts on this solution?
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA