Dear collegaues,
I have to analyse the results of an intervention research where 3 dependent variables were measured with questionnaires. Participiants were randomly assigned to intervention and control groups. In the intervention group, the intervention was a short exercise about goal setting and values clarification. In the control group, the control task was a neutral discussion. The questions: is there a difference in the dependent variables (anxiety, self-efficacy, psychologicla inflexibility) before and after the intervention in the CG? Do the IG and CG differ form each other?
In the intervention groups, a before and after measurements were conducted. But in the control group, there was only one measurement, AFTER the control task. Sample sizes: IG 50, CG 70 people.
My question is which kind of statistical tests can be used for answering the questions (or any other questions based on the available data)? I know I can run (group comparison) independent and paired (in the IG) t-tests. My second gues were linear regression (dummy variable: intervention happened or not) but the assumpations are heavily violated. For a mixed model ANOVA, assumptions would be met, but with the missing "before" measurement in the CG, I do not think is it possible.
Do you have any suggestion more sophisticated and complex than the t-tests?
It is an MA thesis, so it is seuggested to use multivariate methods.
Thank you very much,
Marton Kovacs
Of course, you can use traditional methods, such as t-tests. However, be aware of the debate in the scientific community about the validity of NHST, including t-tests. Therefore, you can use estimation statistics, including gain probability analysis (Trafimow et al. 2022, 2024) or exceedance probability analysis (Huang 2022).
Trafimow D, Hyman M R, Kostyk A, Wang Z, Tong T, Wang T, and Wang C 2022 Gain-probability diagrams in consumer research International Journal of Market Research 64(4) 470–483 https://doi.org/10.1177/14707853221085509
Trafimow D, Tong T, Wang T, Choy S T B, Hu L, Chen X, Wang C, and Wang Z 2024 Improving Inferential Analyses Pre-Data and Post-Data Psychological Methods (to be published)
Huang H 2022 Exceedance probability analysis: a practical and effective alternative to t-tests Journal of Probability and Statistical Science 20(1) 80-97
You can use our paper as a model. We recorded answers before and after surgery. The paper is on RG.
Depression, anxiety, hostility and hysterectomy - PubMed
Post-hysterectomy symptoms constituted a continuum to pre-surgery signs of depression, anxiety or hostility. Sixty-five women (aged 32 - 54 yrs) were assessed at 2 months before to 8 months …
- Author: S Béatrice M Ewalds-Kvist, Toivo Hirvonen, Mårten Kvist, Kaarlo Lertola, Pirkko Niemelä
- Publish Year: 2005
Hi Marton, you're already on the right track with your thinking. Since you don’t have pre-intervention data for the control group, a true mixed-model ANOVA or repeated measures design isn’t possible here.
What you can do is:
ANCOVA could also work if you use pre-test scores from IG as covariates to adjust post-test scores (some people do this, but it's a bit controversial when the covariate is missing for one group).
Regression with group as a dummy variable is fine if you're cautious with interpretation and check for violations. Even better if you run robust regression methods to handle assumption issues.
You might also mention the design limitation (no pre-test for CG) clearly in your discussion, since that restricts the strength of causal inference. Hope this helps clear things up.
- Badges
- Science topic
- Similar topics
- Physical Sciences
- Materials Science
- Materials Chemistry
- Polymer Chemistry